\chapter{Implementation of \mysystem}\label{s:implementation}

\section{Overview}\label{ss:overview_implementation}

\begin{mynote}
	The contribution of this chapter is two-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}]
		\item We implement the real-world prototype of \mysystem (see \Cref{ss:detailed_implementation_overview}) realizing key features of the design.
		\item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming})
	\end{enumerate}
\end{mynote}

\section{Detailed Implementation of \mysystem}\label{ss:detailed_implementation_overview}
In this section we describe the detailed implementation of \gls{my_system}.
After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}.

Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
To understand \gls{my_system} it is crucial to provide a holistic view on the prototype.
Therefore, the rest of this chapter is structured in a top-down approach: \Cref{ss:detailed_implementation_overview}
presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}.
Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{images/implementation.pdf}
	\caption[The prototype and its components based on the architecture.]{The prototype and its components based on the architecture.
		The time-series data flows first to the \texttt{Grafana} (\myCircled{2a}) dashboard, \texttt{PostgreSQL} (\myCircled{3a}) database and \texttt{Redis} (\myCircled{3b}) cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.}
	\label{fig:implementation}
\end{figure}

At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.

\begin{figure}[t]
	\centering
	{\includegraphics[width=0.9\linewidth]{images/grafana_panel.pdf}}
	\caption[Example Grafana dashboard.]{Example Grafana Dashboard with downtime plotted against time.}
	\label{fig:example_grafana_dashboard}
\end{figure}

\code{Grafana} (\myCircled{2a}) is a state-of-the-art industry tool to visualize dashboards.
We posit it is crucial to include a user-friendly performance dashboard.
A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
\code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
Good alternatives to \code{Grafana} (\myCircled{2a}) are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}.

\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time.
We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ).
Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}.
The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval.
With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a})  tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema.
Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf} (see \Cref{lst:protobuf-schema}).

\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}.
In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data.
\code{Redis} (\myCircled{3b}) is the industry leader in key value caching.
The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}).

\code{PostgreSQL} (\myCircled{3a}) is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter.
The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}.
Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}.
Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}).
The many alternatives to \code{PostgreSQL}  (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}.
An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}.

Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning).
For a specific overview of advantages of \code{OpenDC}  (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/flow_diagram.png}
	\caption{The data flow within \mysystem.}
	\label{fig:flow_diagram}
\end{figure}

\section{Data Flow}\label{ss:data_flow}

In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
Efficient data flow is of utmost importance to \gls{dt}s.
In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
We describe the flow of data through a timeline.

First, the datacenter informs the \gls{dt} of an upcoming workload (\grayCircled{2}).
This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
The \gls{dt} stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
The analytics engine queries the \code{OpenDC}
simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
This data for the purposes of the prototype is stored in the \code{.parquet} files.
In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic.

In the meantime, the datacenter executes the workload.
The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}).
As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}).
At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}).
Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}).
This provides real-time feedback to datacenter operators (\grayCircled{8}).

Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}).
Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}).
Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights.
All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}).

\begin{figure}[t]
	\input{sources/listing_schema.tex}
\end{figure}

Of significant importance are data flows (\myCircled{2}) and (\myCircled{3}).
Due to the massive volume of data incoming from the physical datacenter, the \code{Redis} sink (\myCircled{2}) has been configured to filter out relevant packets.
\code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}).
On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}).
This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out.
\begin{figure}[t]
	\input{sources/listing_sinks.tex}
\end{figure}

\section{The \code{OpenDC} Scheduling Paradigm}\label{ss:opendc_scheduling}
In this section we introduce the scheduling paradigm of \code{OpenDC}.
\code{OpenDC}, a robust datacenter simulator uses discrete-event simulation.
``Discrete-event simulation models the operation of a system as a (discrete) sequence of events in time''~\cite{Wikipedia:article/DiscreteEventSimulation}.
Colloquially, it is akin to calling an \code{Update()} method on a set of objects to model changes in the simulator.
Scheduling in \code{OpenDC} also works using this method.
\Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation.

A task (\myCircled{1}) is represented by its submission time, duration and computational requirements.
In order to be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}).
The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time.
\code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}.
In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}.
In the \code{FilterPolicy}, a series of \code{HostFilter}s (\myCircled{3}) and \code{Host Weighter}s (\myCircled{5}) are used to find the matching host for the task.
In this project, the different filters and weighters take into account the default metrics (\ie \gls{cpu} capacity, \gls{ram} capacity, number of \gls{cpu} cores).
The host itself is represented as a \code{HostView} class (\myCircled{6}).
Importantly, the \code{HostView} class does not serve to simulate the behaviour of the host, but to provide a interface for the \emph{current state} of the host.
For actual computation, the \code{SimHost} (\myCircled{8}) class is used.
The \code{SimHost} object is created via the \code{HostProvisioningStep} class (\myCircled{7})~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}.


\section{Extensions to \code{OpenDC}}\label{ss:extensions}

\code{OpenDC} is a state-of-the-art datacenter simulator.
In order to turn it into a \gls{dt}, we have made several design decisions and extensions.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{SmartScheduler}}\\
	      The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions.
	      It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights.
	      For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}).
	      It \emph{replaces} the \code{FilterScheduler} (\myCircled{4}) in \Cref{fig:scheduling_in_opendc}.
	      Importantly, it is functionally almost exactly the same as the \code{FilterScheduler}, with the exception that it can change the scheduling outcome based on the information from the \gls{dt}.
	      It is a \code{FilterScheduler} with an attached network socket.
	\item \textbf{\code{KafkaMonitor}}\\
	      The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}).
	      We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic.
	      The \code{KafkaMonitor} class implements the generic \code{ComputeMonitor} interface, therefore the new extension follows the other metric exporting options in \code{OpenDC} (\ie follows the implementation for exporting  metrics into \code{.parquet files}).
	\item \textbf{\code{HTTPClient}}\\
	      The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter.
	      We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice.
\end{enumerate}
\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/scheduling_opendc.png}
	\caption[OpenDC scheduling paradigm.]{The scheduling paradigm in \code{OpenDC}. Adapted from Musc{\u a} \etal~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. The highlighted \textcolor{Orchid}{\ding{110} \textbf{Filter Scheduler}} is the component that is substituted by the \code{SmartScheduler}.}
	\label{fig:scheduling_in_opendc}
\end{figure}

\section{Python Modules}\label{ss:programming}

\code{Analytics Engine}, \code{HTTPServer}, \code{MonitoringService} are Python modules we prototyped for the reference architecture evaluation.
It is important to note that we do not evaluate the performance of \gls{my_system} due to the substantial overheads of the Python interpreter.
\code{Kafka}, \code{Redis} and \code{PostgreSQL} enable milisecond-latency and huge throughput within the system, however what stops us from a performance evaluation is the high cost of Python interpretation.
For future work, we envision a system that implements the reference architecture in a compiled language, \eg \code{C} or \code{C++}.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{AnalyticsEngine}}\\
	      The \code{AnalyticsEngine} module is necessary in order to encapsulate the logic of data preprocessing and analysis from monitoring.
	      This component can contain capabilities for different statistical metrics, subject to \gls{dt}s focus.
	      In \gls{my_system} \code{AnalyticsEngine} continuously checks whether the incoming datacenter sensor readings exceed different thresholds.
	      For example, the \code{AnalyticsEngine} is capable of calculating a similarity score $S$ between potential failure distributions and the true failure distribution.
	\item \textbf{\code{HTTPServer}}\\
	      The \code{HTTPServer} is crucial for interrupting the operation of the datacenter in order to adjust its operation or offer insights.
	      It maintains a python \code{Queue} structure.
	      The \code{Queue} \emph{producer} is the \code{AnalyticsEngine} (\grayCircled{4}).
	      The \emph{consumer} is the \code{HTTPClient} within \code{OpenDC} (\ie the real datacenter, (\myCircled{2}), (\myCircled{1})).
	      Both the consumer and producer interact with the \code{Queue} via the network and different HTTP endpoints of \code{HTTPServer}.
	\item \textbf{\code{MonitoringService}}\\
	      The \code{MonitoringService} is responsible for interacting with the \code{Redis} key value store.
	      It contains a \code{while True} Python loop which contains the function call to fetch the latest changes to the \code{Redis} stream.
	      Upon update, the \code{MonitoringService} informs the \code{AnalyticsEngine} that new data is awaiting \code{AnalyticsEngine} (\grayCircled{6}).
\end{enumerate}

