\chapter{Implementation of \mysystem}\label{s:implementation}

\section{Overview}\label{ss:overview_implementation}

\begin{mynote}
	The contribution of this chapter is two-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}]
		\item We implement the real-world prototype of \mysystem (see \Cref{ss:detailed_implementation_overview}) realizing key features of the design.
		\item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming})
	\end{enumerate}
\end{mynote}

\section{Detailed Implementation of \mysystem}\label{ss:detailed_implementation_overview}
In this section we describe the detailed implementation of \gls{my_system}.
After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}.

Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
To understand \gls{my_system} it is crucial to provide a holistic view on the prototype.
Therefore, the rest of this chapter is structured in a top-down approach: \Cref{ss:detailed_implementation_overview}
presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}.
Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{images/implementation.pdf}
	\caption[The prototype and its components based on the architecture.]{The prototype and its components based on the architecture.
		The time-series data flows first to the \texttt{Grafana} (\myCircled{2a}) dashboard, \texttt{PostgreSQL} (\myCircled{3a}) database and \texttt{Redis} (\myCircled{3b}) cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.}
	\label{fig:implementation}
\end{figure}

At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.

\begin{figure}[t]
	\centering
	{\includegraphics[width=0.9\linewidth]{images/grafana_panel.pdf}}
	\caption[Example Grafana dashboard.]{Example Grafana Dashboard with downtime plotted against time.}
	\label{fig:example_grafana_dashboard}
\end{figure}

\code{Grafana} (\myCircled{2a}) is a state-of-the-art industry tool to visualize dashboards.
We posit it is crucial to include a user-friendly performance dashboard.
A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
\code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
Good alternatives to \code{Grafana} (\myCircled{2a}) are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}.

\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time.
We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ).
Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}.
The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval.
With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a})  tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema.
Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf} (see \Cref{lst:protobuf-schema}).

\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}.
In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data.
\code{Redis} (\myCircled{3b}) is the industry leader in key value caching.
The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}).

\code{PostgreSQL} (\myCircled{3a}) is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter.
The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}.
Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}.
Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}).
The many alternatives to \code{PostgreSQL}  (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}.
An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}.

Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning).
For a specific overview of advantages of \code{OpenDC}  (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/flow_diagram.png}
	\caption{The data flow within \mysystem.}
	\label{fig:flow_diagram}
\end{figure}

\section{Data Flow}\label{ss:data_flow}

In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
Efficient data flow is of utmost importance to \gls{dt}s.
In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
We describe the flow of data through a timeline.

First, the datacenter informs the \gls{dt} of an upcoming workload (\grayCircled{2}).
This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
The \gls{dt} stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
The analytics engine queries the \code{OpenDC}
simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
This data for the purposes of the prototype is stored in the \code{.parquet} files.
In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic.

In the meantime, the datacenter executes the workload.
The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}).
As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}).
At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}).
Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}).
This provides real-time feedback to datacenter operators (\grayCircled{8}).

Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}).
Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}).
Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights.
All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}).

\begin{figure}[t]
	\input{sources/listing_schema.tex}
\end{figure}

Of significant importance are data flows (\myCircled{2}) and (\myCircled{3}).
Due to the massive volume of data incoming from the physical datacenter, the \code{Redis} sink (\myCircled{2}) has been configured to filter out relevant packets.
\code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}).
On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}).
This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out.
\begin{figure}[t]
	\input{sources/listing_sinks.tex}
\end{figure}

\section{The \code{OpenDC} Scheduling Paradigm}\label{ss:opendc_scheduling}
In this section we introduce the scheduling paradigm of \code{OpenDC}.
\code{OpenDC}, a robust datacenter simulator uses discrete-event simulation.
``Discrete-event simulation models the operation of a system as a (discrete) sequence of events in time''~\cite{Wikipedia:article/DiscreteEventSimulation}.
Colloquially, it is akin to calling an \code{Update()} method on a set of objects to model changes in the simulator.
Scheduling in \code{OpenDC} also works using this method.
\Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation.

A task (\myCircled{1}) is represented by its submission time, duration and computational requirements.
In order to be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}).
The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time.
\code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}.
In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}.
In the \code{FilterPolicy}, a series of \code{HostFilter}s (\myCircled{3}) and \code{Host Weighter}s (\myCircled{5}) are used to find the matching host for the task.
In this project, the different filters and weighters take into account the default metrics (\ie \gls{cpu} capacity, \gls{ram} capacity, number of \gls{cpu} cores).
The host itself is represented as a \code{HostView} class (\myCircled{6}).
Importantly, the \code{HostView} class does not serve to simulate the behaviour of the host, but to provide a interface for the \emph{current state} of the host.
For actual computation, the \code{SimHost} (\myCircled{8}) class is used.
The \code{SimHost} object is created via the \code{HostProvisioningStep} class (\myCircled{7})~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}.


\section{Extensions to \code{OpenDC}}\label{ss:extensions}

\code{OpenDC} is a state-of-the-art datacenter simulator.
In order to turn it into a \gls{dt}, we have made several design decisions and extensions.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{SmartScheduler}}\\
	      The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions.
	      It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights.
	      For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}).
	      It \emph{replaces} the \code{FilterScheduler} (\myCircled{4}) in \Cref{fig:scheduling_in_opendc}.
	      Importantly, it is functionally almost exactly the same as the \code{FilterScheduler}, with the exception that it can change the scheduling outcome based on the information from the \gls{dt}.
	      It is a \code{FilterScheduler} with an attached network socket.
	\item \textbf{\code{KafkaMonitor}}\\
	      The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}).
	      We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic.
	      The \code{KafkaMonitor} class implements the generic \code{ComputeMonitor} interface, therefore the new extension follows the other metric exporting options in \code{OpenDC} (\ie follows the implementation for exporting  metrics into \code{.parquet files}).
	\item \textbf{\code{HTTPClient}}\\
	      The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter.
	      We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice.
	\item \textbf{\code{CpuUtilVictimSelector}}\\
	      In order to ensure \code{OpenDC} can model different failure injection algorithms, we added new methods for selecting the hosts to be stopped.
	      The \code{CpuUtilVictimSelector} selects hosts based on their current \gls{cpu} utilization (\ie the hosts with the highest utilization are stopped first).
	\item \textbf{\code{RoundRobinVictimSelector}}\\
	      The \code{RoundRobinVictimSelector} is another algorithm to used to inject failures into hosts.
	      This method works in a round robin fashion, using the alphabetical order of the hosts (\ie by name).
	      For example, in a scenario hosts \code{H-01} to \code{H-20} are stopped first, and then \code{H-20} to \code{H-40} are stopped next, \etc
	\item \textbf{\code{RandomVictimSelector}}\\
	      This is the default \code{OpenDC} victim selection algorithm.
	      It randomly chooses which host to stop when injecting failures.
	      This component is left unmodified, but we include it here for the purposes of listing all the failure injection algorithms we use together.
\end{enumerate}
\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/scheduling_opendc.png}
	\caption[OpenDC scheduling paradigm.]{The scheduling paradigm in \code{OpenDC}. Adapted from Musc{\u a} \etal~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. The highlighted \textcolor{Orchid}{\ding{110} \textbf{Filter Scheduler}} is the component that is substituted by the \code{SmartScheduler}.}
	\label{fig:scheduling_in_opendc}
\end{figure}

\section{Python Modules}\label{ss:programming}

\code{Analytics Engine}, \code{HTTPServer}, \code{MonitoringService} are Python modules we prototyped for the reference architecture evaluation.
It is important to note that we do not evaluate the performance of \gls{my_system} due to the substantial overheads of the Python interpreter.
\code{Kafka}, \code{Redis} and \code{PostgreSQL} enable milisecond-latency and huge throughput within the system, however what stops us from a performance evaluation is the high cost of Python interpretation.
For future work, we envision a system that implements the reference architecture in a compiled language, \eg \code{C} or \code{C++}.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{AnalyticsEngine}}\\
	      The \code{AnalyticsEngine} module is necessary in order to encapsulate the logic of data preprocessing and analysis from monitoring.
	      This component can contain capabilities for different statistical metrics, subject to \gls{dt}s focus.
	      In \gls{my_system} \code{AnalyticsEngine} continuously checks whether the incoming datacenter sensor readings exceed different thresholds.
	      For example, the \code{AnalyticsEngine} is capable of calculating a similarity score $S$ between potential failure distributions and the true failure distribution.
	\item \textbf{\code{HTTPServer}}\\
	      The \code{HTTPServer} is crucial for interrupting the operation of the datacenter in order to adjust its operation or offer insights.
	      It maintains a python \code{Queue} structure.
	      The \code{Queue} \emph{producer} is the \code{AnalyticsEngine} (\grayCircled{4}).
	      The \emph{consumer} is the \code{HTTPClient} within \code{OpenDC} (\ie the real datacenter, (\myCircled{2}), (\myCircled{1})).
	      Both the consumer and producer interact with the \code{Queue} via the network and different HTTP endpoints of \code{HTTPServer}.
	\item \textbf{\code{MonitoringService}}\\
	      The \code{MonitoringService} is responsible for interacting with the \code{Redis} key value store.
	      It contains a \code{while True} Python loop which contains the function call to fetch the latest changes to the \code{Redis} stream.
	      Upon update, the \code{MonitoringService} informs the \code{AnalyticsEngine} that new data is awaiting \code{AnalyticsEngine} (\grayCircled{6}).
\end{enumerate}

