\chapter{Implementation}\label{s:implementation}
In this chapter we describe the implementation of \gls{my_system}.
The main contribution of this chapter towards answering \emph{RQ3} is the prototype of \gls{my_system}.
After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}.

Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
To understand why \gls{my_system} it is crucial to provide a holistic view on the prototype.
Therefore, the rest of the chapter is structured in a top-down approach: \Cref{ss:implementation_overview}
presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}.
Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules.

\section{Overview}\label{ss:implementation_overview}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{images/implementation.pdf}
	\caption{The prototype and its components based on the architecture.
		The time-series data flows first to the \texttt{Grafana} (\myCircled{2a}) dashboard, \texttt{PostgreSQL} (\myCircled{3a}) database and \texttt{Redis} (\myCircled{3b}) cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.}
	\label{fig:implementation}
\end{figure}

At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.

\code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards.
We posit it is crucial to include a user-friendly \gls{ui}.
A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
\code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
Good alternatives to \code{Grafana} (\myCircled{2a}) are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}.

\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time.
We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ).
Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}.
The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval.
With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a})  tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema.
Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf} (see \Cref{lst:protobuf-schema}).

\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}.
In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data.
\code{Redis} (\myCircled{3b}) is the industry leader in key value caching.
The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}).

\code{PostgreSQL} (\myCircled{3a}) is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter.
The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}.
Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}.
Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}).
The many alternatives to \code{PostgreSQL}  (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}.
An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}.

Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning).
For a specific overview of advantages of \code{OpenDC}  (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/flow_diagram.png}
	\caption{The data flow within \gls{my_system}.}
	\label{fig:flow_diagram}
\end{figure}

\section{Data Flow}\label{ss:data_flow}

In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
Efficient data flow is of utmost importance to \gls{dt}s.
In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
We describe the flow of data through a timeline.

First, the datacenter informs the \gls{dt} of an upcoming workload (\grayCircled{2}).
This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
The \gls{dt} stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
The analytics engine queries the \code{OpenDC}
simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
This data for the purposes of the prototype is stored in the \code{.parquet} files.
In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic.

In the meantime, the datacenter executes the workload.
The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}).
As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}).
At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}).
Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}).
This provides real-time feedback to datacenter operators (\grayCircled{8}).

Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}).
Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}).
Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights.
All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}).


\begin{figure}[t]
	\input{sources/listing_schema.tex}
\end{figure}

Of significant importance are data flows (\myCircled{2}) and (\myCircled{3}).
Due to the massive volume of data incoming from the physical datacenter, the \code{Redis} sink (\myCircled{2}) has been configured to filter out relevant packets.
\code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}).
On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}).
This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out.

\begin{figure}[t]
	\input{sources/listing_sinks.tex}
\end{figure}
\section{Extensions to \code{OpenDC}}\label{ss:extensions}

\code{OpenDC} is a state-of-the-art datacenter simulator.
In order to turn it into a \gls{dt}, we have made several design decisions and extensions.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{SmartScheduler}}\\
	      The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions.
	      It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights.
	      For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}).
	\item \textbf{\code{KafkaMonitor}}\\
	      The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}).
	      We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic.
	      The \code{KafkaMonitor} class implements the generic \code{ComputeMonitor} interface, therefore the new extension follows the other metric exporting options in \code{OpenDC} (\ie follows the implementation for exporting  metrics into \code{.parquet files}).
	\item \textbf{\code{HTTPClient}}\\
	      The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter.
	      We decided to use the HTTP protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice.
\end{enumerate}

\section{Python Modules}\label{ss:programming}

\code{Analytics Engine}, \code{HTTPServer}, \code{MonitoringService} are Python modules we prototyped for the reference architecture evaluation.
It is important to note that we do not evaluate the performance of \gls{my_system} due to the substantial overheads of the Python interpreter.
\code{Kafka}, \code{Redis} and \code{PostgreSQL} enable milisecond-latency and huge throughput within the system, however what stops us from a performance evaluation is the high cost of Python interpretation.
For future work, we envision a system that implements the reference architecture in a compiled language, \eg \code{C} or \code{C++}.

\begin{enumerate}[label=\textbf{\arabic*.}]
	\item \textbf{\code{AnalyticsEngine}}\\
	\item \textbf{\code{HTTPServer}}\\
	\item \textbf{\code{MonitoringService}}\\
\end{enumerate}