\chapter{Implementation}\label{s:implementation}
In this chapter we describe the implementation of \gls{my_system}.
The main contribution of this chapter towards answering \emph{RQ3} is the prototype of \gls{my_system}.
After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}.

Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
To understand why \gls{my_system} it is crucial to provide a holistic view on the prototype.
Therefore, the rest of the chapter is structured in a top-down approach: \Cref{ss:implementation_overview}
presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:programming} details the different modifications and new software extensions.

\section{Overview}\label{ss:implementation_overview}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{images/implementation.pdf}
	\caption{The prototype and its components based on the architecture.
		The time-series data flows first to the \texttt{Grafana} (\myCircled{2a}) dashboard, \texttt{PostgreSQL} (\myCircled{3a}) database and \texttt{Redis} (\myCircled{3b}) cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.}
	\label{fig:implementation}
\end{figure}

At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
To enable the flow of data into the digital twin, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.

\code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards.
We posit it is crucial to include a user-friendly \gls{ui}.
A number of previous publications on digital twinning find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
\code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
Good alternatives to \code{Grafana} (\myCircled{2a}) are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}.

\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time.
We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ).
Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}.
The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval.
With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a})  tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema.
Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf} (see \Cref{lst:protobuf-schema}).

\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}.
In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data.
\code{Redis} (\myCircled{3b}) is the industry leader in key value caching.
The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}).

\code{PostgreSQL} (\myCircled{3a}) is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter.
The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}.
Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}.
Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}).
The many alternatives to \code{PostgreSQL}  (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}.
An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}.

Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning).
For a specific overview of advantages of \code{OpenDC}  (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/flow_diagram.png}
	\caption{The data flow within \gls{my_system}.}
	\label{fig:flow_diagram}
\end{figure}

\section{Data Flow}\label{ss:data_flow}

In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
Efficient data flow is of utmost importance to digital twinning.
In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
We describe the flow of data through a timeline.

First, the datacenter informs the digital twin of an upcoming workload (\grayCircled{2}).
This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
The digital twin stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
The analytics engine queries the \code{OpenDC}
simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
This data for the purposes of the prototype is stored in the \code{.parquet} files.
In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic.

In the meantime, the datacenter executes the workload.
The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}).
As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}).
At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}).
Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}).
This provides real-time feedback to datacenter operators (\grayCircled{8}).

Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}).
Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}).
Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights.
All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}).

\begin{figure}[t]
	\input{sources/listing_schema.tex}
\end{figure}

\input{sources/listing_sinks.tex}
\section{Programming Effort}\label{ss:programming}