From c7ea4c55a58e832330c951de9d6104413b59cada Mon Sep 17 00:00:00 2001
From: mjkwiatkowski <mati.rewa@gmail.com>
Date: Sat, 4 Jul 2026 15:00:08 +0200
Subject: feat: finished section 4.3

---
 content/implementation.tex | 46 +++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 39 insertions(+), 7 deletions(-)

(limited to 'content/implementation.tex')

diff --git a/content/implementation.tex b/content/implementation.tex
index ae100bd..2395963 100644
--- a/content/implementation.tex
+++ b/content/implementation.tex
@@ -6,7 +6,8 @@ After reading one should understand the technical decisions, choice of tools and
 Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
 To understand why \gls{my_system} it is crucial to provide a holistic view on the prototype.
 Therefore, the rest of the chapter is structured in a top-down approach: \Cref{ss:implementation_overview}
-presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:programming} details the different modifications and new software extensions.
+presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}.
+Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules.
 
 \section{Overview}\label{ss:implementation_overview}
 
@@ -21,14 +22,14 @@ presents the rationale for using the specific software packages, \Cref{ss:data_f
 At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
 The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
 In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
-To enable the flow of data into the digital twin, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
+To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
 To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
 To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
 The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.
 
 \code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards.
 We posit it is crucial to include a user-friendly \gls{ui}.
-A number of previous publications on digital twinning find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
+A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
 We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
 \code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
 Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
@@ -67,15 +68,15 @@ For a specific overview of advantages of \code{OpenDC}  (\myCircled{4a}) and a t
 \section{Data Flow}\label{ss:data_flow}
 
 In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
-Efficient data flow is of utmost importance to digital twinning.
+Efficient data flow is of utmost importance to \gls{dt}s.
 In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
 In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
 Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
 We describe the flow of data through a timeline.
 
-First, the datacenter informs the digital twin of an upcoming workload (\grayCircled{2}).
+First, the datacenter informs the \gls{dt} of an upcoming workload (\grayCircled{2}).
 This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
-The digital twin stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
+The \gls{dt} stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
 The analytics engine queries the \code{OpenDC}
 simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
 \code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
@@ -108,4 +109,35 @@ This setup achieves excellent abstraction level, because only the most important
 \begin{figure}[t]
 	\input{sources/listing_sinks.tex}
 \end{figure}
-\section{Programming Effort}\label{ss:programming}
+\section{Extensions to \code{OpenDC}}\label{ss:extensions}
+
+\code{OpenDC} is a state-of-the-art datacenter simulator.
+In order to turn it into a \gls{dt}, we have made several design decisions and extensions.
+
+\begin{enumerate}[label=\textbf{\arabic*.}]
+	\item \textbf{\code{SmartScheduler}}\\
+	      The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions.
+	      It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights.
+	      For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}).
+	\item \textbf{\code{KafkaMonitor}}\\
+	      The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}).
+	      We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic.
+	      The \code{KafkaMonitor} class implements the generic \code{ComputeMonitor} interface, therefore the new extension follows the other metric exporting options in \code{OpenDC} (\ie follows the implementation for exporting  metrics into \code{.parquet files}).
+	\item \textbf{\code{HTTPClient}}\\
+	      The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter.
+	      We decided to use the HTTP protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice.
+\end{enumerate}
+
+\section{Python Modules}\label{ss:programming}
+
+\code{Analytics Engine}, \code{HTTPServer}, \code{MonitoringService} are Python modules we prototyped for the reference architecture evaluation.
+It is important to note that we do not evaluate the performance of \gls{my_system} due to the substantial overheads of the Python interpreter.
+\code{Kafka}, \code{Redis} and \code{PostgreSQL} enable milisecond-latency and huge throughput within the system, however what stops us from a performance evaluation is the high cost of Python interpretation.
+For future work, we envision a system that implements the reference architecture in a compiled language, \eg \code{C} or \code{C++}.
+
+\begin{enumerate}[label=\textbf{\arabic*.}]
+	\item \textbf{\code{AnalyticsEngine}}\\
+	\item \textbf{\code{HTTPServer}}\\
+	\item \textbf{\code{MonitoringService}}\\
+\end{enumerate}
+
-- 
cgit v1.2.3

