1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
\chapter{Implementation}\label{s:implementation}
In this chapter we describe the implementation of \gls{my_system}.
The main contribution of this chapter towards answering \emph{RQ3} is the prototype of \gls{my_system}.
After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}.
Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}.
To understand why \gls{my_system} it is crucial to provide a holistic view on the prototype.
Therefore, the rest of the chapter is structured in a top-down approach: \Cref{ss:implementation_overview}
presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}.
Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules.
\section{Overview}\label{ss:implementation_overview}
\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{images/implementation.pdf}
\caption{The prototype and its components based on the architecture.
The time-series data flows first to the \texttt{Grafana} (\myCircled{2a}) dashboard, \texttt{PostgreSQL} (\myCircled{3a}) database and \texttt{Redis} (\myCircled{3b}) cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.}
\label{fig:implementation}
\end{figure}
At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications.
The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}.
In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}.
To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}.
To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}.
To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}.
\code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards.
We posit it is crucial to include a user-friendly \gls{ui}.
A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}.
We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}).
\code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself.
Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}.
Good alternatives to \code{Grafana} (\myCircled{2a}) are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}.
\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time.
We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ).
Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}.
The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval.
With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a}) tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema.
Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf} (see \Cref{lst:protobuf-schema}).
\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}.
In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data.
\code{Redis} (\myCircled{3b}) is the industry leader in key value caching.
The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}).
\code{PostgreSQL} (\myCircled{3a}) is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter.
The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}.
Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}.
Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}).
The many alternatives to \code{PostgreSQL} (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}.
An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}.
Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}.
\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning).
For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/flow_diagram.png}
\caption{The data flow within \gls{my_system}.}
\label{fig:flow_diagram}
\end{figure}
\section{Data Flow}\label{ss:data_flow}
In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram.
Efficient data flow is of utmost importance to \gls{dt}s.
In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}.
In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data.
Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}).
We describe the flow of data through a timeline.
First, the datacenter informs the \gls{dt} of an upcoming workload (\grayCircled{2}).
This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains).
The \gls{dt} stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}).
The analytics engine queries the \code{OpenDC}
simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}).
\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}).
This data for the purposes of the prototype is stored in the \code{.parquet} files.
In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic.
In the meantime, the datacenter executes the workload.
The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}).
As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}).
At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}).
Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}).
This provides real-time feedback to datacenter operators (\grayCircled{8}).
Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}).
Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}).
Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights.
All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}).
\begin{figure}[t]
\input{sources/listing_schema.tex}
\end{figure}
Of significant importance are data flows (\myCircled{2}) and (\myCircled{3}).
Due to the massive volume of data incoming from the physical datacenter, the \code{Redis} sink (\myCircled{2}) has been configured to filter out relevant packets.
\code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}).
On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}).
This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out.
\begin{figure}[t]
\input{sources/listing_sinks.tex}
\end{figure}
\section{Extensions to \code{OpenDC}}\label{ss:extensions}
\code{OpenDC} is a state-of-the-art datacenter simulator.
In order to turn it into a \gls{dt}, we have made several design decisions and extensions.
\begin{enumerate}[label=\textbf{\arabic*.}]
\item \textbf{\code{SmartScheduler}}\\
The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions.
It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights.
For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}).
\item \textbf{\code{KafkaMonitor}}\\
The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}).
We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic.
The \code{KafkaMonitor} class implements the generic \code{ComputeMonitor} interface, therefore the new extension follows the other metric exporting options in \code{OpenDC} (\ie follows the implementation for exporting metrics into \code{.parquet files}).
\item \textbf{\code{HTTPClient}}\\
The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter.
We decided to use the HTTP protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice.
\end{enumerate}
\section{Python Modules}\label{ss:programming}
\code{Analytics Engine}, \code{HTTPServer}, \code{MonitoringService} are Python modules we prototyped for the reference architecture evaluation.
It is important to note that we do not evaluate the performance of \gls{my_system} due to the substantial overheads of the Python interpreter.
\code{Kafka}, \code{Redis} and \code{PostgreSQL} enable milisecond-latency and huge throughput within the system, however what stops us from a performance evaluation is the high cost of Python interpretation.
For future work, we envision a system that implements the reference architecture in a compiled language, \eg \code{C} or \code{C++}.
\begin{enumerate}[label=\textbf{\arabic*.}]
\item \textbf{\code{AnalyticsEngine}}\\
\item \textbf{\code{HTTPServer}}\\
\item \textbf{\code{MonitoringService}}\\
\end{enumerate}
|
