1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
|
% Changed chapter name, as suggested by Alexandru
\chapter{Design of \mysystem, a Digital Twin For Predictive Analysis of Datacenters}\label{s:design}
\section{Overview}\label{ss:design_overview}
\begin{mynote}
Our contribution in this chapter is three-fold:
\vspace{-0.2cm}
\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt]
\item We analyze the requirements for \mysystem (\Cref{ss:requirements_analysis}).
\item We propose a conceptual design for \mysystem's architecture (\Cref{ss:design_of_mysystem})
\item We describe how \mysystem enables predictive analytics through digital twinning (\Cref{ss:design_discussion})
\end{enumerate}
\end{mynote}
\section{Requirements Analysis}\label{ss:requirements_analysis}
In this section we determine the requirements that should be fulfilled by \mysystem.
We present here the stakeholders identified by our literature survey (see \Cref{sss:digital_twins_for_datacenters}) and the relevant use-cases.
Afterwards, we list the functional and non-functional requirements for \mysystem.
\subsection{Stakeholders}\label{sss:stakeholders}
We identify four main stakeholders of a predictive datacenter digital twin:
\begin{enumerate}[label=\textbf{S\arabic* --},align=left]
\item \textbf{Datacenter Managers}\\
Responsible for maintenance and operation of the warehouse, operators manage the datacenter daily.
They interact with the servers, bring downed hosts up and ensure customers' services run smoothly at all times.
Datacenter operators need to ensure different \gls{sla}s are met, energy costs are balanced and carbon emission quota is maintained.
\item \textbf{Datacenter Technicians}\\
The term datacenter engineers encompasses datacenter architects and technicians alike.
From the moment the datacenter layout is determined, to the physical process of booting the server racks for the first time, datacenter engineers help build and maintain the datacenter.
They must continuously adapt to changing requirements and ensure everything goes smoothly~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
\item \textbf{Scientists and Academia}\\
Digital twinning generates unprecedented amount of data.
To bring valuable insights, the ingested metrics must be effectively analyzed.
Scientists can draw conclusions from the monitoring data and in some deployments already benefit from the voluminous output of \gls{dcdt}s~\cite{DBLP:conf/noms/ZhangZLZWC22}.
\item \textbf{Customers and Users}\\
Digital twins are already used to visualize both facilities and human beings alike for the benefit of users and customers.
Cloud and HPC users are not directly interacting with the system, but pay for services hosted in the datacenter and are one of the primary stakeholders of digital twinning.
Not only through 3D datacenter visualizations, but also from the continuously generated metrics can users and customers benefit from \gls{dcdt}s.
\end{enumerate}
\subsection{Use-cases}\label{sss:use_cases}
Based on the identified stakeholders we list 6 potential use-cases for a predictive datacenter digital twin:
\begin{enumerate}[label=\textbf{UC\arabic* --},align=left]
\item \textbf{Energy Optimization} \\
Predicting and modeling energy optimization is curcial for \gls{dcdt}s.
It is the main use-case of some existing systems~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24}.
Effective energy optimization, including the adjustments to the consumed energy type is a crucial use-case of \gls{dcdt}s.
\item \textbf{Failure Management} \\
Predictive maintenance of both hardware and software failures alike, by simulating the possible failure distribution of a running workload, can lower downtime, terminated tasks and ensure \gls{sla}s are not missed~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
\item \textbf{Heat Modelling} \\
Heat modeling is the primary use case of existing \gls{dcdt}s (see \Cref{tab:dt_features_comparison}).
Correct thermal management of the warehouse can optimize cooling strategies, leading to lower bills and maintenance costs.
\item \textbf{Network Traffic Modelling} \\
Congestion management and traffic routing can effectively benefit from digital twinning.
Detecting bottlenecks, adjusting datacenter protocols and calibrating switches and interconnects is already a important use-case for \gls{dcdt}~\cite{DBLP:conf/sigcomm/HongWDSSHZY21}.
\item \textbf{Virtual Prototyping} \\
Digital twinning can be used to provide insight into the system before changes are made.
Using a \gls{dcdt}, the operators can change, model and shape the datacenter to estimate their effect.
Virtual prototyping encompasses interacting with an existing datacenter model, or with a proof-of-concept, not yet constructed warehouse.
\item \textbf{Monitoring and Visualization} \\
3D visualizations and dashboards are of the utmost importance to all stakeholders, due to the insights they provide.
With real-time data ingestion and the two-way feedback loop, \gls{dcdt}s can empower descriptive analytics.
This use-case already shapes many existing systems (see \Cref{tab:dt_features_comparison}).
\end{enumerate}
\subsection{Functional Requirements}\label{sss:functional_requirements}
Based on a subset of the above use-cases, we formulate the functional and non-functional requirements for \mysystem:
\begin{enumerate}[label=\textbf{FR\arabic* --},align=left]
\item \textbf{The system should be able to handle workloads of arbitrary size.} \\
Existing systems range from Cloud through the Edge to HPC digital twins.
Therefore, \mysystem must support workloads similar in length and type to the commercial setting.
Without \textbf{FR1}, \mysystem will be incomplete, and like the majority of the \Cref{tab:dt_features_comparison} systems, its use-case will be niche.
\textbf{FR1} is necessary to avoid overly-specializing the \gls{dcdt}.
\item \textbf{The system should support failure detection.}\\
Failures are of crucial concern to datacenter operators.
The system detect and report failures to datacenter operators.
Without \textbf{FR2}, the system will not be able to help datacenter operators meet \gls{sla}s.
Including \textbf{FR2} is necessary to ensure that failures, which which can cause financial penalties, are detected in a timely manner so as to meet the different \gls{sla}s.
\item \textbf{The system should be capable of long-term and short-term data storage.}\\
\textbf{FR3} is necessary for \textbf{FR4} and \textbf{FR5}.
Without \textbf{FR3}, the system cannot support \gls{oda} techniques.
A system cannot be considered a \gls{dt} without insights stemming from accurate data analytics~\cite{DBLP:usdoe/report/AP26894}.
\gls{nasem} digital twin definition requires both real-time insights, and guidance based on historical-patterns.
Therefore, it is imperative to ensure \textbf{FR3}.
\item \textbf{The system should support descriptive data analytics.}\\
There are many types of data analytics present is existing deployments~\cite{DBLP:conf/wosp/SumanCNTMI24}.
In order to scope down the project, we only select several use-cases for \gls{my_system}.
Out of the 4 prime analytics type, we find in the literature survey the predictive analytics to be the most urgently needed, and descriptive analytics to be the most prevailing type of data processing.
Therefore, in order to achieve performance on par with previous systems, and to ensure our system meets the official \gls{nasem} definition, \textbf{FR4} is needed.
\item \textbf{The system should enable predictive data analytics.}\\
The system should help future researchers incorporate predictive data analytics engines, regardless of the statistical modelling technique.
This is our novel contribution to the scientific field of \gls{dcdt}s.
Without \textbf{FR5}, we miss the aim of our entire project.
\item \textbf{The system should be able to process arbitrary amounts of telemetry.}\\
The size of the \gls{ai} economy is expected to grow~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
As a result, both current and future datacenters will generate huge amount of data.
Our system must be capable of ingesting and processing different metrics regardless of the volume of incoming messages.
Without \textbf{FR6}, we hinder the adoption of \mysystem in modern and future datacenters.
\item \textbf{The model should be capable of clear data visualization.}
The system must have a user-friendly, visual interface for data analytics, and support them in real-time.
Without \textbf{FR7}, we exclude important stakeholders such as customers and users from the potential benefactors of our system.
\end{enumerate}
\subsection{Non-functional Requirements}\label{sss:non_functional_requirements}
In addition to the functional requirements, we also present non-functional requirements for \mysystem:
\begin{enumerate}[label=\textbf{NFR\arabic* --},align=left]
\item \textbf{The system should enable real-time insights and visualizations.} \\
The system must work in real-time, without significant delay.
The system must support datacenter operators with insights at fine-grained granularity, so that insights derived from data analysis remain accurate upon reception by datacenter operators.
Without \textbf{NFR1}, \mysystem's insights will not be timely, and will be useless to datacenter operators.
\item \textbf{The system should log the ingestion and processing of metrics.} \\
The system should provide a log of the current network traffic to and from the digital twin.
\item \textbf{The system should adhere to modern software development standards.} \\
The system should follow modern coding principles and guidelines to ensure reproducibility and usefulness for future work.
Without \textbf{NFR3}, it will be difficult for current and future digital twin developers to include predictive analytics using \mysystem in their deployments.
\item \textbf{The system should provide insights at varying levels of confidence.} \\
With the huge amount of telemetry data incoming from the physical twin, the digital counterpart must be able to filter and pre-process the telemetry.
Without \textbf{NFR4}, the system will present overwhelming amount of information to its users, rendering it unusable.
\end{enumerate}
\begin{figure}[ht]
\centering
\includegraphics[width=0.75\linewidth]{images/ref_architecture.png}
\caption[The predictive datacenter digital twin architecture.]{The predictive datacenter digital twin reference architecture.
We call the system \emph{Sunfish}.
The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months.}
\label{fig:reference_architecture}
\end{figure}
\section{Overview of \mysystem Architecture}\label{ss:design_of_mysystem}
As a result of the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} designed a reference architecture for a predictive datacenter digital twin.
\Cref{fig:reference_architecture} encompasses 4 main elements:
\begin{enumerate*}[label=(\Roman*)]
\item physical datacenter
\item digital thread
\item digital twin
\item predictive analytics.
\end{enumerate*}
The physical datacenter (I) encompasses 3 core elements important to digital twinning.
Workloads (\myCircled{1a}) include the hardware requirements of each datacenter job and the submission time.
They are executed on the datacenter compute (\myCircled{1b}), which is controlled partly by the Datacenter Operators (\myCircled{1c}).
Component (\myCircled{1c}), while seemingly unimportant, is crucial to the digital twin design.
We envision \gls{dcdt}s as systems that contain a human-in-the-loop, which can control and overwrite the system's autonomous decisions.
Datacenter Operators (\myCircled{1c}) interact with both the Servers (\myCircled{1b}), and have the ability to overwrite the potential autonomous digital twin decisions, stemming from component (\myCircled{2c}), the System Knobs.
The Digital Thread (II) is a novel contribution from \Cref{s:background}.
It separates the physical world from the virtual world, and contains components that do not belong to either twin, or belong to both twins.
It contains the Interactive Dashboard (\myCircled{2a}), the Message Broker (\myCircled{2b}), and System Knobs (\myCircled{2c}).
To fulfill the functional requirements of our system, we incorporate element (\myCircled{2a}), the Interactive Dashboard to our system.
This dashboard ingests data coming directly from the Message Broker (\myCircled{2b}) and from the long-term storage (\myCircled{3a}), the Database.
The Interactive Dashboard (\myCircled{2a}) allows datacenter operators to see the telemetry data arrive to the digital twin in real time.
The Message Broker (\myCircled{2b}) is a crucial component to the reference architecture, because it facilities the physical twin $\rightarrow$ virtual twin connection.
A low-latency, high-throughput message broker partly meets our functional requirements to enable arbitrary amounts of telemetry data transfer.
We elaborate on the specific components that make up the message broker (\myCircled{2b}) in \Cref{sss:message_broker}.
The System Knobs (\myCircled{2c}) represent the different cogs within the datacenter software and hardware (\myCircled{1b}) that can be adjusted during runtime (\eg to optimize \gls{pue}, change cooling strategy, allocate compute resources).
For example, System Knobs (\myCircled{2c}) within the datacenter scheduler can be tuned to schedule jobs on Servers (\myCircled{1b}) that are least likely to experience future downtime.
The autonomous actions of the digital twin (the tuning of the System Knobs (\myCircled{2c})) can be further adjusted by Datacenter Operators (\myCircled{1c}).
In our design, we explicitly differentiate between the physical and virtual space by including the Digital Twin (III) in a separate box.
The Digital Twin (III) constitutes of the long-term storage (\myCircled{3a}), short-term storage (\myCircled{3b}), the API Server (\myCircled{3c}) and the Predictive Analytics (IV) module.
Long-term storage, namely a Database (\myCircled{3a}) is crucial to enable real-time visualizations (\myCircled{2a}), and to model long-term system behaviour.
This fulfills the functional requirements for our system, as we set out to differentiate between in-band data analytics, and out-of-band data analysis.
The data is consumed by the Database (\myCircled{3a}) directly from the Message Broker (\myCircled{2b}).
In-between, simple data pre-processing can take place.
Short-term storage, in the form of a Cache (\myCircled{3b}) is essential for in-band data analytics, ``on-the-go''.
The Cache (\myCircled{3b}) retains the data consumed from the Message Broker (\myCircled{2b}), and for a short period of time keeps a copy.
The data analytics performed on the Database (\myCircled{3a}) dataset and the Cache (\myCircled{3b}) adhere to the use-cases for predictive (long-term) and descriptive (long-term, short-term) data analytics.
The Message Broker (\myCircled{2b}) enables one-way connection (physical twin $\rightarrow$ digital twin).
The API Server (\myCircled{3c}) enables the digital twin $\rightarrow$ link.
The API Server (\myCircled{3c}) communicates directly with the System Knobs (\myCircled{2c}) in order to take meaningful action, based on the (predictive) insights generated by the Digital Twin (III).
Additionally, the Physical Twin (III) can query the API Server (\myCircled{3c}) for one-shot requests (\eg to create a new datacenter prototype configuration, to request special data analysis).
Moreover, the Datacenter Operators (\myCircled{1c}) can query the API Server (\myCircled{3c}) for extra insights, when necessary.
The Predictive Analytics (IV) module is an extensible part of the reference architecture, enabling different kinds of predictive analysis.
In our design, to facilitate meaningful predictions we incorporate an Event-driven Simulator (\myCircled{4a}), Analytics Engine (\myCircled{4b}), and a Monitoring Service (\myCircled{4c}).
We chose an Event-driven Simulator (\myCircled{4a}) as a core element of predictive analytics.
For the rationale behind this decision see \Cref{ss:datacenters}.
The predictions from the Event-driven Simulator (\myCircled{4a}) communicate directly with the Database (\myCircled{3a}) to store the simulation results, and with the Analytics Engine (\myCircled{4b}).
The Analytics Engine (\myCircled{4b}) performs descriptive and predictive data analytics on the results from the Event-driven Simulator (\myCircled{4a}) and the telemetry data collected from the physical datacenter (stored in (\myCircled{3a}, \myCircled{3b})).
This component is highly extensible to accommodate the different types of analytics (\eg \gls{ml}, \gls{ai}, statistical methods).
The results of the data analysis are propagated to the API Server (\myCircled{3c}) to be queued for retrieval by the physical twin.
The Monitoring Service (\myCircled{4c}) serves as a separate thread to watch over the collected telemetry in real-time.
It facilities detection of irregularities in collected data, which adheres to the functional requirements set out for the system.
Any discrepancies are communicated to the Analytics Engine (\myCircled{4b}) for further analysis, and potential insights.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/message_broker.png}
\caption[The detailed view of the Message Broker.]{The detailed view of the Message Broker (\myCircled{2b}) from \Cref{fig:reference_architecture}.}
\label{fig:message_broker}
\end{figure}
\section{The Digital Thread and Predictive Analytics}\label{ss:detailed_design}
\subsection{Message Broker}\label{sss:message_broker}
The Message Broker (\myCircled{2b}) is a component is slightly more complex, and necessities a separate diagram.
In \Cref{fig:message_broker} we present the composite elements that make up the Message Broker.
In particular, the \emph{schema registry} and the \emph{connector manager} play a crucial part in fulfilling the functional requirements of our system.
The \emph{schema registry} allows the telemetry producer to submit \emph{any} data format for sending (and storing) the telemetry data.The registry is responsible for detecting what kind of format is the data sent in, and automatically adjusting the schema within the \emph{data pipeline}.
The connector manager is responsible for joining multiple distinct services to a single data pipeline.
In the reference architecture, the consumers would constitute the Database (\myCircled{3a}), the Cache (\myCircled{3b}), and the Interactive Dashboard (\myCircled{2a}).
What is remarkable about the connector manager is the ability to swiftly connect more consumers to the system.
This way, the predictive digital twin can facilitate multiple different types of analytics engines or techniques.
Additionally, the setup is \Cref{fig:message_broker} is currently standard industry practice for large software deployments.
\section{Discussion}\label{ss:design_discussion}
|
