% Changed chapter name, as suggested by Alexandru
\chapter{Design of \mysystem, a Digital Twin For Predictive Analysis of Datacenters}\label{s:design}
\section{Overview}\label{ss:design_overview}
\begin{mynote}
	Our contribution in this chapter is three-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt]
		\item We analyze the requirements for \mysystem (\Cref{ss:requirements_analysis}).
		\item We propose a conceptual design for \mysystem's architecture (\Cref{ss:design_of_mysystem})
		\item We describe how \mysystem fulfills the functional and non-functional requirements in \Cref{ss:requirement_validation}.
	\end{enumerate}
\end{mynote}
\section{Requirements Analysis}\label{ss:requirements_analysis}
In this section we determine the requirements that should be fulfilled by \mysystem.
We present here the stakeholders identified by our literature survey (see \Cref{ss:digital_twins_for_datacenters}) and the relevant use-cases.
Afterwards, we list the functional and non-functional requirements for \mysystem.

\subsection{Stakeholders}\label{sss:stakeholders}
We identify four main stakeholders of a predictive datacenter digital twin:
\begin{enumerate}[label=\textbf{S\arabic* --},align=left]
	\item \textbf{Datacenter Managers}\\
	      Responsible for maintenance and operation of the warehouse, operators manage the datacenter daily.
	      They interact with the servers, bring downed hosts up and ensure customers' services run smoothly at all times.
	      Importantly, datacenter operators follow a strict set of policies for taking different action.
	      Datacenter operators need to ensure different \gls{sla}s are met, energy costs are balanced and carbon emission quota is maintained.
	\item \textbf{Datacenter Engineers}\\
	      The term datacenter engineers encompasses datacenter architects and technicians alike.
	      From the moment datacenter architects determine the warehouse layout, to the physical process of booting the server racks for the first time by the technicians, datacenter engineers help build and maintain the datacenter.
	      They must continuously adapt to changing requirements and ensure everything goes smoothly~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
	\item \textbf{Scientists and Academia}\\
	      Digital twinning generates unprecedented amount of data.
	      To bring valuable insights, the ingested metrics must be effectively analyzed.
	      Scientists can draw conclusions from the monitoring data and in some deployments already benefit from the voluminous output of \gls{dcdt}s~\cite{DBLP:conf/noms/ZhangZLZWC22}.
	\item \textbf{Customers and Users}\\
	      Digital twins are already used to visualize both facilities and human beings alike for the benefit of users and customers.
	      Cloud and HPC users are not directly interacting with the system, but pay for services hosted in the datacenter and are one of the primary stakeholders of digital twinning.
	      Not only through 3D datacenter visualizations, but also from the continuously generated metrics can users and customers benefit from \gls{dcdt}s.
\end{enumerate}

\subsection{Use-cases}\label{sss:use_cases}
Based on the identified stakeholders we list 6 potential use-cases for a predictive datacenter digital twin:
\begin{enumerate}[label=\textbf{UC\arabic* --},align=left]
	\item \textbf{Energy Optimization} \\
	      Predicting and modeling energy optimization is curcial for \gls{dcdt}s.
	      It is the main use-case of some existing systems~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24}.
	      Effective energy optimization, including the adjustments to the consumed energy type is a crucial use-case of \gls{dcdt}s.
	\item \textbf{Failure Management} \\
	      Predictive maintenance of both hardware and software failures alike, by simulating the possible failure distribution of a running workload, can lower downtime, terminated tasks and ensure \gls{sla}s are not missed~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
	\item \textbf{Heat Modelling} \\
	      Heat modeling is the primary use case of existing \gls{dcdt}s (see \Cref{tab:dt_features_comparison}).
	      Correct thermal management of the warehouse can optimize cooling strategies, leading to lower bills and maintenance costs.
	\item \textbf{Network Traffic Modelling} \\
	      Congestion management and traffic routing can effectively benefit from digital twinning.
	      Detecting bottlenecks, adjusting datacenter protocols and calibrating switches and interconnects is already a important use-case for \gls{dcdt}~\cite{DBLP:conf/sigcomm/HongWDSSHZY21}.
	\item \textbf{Virtual Prototyping} \\
	      Digital twinning can be used to provide insight into the system before changes are made.
	      Using a \gls{dcdt}, the operators can change, model and shape the datacenter to estimate their effect.
	      Virtual prototyping encompasses interacting with an existing datacenter model, or with a proof-of-concept, not yet constructed warehouse.
	\item \textbf{Monitoring and Visualization} \\
	      3D visualizations and dashboards are of the utmost importance to all stakeholders, due to the insights they provide.
	      With real-time data ingestion and the two-way feedback loop, \gls{dcdt}s can empower descriptive analytics.
	      This use-case already shapes many existing systems (see \Cref{tab:dt_features_comparison}).
\end{enumerate}

\subsection{Functional Requirements}\label{sss:functional_requirements}
Based on a subset of the above use-cases, we formulate the functional and non-functional requirements for \mysystem:
\begin{enumerate}[label=\textbf{FR\arabic* --},align=left]
	\item \textbf{The system should be able to handle workload size representative of its modelling technique.} \\
	      Existing \gls{dt}s range from Cloud through the Edge to HPC.
	      Therefore, \mysystem must not impose any significant limitations on the size or type of the workload it runs.
	      In particular, \mysystem must be able to handle workloads as large as the underlying modelling technique permits.
	      Without \textbf{FR1}, \mysystem will be incomplete, and like the majority of the \Cref{tab:dt_features_comparison} systems, its use-case will be niche.
	      \textbf{FR1} is necessary to avoid overly-specializing the \gls{dcdt}.
	\item \textbf{The system should support failure detection.}\\
	      Failures are of crucial concern to datacenter operators.
	      The system detect and report failures to datacenter operators.
	      Without \textbf{FR2}, the system will not be able to help datacenter operators meet \gls{sla}s.
	      Including \textbf{FR2} is necessary to ensure that failures, which which can cause financial penalties, are detected in a timely manner so as to meet the different \gls{sla}s.
	\item \textbf{The system should be capable of long-term and short-term data storage.}\\
	      \textbf{FR3} is necessary for \textbf{FR4} and \textbf{FR5}.
	      Without \textbf{FR3}, the system cannot support \gls{oda} techniques.
	      A system cannot be considered a \gls{dt} without insights stemming from accurate data analytics~\cite{DBLP:usdoe/report/AP26894}.
	      \gls{nasem} digital twin definition requires both real-time insights, and guidance based on historical-patterns.
	      Therefore, it is imperative to ensure \textbf{FR3}.
	\item \textbf{The system should support descriptive data analytics.}\\
	      There are many types of data analytics present is existing deployments~\cite{DBLP:conf/wosp/SumanCNTMI24}.
	      In order to scope down the project, we only select several use-cases for \gls{my_system}.
	      Out of the 4 prime analytics type, we find in the literature survey the predictive analytics to be the most urgently needed, and descriptive analytics to be the most prevailing type of data processing.
	      Therefore, in order to achieve performance on par with previous systems, and to ensure our system meets the official \gls{nasem} definition, \textbf{FR4} is needed.
	\item \textbf{The system should enable predictive data analytics.}\\
	      The system should help future researchers incorporate predictive data analytics engines, regardless of the statistical modelling technique.
	      This is our novel contribution to the scientific field of \gls{dcdt}s.
	      Without \textbf{FR5}, we miss the aim of our entire project.
	\item \textbf{The system should be able to process arbitrary amounts of telemetry.}\\
	      The size of the \gls{ai} economy is expected to grow~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
	      As a result, both current and future datacenters will generate huge amount of data.
	      Our system must be capable of ingesting and processing different metrics regardless of the volume of incoming messages.
	      Without \textbf{FR6}, we hinder the adoption of \mysystem in modern and future datacenters.
	\item \textbf{The model should be capable of clear data visualization.}
	      The system must have a user-friendly, visual interface for data analytics, and support them in real-time.
	      Without \textbf{FR7}, we exclude important stakeholders such as customers and users from the potential benefactors of our system.
\end{enumerate}

\subsection{Non-functional Requirements}\label{sss:non_functional_requirements}
In addition to the functional requirements, we also present non-functional requirements for \mysystem:
\begin{enumerate}[label=\textbf{NFR\arabic* --},align=left]
	\item \textbf{Using \mysystem should not introduce any delays in visualizations longer than 1 second.}\\
	      The system must work in real-time, without significant delay.
	      We impose 1 second delay as acceptable to datacenter engineers.
	      The system must support datacenter operators with insights at fine-grained granularity, so that insights derived from data analysis remain accurate upon reception by datacenter operators.
	      Without \textbf{NFR1}, \mysystem's insights will not be timely, and will be useless to datacenter operators.
	\item \textbf{The system should log the ingestion and processing of metrics.} \\
	      The system should provide a log of the current network traffic to and from the digital twin.
	\item \textbf{The system should adhere to modern software development standards.} \\
	      The system should follow modern coding principles and guidelines to ensure reproducibility and usefulness for future work.
	      Without \textbf{NFR3}, it will be difficult for current and  future digital twin developers to include predictive analytics using \mysystem in their deployments.
	\item \textbf{The system should provide insights at varying levels of confidence.} \\
	      With the huge amount of telemetry data incoming from the physical twin, the digital counterpart must be able to filter and pre-process the telemetry.
	      Without \textbf{NFR4}, the system will present overwhelming amount of information to its users, rendering it unusable.
\end{enumerate}


\section{Overview of \mysystem Architecture}\label{ss:design_of_mysystem}
As a result of the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} designed a reference architecture for a predictive datacenter digital twin.
\Cref{fig:reference_architecture} encompasses 4 main elements:
\begin{enumerate*}[label=(\Roman*)]
	\item physical datacenter
	\item digital thread
	\item digital twin
	\item predictive analytics.
\end{enumerate*}

\subsection{The Physical Datacenter}\label{sss:physical_datacenter}
The Physical Datacenter (I) encompasses 3 core elements important to digital twinning.
Workloads (\myCircled{1a}) include the hardware requirements of each datacenter job and the submission time.
They are executed on the datacenter compute (\myCircled{1b}), which is controlled partly by the Datacenter Operators (\myCircled{1c}).
Component (\myCircled{1c}), while seemingly unimportant, is crucial to the digital twin design.
We envision \gls{dcdt}s as systems that contain a human-in-the-loop, which can control and overwrite the system's autonomous decisions.
Datacenter Operators (\myCircled{1c}) interact with both the Servers (\myCircled{1b}), and have the ability to overwrite the potential autonomous digital twin decisions, stemming from component (\myCircled{2c}), the System Knobs.

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.75\linewidth]{images/ref_architecture.png}
	\caption[The predictive datacenter digital twin architecture.]{The predictive datacenter digital twin reference architecture.
		We call the system \emph{Sunfish}.
		The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months.}
	\label{fig:reference_architecture}
\end{figure}
\subsection{The Digital Thread}\label{sss:digital_thread}
The Digital Thread (II) is a novel contribution from \Cref{s:background}.
It is a crucial element of our architecture.
It separates the physical world from the virtual world, and contains components that do not belong to either twin, or belong to both twins.
Otherwise, it would be unclear how to model the data flow between the physical and the digital twin (\ie which elements take part in the information exchange).
Conceptually separating the physical and virtual twin makes it easier to comprehend and compartmentalize the design.
It contains the Interactive Dashboard (\myCircled{2a}), the Message Broker (\myCircled{2b}), and System Knobs (\myCircled{2c}).

To fulfill the functional requirements of our system, we incorporate element (\myCircled{2a}), the Interactive Dashboard to our system.
This dashboard ingests data coming directly from the Message Broker (\myCircled{2b}) and from the long-term storage (\myCircled{3a}), the Database.
The Interactive Dashboard (\myCircled{2a}) allows datacenter operators to see the telemetry data arrive to the digital twin in real time.

The Message Broker (\myCircled{2b}) is a crucial component to the reference architecture, because it facilities the physical twin $\rightarrow$ virtual twin connection.
A low-latency, high-throughput message broker partly meets our functional requirements to enable arbitrary amounts of telemetry data transfer.
The Message Broker (\myCircled{2b}) is a component is slightly more complex, and necessities a separate diagram.
In \Cref{fig:message_broker} we present the composite elements that make up the Message Broker.
In particular, the \emph{schema registry} and the \emph{connector manager} play a crucial part in fulfilling the functional requirements of our system.
The \emph{schema registry} allows the telemetry producer to submit \emph{any} data format for sending (and storing) the telemetry data.The registry is responsible for detecting what kind of format is the data sent in, and automatically adjusting the schema within the \emph{data pipeline}.
The connector manager is responsible for joining multiple distinct services to a single data pipeline.
In the reference architecture, the consumers would constitute the Database (\myCircled{3a}), the Cache (\myCircled{3b}), and the Interactive Dashboard (\myCircled{2a}).
What is remarkable about the connector manager is the ability to swiftly connect more consumers to the system.
This way, the predictive digital twin can facilitate multiple different types of analytics engines or techniques.
Additionally, the setup in \Cref{fig:message_broker} is currently standard industry practice for large software deployments.

The System Knobs (\myCircled{2c}) represent the different cogs within the datacenter software and hardware (\myCircled{1b}) that can be adjusted during runtime (\eg to optimize \gls{pue}, change cooling strategy, allocate compute resources).
For example, System Knobs (\myCircled{2c}) within the datacenter scheduler can be tuned to schedule jobs on Servers (\myCircled{1b}) that are least likely to experience future downtime.
The autonomous actions of the digital twin (the tuning of the System Knobs (\myCircled{2c})) can be further adjusted by Datacenter Operators (\myCircled{1c}).

\subsection{The Digital Twin}\label{sss:the_digital_twin_archi}
In our design, we explicitly differentiate between the physical and virtual space by including the Digital Twin (III) in a separate box.
The Digital Twin (III) constitutes of the long-term storage (\myCircled{3a}), short-term storage (\myCircled{3b}), the API Server (\myCircled{3c}) and the Predictive Analytics (IV) module.

Long-term storage, namely a Database (\myCircled{3a}) is crucial to enable real-time visualizations (\myCircled{2a}), and to model long-term system behaviour.
This fulfills the functional requirements for our system, as we set out to differentiate between in-band data analytics, and out-of-band data analysis.
The data is consumed by the Database (\myCircled{3a}) directly from the Message Broker (\myCircled{2b}).
In-between, simple data pre-processing can take place.

Short-term storage, in the form of a Cache (\myCircled{3b}) is essential for in-band data analytics, ``on-the-go''.
The Cache (\myCircled{3b}) retains the data consumed from the Message Broker (\myCircled{2b}), and for a short period of time keeps a copy.
The data analytics performed on the Database (\myCircled{3a}) dataset and the Cache (\myCircled{3b}) adhere to the use-cases for predictive (long-term) and descriptive (long-term, short-term) data analytics.

The Message Broker (\myCircled{2b}) enables one-way connection (physical twin $\rightarrow$ digital twin).
The API Server (\myCircled{3c}) enables the digital twin $\rightarrow$ link.
The API Server (\myCircled{3c}) communicates directly with the System Knobs (\myCircled{2c}) in order to take meaningful action, based on the (predictive) insights generated by the Digital Twin (III).
Additionally, the Physical Twin (III) can query the API Server (\myCircled{3c}) for one-shot requests (\eg to create a new datacenter prototype configuration, to request special data analysis).
Moreover, the Datacenter Operators (\myCircled{1c}) can query the API Server (\myCircled{3c}) for extra insights, when necessary.

\subsection{The Predictive Analytics Engine}\label{sss:predictive_analytics_engine}
The Predictive Analytics (IV) module is an extensible part of the reference architecture, enabling different kinds of predictive analysis.
In our design, to facilitate meaningful predictions we incorporate an Event-driven Simulator (\myCircled{4a}), Analytics Engine (\myCircled{4b}), and a Monitoring Service (\myCircled{4c}).

We chose an Event-driven Simulator (\myCircled{4a}) as a core element of predictive analytics.
For the rationale behind this decision see \Cref{ss:datacenters}.
The predictions from the Event-driven Simulator (\myCircled{4a}) communicate directly with the Database (\myCircled{3a}) to store the simulation results, and with the Analytics Engine (\myCircled{4b}).

The Analytics Engine (\myCircled{4b}) performs descriptive and predictive data analytics on the results from the Event-driven Simulator (\myCircled{4a}) and the telemetry data collected from the physical datacenter (stored in (\myCircled{3a}, \myCircled{3b})).
This component is highly extensible to accommodate the different types of analytics (\eg \gls{ml}, \gls{ai}, statistical methods).
The results of the data analysis are propagated to the API Server (\myCircled{3c}) to be queued for retrieval by the physical twin.

The Monitoring Service (\myCircled{4c}) serves as a separate thread to watch over the collected telemetry in real-time.
It facilities detection of irregularities in collected data, which adheres to the functional requirements set out for the system.
Any discrepancies are communicated to the Analytics Engine (\myCircled{4b}) for further analysis, and potential insights.

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{images/message_broker.png}
	\caption[The detailed view of the Message Broker.]{The detailed view of the Message Broker (\myCircled{2b}) from \Cref{fig:reference_architecture}.}
	\label{fig:message_broker}
\end{figure}

\section{Requirement Validation}\label{ss:requirement_validation}
%How does this model satisfy the requirements? 
%How did you arrive at this specific model?
In this section we describe the rationale behind the different design decisions of \mysystem.
To show that our model satisfies the functional and non-functional requirements, we map each component of \mysystem onto the requirements it fulfills.
\input{sources/requirements_validation.tex}

The Interactive Dashboard (\myCircled{2a}) supports (\textbf{FR2}), as it allows to display the failures detected to datacenter operators; it is the core of (\textbf{FR4}) because descriptive analytics are synonymous to visualizations.
It enables (\textbf{FR5}) because the predictive insights can be displayed directly through the dashboard to the datacenter operators.
Operators can then overwrite the autonomous decisions performed by the \gls{dt}.
Dashboards are the current industry practice for clear data visualization (\textbf{FR7}).
For our implementation, we ensure (\textbf{NFR1}) is met by including a state-of-the-art dashboard.
(\textbf{NFR2}) and (\textbf{NFR3}) are met by including in the system implementation a dashboard that enables logging of metrics.

The Message Broker (\myCircled{2b}) indirectly fulfills (\textbf{FR1}).
A good and robust message broker alleviates potential bottlenecks that might stem from large data transfer.
My including a dedicated message broker, we ensure the bottle neck when twinning is not \mysystem.
As the Message Broker (\myCircled{2b}) is a vital component in the communication between the datacenter operators and the physical twin, it indirectly enables (\textbf{FR2}), (\textbf{FR3}), (\textbf{FR4}), (\textbf{FR5}).
In the other direction, to ensure that the \gls{dcdt} can ingest any amount of data, the Message Broker (\myCircled{2b}) is needed for (\textbf{FR6}).
The non-functional requirements are met by including a state-of-the-art message broker from either academia or the industry in the implementation.

The inclusion of System Knobs (\myCircled{2c}) in the reference architecture fulfills (\textbf{FR2}).
With autonomous actions, the \gls{dt} can alter the operation of the datacenter based on the detected failures.
Failure detection must support datacenter operators in meeting \gls{sla}s.
Autonomous actions, enabled by System Knobs (\myCircled{2c}) help achieve that.
(\textbf{FR4}) and (\textbf{FR5}) are met by ensuring the insights that come from the \gls{dt} can be automated with the system knobs.
Supporting both predictive and descriptive analytics is core to the system knobs.
(\textbf{NFR4}) is met as the existing digital twin deployments often contain (indirectly) the System Knobs (\myCircled{2c}) element.
Including it in our design adheres to best design practice.
Importantly, (\textbf{NFR5}) is fulfilled together by creating a predictive analytics engine that is capable of intelligent insights and system knobs.
In order for the predictive insights to be realized, systems knobs are needed.

The Database (\myCircled{2c}) meets (\textbf{FR3}), (\textbf{FR4}), (\textbf{FR5}), (\textbf{FR6}) and (\textbf{NFR2}).
Trivially, including a simple \gls{dbms} in the architecture fulfills (\textbf{FR3}).
For the different types of analytics (\textbf{FR4}), (\textbf{FR5}), the Database (\myCircled{2c}) enables insights from historical patterns.
Database are the current state-of-the-art in storing telemetry (\textbf{FR6}).
They are designed to hold arbitrary volume of data.
Lastly, (\textbf{NFR2}) is met by storing in the database the logs from both the \gls{dcdt} and the physical datacenter.

The API Server (\myCircled{3c}) ensures (\textbf{FR4}), (\textbf{FR5}), (\textbf{FR2}) and (\textbf{NFR3}).
The server, which facilitates the insights that come into the System Knobs (\myCircled{2c}), by proxy enables (\textbf{FR4}) and (\textbf{FR5}).
To detect failures, some sort of notification daemon is needed for the datacenter operators, not just the dashboard.
The API Server (\myCircled{3c}) fulfills that role, ensuring (\textbf{FR2}).
The API server (\myCircled{3c}), specifically a server using the \gls{http} protocol is a state-of-the-art approach to existing digital twin communication (\textbf{NFR3}).

The Cache (\myCircled{3b}) ensures the same requirements as the Database (\myCircled{3a}), with a couple of exceptions.
Namely, it also ensures (\textbf{NFR1}) and (\textbf{NFR3}).
Caching is needed for sub 1-second visualizations, and the inclusion of a cache for in-band data analytics is the current community practice.

In our design, we include an Event-driven Simulator (\myCircled{4a}).
This ensures (\textbf{FR1}), (\textbf{FR2}), (\textbf{FR4}), (\textbf{FR5}) and all non-functional requirements except (\textbf{NFR2}).
Using a discrete-event simulator ensures that our system can handle workloads that are representative of the simulator at use.
Simulation, a method that is widely accepted by the scientific community, is one way \mysystem can ensure (\textbf{FR1}).
Detecting failures, enabling both descriptive and predictive analytics depends on how good of a simulator (\myCircled{4a}) is.
In principle, a robust, holistic simulator incorporated into the predictive engine is capable of meeting (\textbf{FR4}), (\textbf{FR2}) and (\textbf{FR5}).
Simulation also allows to model large workloads and to interact with the simulation in real-time (\textbf{NFR1}), it is considered a state-of-the-art approach to datacenter modelling (\textbf{NFR3}) and, with good data analytics provides enough data for insights at varying levels of confidence (\textbf{NFR4}).

The Analytics Engine component (\myCircled{4b}) and the Monitoring Service (\myCircled{4c}) fulfill the same set of requirements, namely (\textbf{FR1}), (\textbf{FR2}), (\textbf{FR4}) and (\textbf{FR5}).
Additionally the Analytics Engine (\myCircled{4b}) also fulfills (\textbf{NFR4}), as it provides the capability to differentiate between different levels of confidence.
Both components are working together in unison, closely watching and reacting to the metrics from the real datacenter.
Hence, they fulfill (\textbf{FR2}), (\textbf{FR4}) and (\textbf{FR5}) and are crucial to the \gls{dcdt} operation.
(\textbf{FR1}) is fulfilled by ensuring both the Analytics Engine and the Monitoring Service are not the source of any bottlenecks during the system implementation.
