\chapter{Background}\label{s:background}

\section{Datacenters}\label{ss:datacenters}
In this section we provide a short background on datacenters, datacenter simulation and hardware failures.

Datacenters are important building blocks of the digital society~\cite{DBLP:journals/corr/IosupKLVG22}.
Reliable warehouse management is a key priority for datacenter operators, because incorrect decisions can lead to missed \gls{sla}s.
However, reliable and timely management is a difficult challenge, because datacenters are extremely complex facilities.
To help datacenter operators, the scientific community proposes to simulate datacenter operations to make more informed operational decisions.

\subsection{Datacenter Simulation}\label{sss:simulation}
\input{sources/simulator_comparison.tex}
Simulation is an excellent way to help design, test and manage datacenters.
To simulate is to ``imitate of real-world process or system over time, enabling the study of, and experimentation with the internal interactions of complex systems''~\cite{DBLP:books/daglib/0034857}
Alternatives to simulation include real-world experimentation and mathematical analysis.
However, experimentation \emph{in situ} is unsustainable, and mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.
A good datacenter simulator can estimate the results of running a months long workload in a matter of minutes or hours.
The scientific community has developed several datacenter simulators (see \Cref{tab:datacenter_simulator_comparison}).
In our work, we decided to use \code{OpenDC}, because we find it important for a simulator to model hardware failures well.
\emph{Failure models} are a carefully calibrated, advanced feature of \code{OpenDC}.

\subsection{Compute Failures}\label{sss:failures}
A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}.
Hardware and software failures in datacenters result in service downtime, missed \gls{sla} and user inconvenience~\cite{DBLP:conf/acsos/TalluriOVTI21, DBLP:journals/jpdc/JavadiKIE13}.
A good example of a software failure is a hypervisor crash.
Each \gls{vm} within the crashed hypervisor is killed as a result.
An example of a hardware failure is a host crash, where a single server stops working (\eg as a result of a disk fault, or faulty power supply cable).

\code{OpenDC} uses the notion of a \emph{failure model} to simulate failures, alongside \emph{failure traces}.
A failure model consists of two statistical distributions:
\begin{enumerate*}[label=(\arabic*)]
	\item to model service unavailability
	\item to model service availability.
\end{enumerate*}
A failure trace is defined by an interval, duration, and intensity of several failures, which are later looped throughout the simulated workload (source \url{opendc.org}).
In summary \code{OpenDC} enables experimentation with failures that enables insights that are not provided by other state-of-the-art software.
However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter.
Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures.

\section{Digital Twinning}\label{ss:digital-twinning}
% To fix: remove the \gls commands for ExaDigiT.
% This is getting silly.
In this section we explore how the datacenter management can be improved using a novel modelling technique, digital twinning.
We present the generic, field-agnostic \gls{dt} definition and investigate how \emph{datacenter} digital twinning applies the definition in practice.

\subsection{What is Digital Twinning?}\label{sss:what_is_digital_twinning}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.pdf}
	\caption{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.}
	%Fei Tao is a renowned figure with over 62k citations. He is a figure of authority on digital twins.%
	\label{fig:five_dimensional_dt}
\end{figure}
% Here talk a bit about different types of data analytics that are performed in a digital twin.
``A \emph{digital twin} is a set of virtual information constructs that mimics the structure, context and behaviour of a natural, engineered or social system, is dynamically updated with data from its physical twin, has predictive capability, and informs decisions that realize value''~\cite{DBLP:usdoe/report/AP26894}.
A crucial characteristic that differentiates digital twinning from simulation and statistical modelling is the \emph{digital thread}: a bi-directional channel that enables continuous interaction between the virtual and physical entities.
The longer the \gls{dt} is working, the more accurate its predictions, because a holistic twin aggregates historical patterns together with up-to-date monitoring data.
A generic \gls{dt} architecture is depicted in Figure \ref{fig:five_dimensional_dt} Section \ref{s:intro} from Tao \etal~\cite{DBLP:conf/cirp/TAO2018169}.

% Why has not anyone done this before?
Digital twinning has only recently become feasible because of the developments in \gls{hpc}.
Between 2003 and 2011 the compute needed to run a Digital Twin was simply not present.
As such, while the concept existed, the hardware did not catch up yet.
However, in the last decade, multicore computing paradigms and the advent of GPU computing has finally enabled computation needed to run digital twins.
As a result, digital twins have become more relevant today than 10 years ago~\cite{DBLP:conf/cirp/TAO2018169}.

A crucial part any of any \gls{dt} should be \emph{predictive modelling}, which drives actionable insights~\cite{DBLP:usdoe/report/AP26894}.
Predictive modelling uses statistics to predict outcomes.
When deployed commercially, for example in datacenters, predictive modelling is often referred to as predictive analytics~\cite{Wikipedia:PredictiveModelling}.
Almost any statistical model can be used for prediction purposes, but nowadays predictive analysis is synonymous with machine learning.
A primary example of popular analysis type is linear regression.
However, any modelling technique, \eg \emph{discrete-event simulation} can be used to make the predictions.

A major limitation of predictive analytics is that history cannot always predict the future.
Using historical data to predict outcomes works only under the assumption that there are certain long lasting patterns in the system.
Additionally, no matter how extensive is the training data, there is always the possibility of new variables that have not been considered or even defined, yet are critical to the outcome of the prediction~\cite{Wikipedia:PredictiveModelling}.

%Here you have to cite Deisenroth, 2024, chapter 8.1.4.
%An inference function is a machine learning model which uses probabilistic parameter estimation~\cite{}.
%A prime example of using probability to find a good machine learning model is Bayesian inference.
% Stanford Encyclopedia of Philosophy, Douven 2017
%The process of inference from data to provide the best explanation is called abduction.
% (3) in the original paper by Fei Tao is referenced to just `Services`.
% Nonetheless I name them here as Data Analysis Services, because what Fei Tao lists (e.g., fault detection, fault determination, fault-tolerant management, maintenance) is inherently reliant on good data analytics.
%\subsection{Digital Twins across Domains}\label{sss:digital_twins_across_domains}

\subsection{Digital Twins for Datacenters}\label{sss:digital_twins_for_datacenters}

In this section, we survey the work related to datacenter digital twinning.
We summarize our results in Table \ref{tab:dt_features_comparison} to compare and contrast the features of existing datacenter digital twins.
We select only the digital twins that adhere closest to the \gls{nasem} definition~\cite{DBLP:usdoe/report/AP26894}.

\input{sources/dt_features_comparison.tex}

ExaDigiT~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24} is an open-source framework for developing digital twins of supercomputers.
It consists of 3 modules:
\begin{enumerate*}[label=(\arabic*)]
	\item resource allocator and power simulator
	\item thermal cooling model
	\item augmented reality 3D model
\end{enumerate*}
of the supercomputer.
ExaDigiT has been used at the Frontier supercomputer at the Oak Ridge National Laboratory in the USA, successfully predicting potential energy losses at the supercomputer.
Brewer \etal include alongside the framework architecture an open-source artifact and a set of extensive verification and validation experiments.
The authors differentiate between different digital twins within ExaDigiT, such as \begin{enumerate*}[label=(\arabic*)]
	\item descriptive twin
	\item informative twin
	\item predictive twin
	\item comprehensive twin
	\item autonomous twin
\end{enumerate*}
that together form the system.
The \emph{predictive twin} leverages data driven operational analytics to create \gls{ml} models. Authors argue that alongside simulation, \gls{ml} models should also have a significant role for modeling system workloads in \eg application fingerprinting.
Within the \emph{autonomous twin} the authors use \gls{rl} to train agents that can be used to make control decisions in order to optimize different processes.
In order to model the cooling system the authors use the Modelica software, and to predict energy power draw they coded a Python script.
The authors provide a intuitive way to interact with the system using a visual dashboard, and an advanced augmented reality model.
The authors posit that the best way to address the 3V's of data (velocity, volume and variety) is to use augmented reality coupled with dashboards.

SmartDC~\cite{DBLP:conf/noms/ZhangZLZWC22} is a digital twin solution for optimization of power consumption in datacenters.
Specifically, Zhang \etal propose that using \gls{ai} enhanced modeling paired with digital twinning can help make dynamic adjustments to the datacenter cooling subsystem.
SmartDC has been proven to ensure efficient energy-saving rate of a China Telecom datacenter at 41\%.
However, the main purpose of SmartDC is not to continuously interact with the facility, but to provide additional training data for a more accurate, \gls{ml} solution.
The digital twin is designed to provide extra datasets for training \gls{ai} models.
% This digital twin together with ExaDigiT use computational fluid dynamics (CFD).
% ExaDigiT uses open-source Modelica software and SmartDC uses proprietary 6SigmaDC.
% At this point it would make sense to create the distinction between _structural_ digital twinning and _behavioural_ digital twinning.
% Link to 6SigmaDC: https://www5.cadence.com/trial_datacenter_insights_lp.html

DyTwin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24} is an adaptive digital twin with visualization and anomaly detection features.
The system, developed at \gls{hp} is a precursor to the vision on datacenter digital twinning published by Athavale \etal~\cite{DBLP:journals/computer/AthavaleBBMMPS24}.
DyTwin is the only system capable of failure detection in datacenters.
Moreover, it is the only system to incorporate the idea of federation into the concept of digital twinning.
DyTwin is designed to interact not only with the physical facility, but also other federated digital twins.
Taheri \etal show that DyTwin can effectively detect 100\% of CPU usage anomalies (\ie irregularities that affect CPU utilization, ranging from 5\% to 60\%).

ChatTwin~\cite{DBLP:conf/sensys/LiW0Z0T23} is an \gls{ai} and \gls{llm} powered system that enables easy deployment and configuration of digital twins for datacenters.
It is a \emph{text-to-3D} approach to building digital twins of datacenters.
ChatTwin is the only work in the field that does not share the simulation technique used to construct the digital twin based on ChatTwin's configuration.
Li \etal provide a thorough set of experiments to show ChatTwin generates the \gls{json} \gls{dcdt} configuration efficiently, but do not share the final 3D visualization results.

Reducio~\cite{DBLP:conf/sensys/CaoW0022} is a system designed to further optimize the \gls{cfd} approach to datacenter modeling.
Instead of using plain \gls{cfd} the authors focus on \gls{pod} approaches to approximate the heat transfer.
Using the \gls{pod} technique, the authors are able to model the datacenter more efficiently, achieving sub 1 degree Celsius \gls{mae} in temperature prediction.
Moreover, their model outperforms the \gls{cfd} approaches.
Cao \etal evaluate their system on an edge datacenter with 70 CPU-only server racks (see Figure 3 in \cite{DBLP:conf/sensys/CaoW0022}), and on a hyper-scale datacenter with thousands of servers (see Figure 4 in \cite{DBLP:conf/sensys/CaoW0022}).
Their results show promising gains in physics-based datacenter modelling over the conventional approaches.

NetGraph~\cite{DBLP:conf/sigcomm/HongWDSSHZY21}, designed by Huawei Technologies and China Mobile.
NetGraph is the only system in our literature review that focuses on network management (see \Cref{tab:dt_features_comparison}).
Moreover, NetGraph employs a unique modelling technique, combining device, network and service models using graph theory.
The authors evaluate their system in a Huawei datacenter with over 50000 server racks.
With over 20 million connections in the network graphs, the system is a prime example of datacenter digital twin potential.

Kalibre~\cite{DBLP:conf/sensys/WangZD0TCWZ20} is a system designed by Wang \etal in order to overcome the cons of \gls{cfd}.
To lessen the computational overhead, Wang \etal propose to use a knowledge-based neural surrogate to calibrate the different \gls{cfd} models.
Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves sub 1 degree Celsius \gls{mae}, similarly to Reducio~\cite{DBLP:conf/sensys/CaoW0022}.

% What is more, Microsoft already offers digital twinning as a service https://azure.microsoft.com/en-us/products/digital-twins/
% Documentation: https://learn.microsoft.com/en-us/azure/digital-twins/
% Moreover, NVIDIA is doing too as well https://www.nvidia.com/en-sg/omniverse/

\subsection{System Model for Datacenter Digital Twinning}
\label{ss:system_model_for_dcdts}

\begin{figure}[tb]
	\centering
	\includegraphics[width=\linewidth]{images/system_model.pdf}
	\caption{A generic system model for data center digital twin deployments.
		The design of DyTwin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}  indirectly incorporates in its architecture a ``virtual-to-virtual`` digital thread between different digital twins.
		Zhao \etal likewise present key elements to the digital thread in their architecture~\cite{DBLP:conf/AppliedEnergy/Zhao20}. We add the \emph{Digital Thread} to our model explicitly.}
	\label{fig:system_model}
\end{figure}

To summarize, many \gls{dcdt}'s model the cooling systems inside the warehouse, because in a typical datacenter cooling accounts for more than 40\% of total electricity usage~\cite{DBLP:conf/AppliedEnergy/Zhao20}.
Since the cooling subsystem is mainly airflow-based, \gls{dt} designers often opt for a \gls{cfd} approach to model the facility.
%The reason why a digital twin might be needed for a cooling subsystem is primarily because of inefficient operational strategy.
%The cooling system parameters are often set constant, regardless of outdoor temperature \etc~\cite{DBLP:conf/AppliedEnergy/Zhao20}.
%Zhang \etal argues that their system is akin to an IoT sensor, essentially.
% This is an important consideration -- DT is not simply a sensor, it must have predictive capabilities and be able to simulate the future.
% Zhang argues that ``digital twin services'' are enabled by simulation monitoring \etc.
% Nonetheless, I dub that they are primarily data analysis services.

%\gls{oda} can be performed in-band (real-time) and out-of-band (from historical data).
%Likewise, Zhao \etal shows that crucial to the digital twin system are ``always-on'' analytics (akin to in-band \gls{oda}) and ``on-demand`` analytics (akin to out-of-band \gls{oda}).

%Include something about data-preprocessing in the pipeline.
%See the article by Fei Tao


%Moreover, a crucial parallel between the work of Zhao \etal and ExaDigiT is the concept of multiple models within a single digital twin.
%Brewer \etal argue ExaDigiT is compromised of 5 ``smaller'' twins too.
%In Zhang \etal the digital twin can communicate with different other digital twins, as in the work of Taheri \etal.
%To do this, the working program has an API, with a specific API endpoint to communicate with other Digital Twins.


%In your work, consider adding such an endpoint, albeit explain in future work that you envision \emph{implementing} this endpoint in the future.