diff options
Diffstat (limited to 'content/background.tex')
| -rw-r--r-- | content/background.tex | 196 |
1 files changed, 144 insertions, 52 deletions
diff --git a/content/background.tex b/content/background.tex index 5ab53dc..74083e5 100644 --- a/content/background.tex +++ b/content/background.tex @@ -1,36 +1,97 @@ \chapter{Background}\label{s:background} \section{Overview}\label{ss:background_overview} \begin{mynote} - The contribution in this chapter is three-fold: + Addressing RQ$_1$, the contribution in this chapter is three-fold: \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt] - \item We provide a brief overview on datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning (\Cref{sss:what_is_digital_twinning}). - \item We survey the state-of-the-art concerning datacenter digital twinning (\Cref{ss:digital_twins_for_datacenters}). - \item We construct a system model for existing datacenter digital twins (\Cref{ss:system_model_for_dcdts}) + \item We provide a brief overview on datacenters (\Cref{ss:datacenters}) datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning~ (\Cref{ss:digital-twinning}). + \item We survey the state-of-the-art concerning datacenter digital twinning~ (\Cref{sss:advanced_dts}). + \item We propose a system model (architecture) for datacenter digital twins, including a novel Digital Thread component for the operational needs linking the digital with the physical worlds~(\Cref{ss:system_model_for_dcdts}). \end{enumerate} \end{mynote} \section{Datacenters}\label{ss:datacenters} -In this section we provide a short background on datacenter simulation and compute failures. -We find it useful to provide a brief introduction to both topics so as to ensure reader's fullest understanding of subsequent chapters. -Since datacenters are important building blocks of the digital society, reliable warehouse management is a key priority for datacenter operators. -Incorrect management decisions can lead to missed \gls{sla}s~\cite{DBLP:journals/corr/IosupKLVG22} and even large financial penalties~\cite{DBLP:report/AnnualOutageAnalysis2025}. -However, efficient and timely management is a difficult challenge, because datacenters are extremely complex facilities. -To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. - -\subsection{Datacenter Simulation}\label{sss:simulation} +In this section we provide a short background on datacenters, datacenter simulation and compute failures. +We find it useful to provide a brief introduction to these topics so as to ensure reader's fullest understanding of subsequent chapters. +%What are the parts of a data center? + +A datacenter is ``a physical room, building, or facility for the purpose of the storage, management, and dissemination of data and information, including training artificial intelligence, housing IT infrastructure, computer systems, and associated components.''~\cite{Wikipedia:article/Datacenter}. +In essence, datacenters contain a large amount servers, and everything that is needed to maintain them. +Most often servers are specially-designed motherboards with a (multicore) \gls{cpu}, \gls{ram} and storage. +More diverse servers include a \gls{cpu}, \gls{tpu}, or \gls{npu}. +To efficiently organize the datacenter, servers are placed within server \emph{racks}. +To maintain a large number of server racks, datacenters contain a cooling system to control the heat transfer and temperature of both the hardware and the entire facility. +Additionally, datacenters consume vast amounts of electricity~\cite{Wikipedia:article/Datacenter}. +Because of this, the datacenter power supply play a critical role in keeping the services running on the servers always available. +An example datacenter in \gls{cern}, is depicted on \Cref{fig:datacenter}. + +\begin{figure}[t] + \centering + \includegraphics[width=0.9\linewidth]{images/datacenter.jpg} + \caption[Datacenter in CERN.]{Example of a datacenter in \gls{cern}, Switzerland (2010)~\cite{Wikipedia:article/Datacenter}. The figure depicts servers within servers racks, and network cables interconnecting them.} + \label{fig:datacenter} +\end{figure} + +%Who are the stakeholders? +Datacenters form the backbone of the digital society. +The main stakeholders, besides the companies in the \gls{it} sector, are intelligent healthcare, remote work, online gaming, digital government and education, banking and finance, transport and logistics~\cite{DBLP:journals/corr/IosupKLVG22}. +All of the above industries need reliable datacenters to work well in the 21\textsuperscript{st} century. + +%Where does the actual complexity come from? +The high demand for online services drives datacenter complexity. +Moreover, due to the Jevon's paradox of Computer Systems~\cite{Wikipedia:article/JevonsParadox}, improved availability increases the demand. +As a result, datacenters contain hundreds, or even thousands of hardware components. +Every device may have a different vendor, new configuration, unusual interface \etc +Because of this, datacenter operators are often faced with difficult operational and architectural challenges~\cite{Wiley:book/Condor2005,DBLP:conf/ccgrid/MastenbroekAJLB21}, which span software and performance engineering. +Making sure that all the parts of the datacenter work together is a tough task. +What drives datacenter complexity even further is that sophisticated systems are not merely a sum of their parts~\cite{Wikipedia:article/Systems_Thinking}. +The combination of the above factors makes datacenter management a difficult, non-trivial challenge. + +\section{Compute Failures}\label{sss:failures} +A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}. +A simple example of a failure is when an old hard drive stops working. +Data on the disk is lost, and services running on the respective server are disrupted. +In reality, problems with the power supply account for most failures (54\%). +The runner-ups are problems with cooling (13\%), and \gls{it}/software (12\%)~\cite{DBLP:report/AnnualOutageAnalysis2025}. +Power related failures may stem from software/firmware issues, battery degradation, overheating, power generator failure, mechanical problems, faulty control logic \etc~\cite{DBLP:report/AnnualOutageAnalysis2025}. + +Failure-caused outages are costly. +According to the Uptime Institute, 20\% of all outages cost more than 1 million +USD\$~\cite{DBLP:report/AnnualOutageAnalysis2025}. +Moreover, failures in datacenters result in service downtime, missed \gls{sla} and user inconvenience~\cite{DBLP:conf/acsos/TalluriOVTI21, DBLP:journals/jpdc/JavadiKIE13}. +Industries that rely on 24 hour access suffer the most from datacenter outages. +The impact of failures on medical informatics, nuclear power-plants, banks and financial institutions, airlines, and e-commerce is the most severe~\cite{Wikipedia:article/Downtime}. +Because of this, it is important to prevent failures. + +OpenDC uses the notion of a \emph{failure model} to simulate failures, alongside \emph{failure traces}. +In OpenDC, a failure constitutes a full host crash, regardless of whether the cause of the failure is a hardware of software problem. +A result of a failure in OpenDC, all tasks running on the given host are killed, and need to be rescheduled. +A failure model consists of two statistical distributions: +\begin{enumerate*}[label=(\arabic*)] + \item to model service unavailability + \item to model service availability. +\end{enumerate*} +A failure trace is defined by an interval, duration, and intensity of several failures, which are later looped throughout the simulated workload~\cite{GitHub:software/OpenDC}. +In summary OpenDC enables experimentation with failures that enables insights that are not provided by other state-of-the-art software. +However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter. +Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures. +\section{Datacenter Simulation}\label{sss:simulation} \input{sources/simulator_comparison.tex} + +Efficient and timely datacenter management is a difficult challenge, because datacenters are extremely complex facilities. +They require deep understanding to operate properly. +However, running real-world experiments is costly in both time and resources. +Additionally, experimentation \emph{in situ} is unsustainable and difficult to reproduce. +Alternatives to real-world experiments include simulation and mathematical analysis. +Because mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}, in this project we only consider simulation as a foundation for the \gls{dcdt}. +%To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. + Simulation empowers better design, testing and management of datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. A well-designed datacenter simulator can estimate a months-long workload in a few minutes or hours. To simulate is to ``imitate of real-world process or system over time, enabling the study of, and experimentation with the internal interactions of complex systems''~\cite{DBLP:books/daglib/0034857} In this project we only consider \emph{discrete-event simulation}. - Discrete-event simulation represents system operations as a sequence of events over time, with an assumption that no changes occur between the events. Due to the scale and complexity of datacenters, most simulators use discrete-event simulation~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. - -Alternatives to simulation include real-world experimentation and mathematical analysis. -However, experimentation \emph{in situ} is unsustainable, expensive and difficult to reproduce and mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -Therefore, in this project we only consider simulation as a foundation for the \gls{dcdt}. There exist many datacenter simulation tools, for example DGSim~\cite{DBLP:conf/europar/IosupSE08}, CloudSim~\cite{DBLP:journals/spe/CalheirosRBRB11}, SimGrid~\cite{DBLP:journals/corr/CasanovaGLQS13}, iCanCloud~\cite{DBLP:journals/grid/NunezVCCCL12}, GroudSim~\cite{DBLP:conf/europar/OstermannPPF10} and OpenDC~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. See \Cref{tab:datacenter_simulator_comparison} for a comparison of selected datacenter simulators, combined by Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. In order to narrow the scope of the project, we only consider {OpenDC} as a simulator for the digital twin design. @@ -38,31 +99,11 @@ We decided to use {OpenDC}, because we find it important for a simulator to mode \emph{Failure models} are a carefully calibrated, advanced feature of {OpenDC}. Further details about {OpenDC} can be referred to in the linked literature \cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -\subsection{Compute Failures}\label{sss:failures} -A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}. -We distinguish 2 failure types: \begin{enumerate*}[label=(\arabic*)] - \item software failures - \item hardware failures. -\end{enumerate*} -For example, a hypervisor crash a software failure. -Each \gls{vm} within the crashed hypervisor is killed as a result. -An example of a hardware failure is a host crash, where a single server stops working (\eg as a result of a disk fault, or faulty power supply cable). -Hardware and software failures in datacenters result in service downtime, missed \gls{sla} and user inconvenience~\cite{DBLP:conf/acsos/TalluriOVTI21, DBLP:journals/jpdc/JavadiKIE13}. -OpenDC uses the notion of a \emph{failure model} to simulate failures, alongside \emph{failure traces}. -A failure model consists of two statistical distributions: -\begin{enumerate*}[label=(\arabic*)] - \item to model service unavailability - \item to model service availability. -\end{enumerate*} -A failure trace is defined by an interval, duration, and intensity of several failures, which are later looped throughout the simulated workload (source \url{opendc.org}). -In summary OpenDC enables experimentation with failures that enables insights that are not provided by other state-of-the-art software. -However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter. -Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures. -Digital twinning is an improvement upon pure simulation. \begin{figure}[t] \centering - \includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.pdf} - \caption[A basic framework for the \gls{dt}.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} + \includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.png} + \caption[A basic framework for the Digital Twin.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} + %Fei Tao is a renowned figure with over 62k citations. He is a figure of authority on digital twins.% \label{fig:five_dimensional_dt} \end{figure} @@ -79,11 +120,11 @@ The longer the \gls{dt} is working, the more accurate its predictions, because a A generic \gls{dt} architecture is depicted in Figure \ref{fig:five_dimensional_dt} from Tao \etal~\cite{DBLP:conf/cirp/TAO2018169}. % Why has not anyone done this before? -Digital twinning has only recently become feasible because of the developments in \gls{hpc}. -Between 2003 and 2011 the compute needed to run a \gls{dt} was simply not present. +Although digital twinning through simplified models exists since the 1960s, detailed and accurate digital twinning of complex and realistic scenarios has only recently become feasible because of developments in \gls{hpc}. +Between 2003 and 2011, the compute needed to run a \gls{dt} was simply not present or was (largely) unaffordable. As such, while the concept existed, the hardware did not catch up yet. However, in the last decade, multicore computing paradigms and the advent of GPU computing has finally enabled computation needed to run digital twins. -As a result, digital twins have become more relevant today than 10 years ago~\cite{DBLP:conf/cirp/TAO2018169}. +As a result, digital twins have become more relevant today than a decade ago~\cite{DBLP:conf/cirp/TAO2018169}. A crucial part any of any \gls{dt} is \emph{predictive modelling}, which drives actionable insights~\cite{DBLP:usdoe/report/AP26894} (see \Cref{fig:predictive_analytics}). \begin{figure}[t] @@ -118,13 +159,13 @@ We select only the digital twins that adhere closest to the \gls{nasem} definiti The aim of this survey is to search and organize the field of \gls{dcdt}s. In this subsection, we describe the methods for collecting relevant scientific articles and present the design of the system model for generic \gls{dcdt}s. -\begin{enumerate}[label=\textbf{\arabic*.}, align=left] +\begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{Review Strategy}\\ - The most common methods for conducting literature surveys are \begin{enumerate*}[label=(\arabic*)] + According to Suman \etal~\cite{DBLP:conf/wosp/SumanCNTMI24}, the most common methods for conducting literature surveys are \begin{enumerate*}[label=(\arabic*)] \item random traversal of the related literature, \item snowballing~\cite{ACM:article/Webster2002}, \item systematic literature survey as proposed by Kitchenham \etal~\cite{DBLP:journals/infsof/KitchenhamPBBTNL10} - \end{enumerate*}~\cite{DBLP:conf/wosp/SumanCNTMI24} + \end{enumerate*}. Random traversal encompasses surveying the field by following suggestions from portals like Google Scholar and randomly querying the different databases. It is an unstructured way to conduct the literature review, and requires little effort. Snowballing is similar to random traversal, but it is more structured. @@ -219,18 +260,69 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su % What is more, Microsoft already offers digital twinning as a service https://azure.microsoft.com/en-us/products/digital-twins/ % Documentation: https://learn.microsoft.com/en-us/azure/digital-twins/ % Moreover, NVIDIA is doing too as well https://www.nvidia.com/en-sg/omniverse/ - -\subsection{System Model for Datacenter Digital Twinning} -\label{ss:system_model_for_dcdts} - \begin{figure}[t] \centering - \includegraphics[width=0.95\linewidth]{images/system_model.pdf} + \includegraphics[width=0.95\linewidth]{images/system_model.png} \caption[A system model for datacenter digital twins.]{A generic system model for datacenter digital twin deployments. The design of DyTwin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24} indirectly incorporates in its architecture a ``virtual-to-virtual`` digital thread between different digital twins. Zhao \etal likewise present key elements to the digital thread in their architecture~\cite{DBLP:conf/AppliedEnergy/Zhao20}. We add the \emph{Digital Thread} to our model explicitly.} \label{fig:system_model} \end{figure} +\subsection{System Model for Datacenter Digital Twinning} +\label{ss:system_model_for_dcdts} +In \Cref{fig:system_model} we propose a new, holistic model of \gls{dcdt}s from \Cref{sss:advanced_dts}. +The figure includes the functionality present in the majority of \gls{dcdt}s, combined together into a unified model. +We distinguish 3 core elements of every \gls{dcdt}: +\begin{enumerate*}[label=(\arabic*)] + \item the virtual world + \item digital thread + \item the physical world +\end{enumerate*}. +\begin{enumerate}[label=\textbf{\arabic*.}] + \item \textbf{Virtual World} contains the \gls{dcdt}. + It represents all the components that exist in software. + Every \gls{dcdt} model can be categorized into two sub-categories: + \begin{enumerate*}[label=(\arabic*)] + \item infrastructure model + \item operations model + \end{enumerate*}. + Each \gls{dcdt} from \Cref{sss:advanced_dts} contains a model of the infrastructure within the datacenter. + This includes virtual replicas of the hardware elements (\eg servers, networking, server racks, rooms). + These elements have varying degrees of fidelity. + For example, NetGraph models the datacenter interconnect using purely configuration files. + On the contrary ExaDigiT models the datacenter hardware fully in 3D. + Both offer virtual infrastructure models as a part of the \gls{dcdt}. + + The operations model is likewise present in all deployments. + It models the \emph{behaviour} of the datacenter, \ie the data flow, the different workloads running on the compute, the amount of data stored in each hosts \etc. + Both the infrastructure model and the operations model are part of all \gls{dcdt} deployments from \Cref{sss:advanced_dts}. + A digital twin that contains only the infrastructure model, cannot enable insights into the real-time operation of the datacenter. + Likewise, a \gls{dcdt} containing just the operations model does not possess a capability to \eg simulate the datacenter. + Only both, combined together enable the insights envisioned by the \gls{nasem} \gls{dt} definition~\cite{DBLP:usdoe/report/AP26894}. + \item \textbf{(novel) Digital Thread} connects the virtual world to the physical world. + This is a novel contribution of our thesis. + The digital thread is a \emph{conceptual} element that unites the components which do not belong in either of the worlds. + All \gls{dcdt} programs from \Cref{sss:advanced_dts} contain elements that are ``in-between'' the physical and virtual twin. + After comparing and corroborating these components across deployments, we find 4 that prevail the most: + \begin{enumerate*}[label=(\arabic*)] + \item the visualization interface + \item the message broker + \item the monitoring system + \item the system knobs + \end{enumerate*}. + These elements \emph{connect} facilitate the connection between the physical and the virtual. + For example, the visualization interface provides insights from the metrics collected by the \gls{dcdt} (virtual world) to the datacenter operators (physical world). + The message broker transfers the data from the real datacenter (physical world) to the digital twin (virtual world). + \item \textbf{Physical World} models the real datacenter. + All deployments in \Cref{sss:advanced_dts} contain this element. + Moreover, within the datacenter, we distinguish between 3 core elements that are necessary to model the datacenter faithfully\begin{enumerate*}[label=(\arabic*)] + \item the \gls{it} equipment + \item cooling subsystem + \item power supply + \end{enumerate*}. + All of the aforementioned systems from \Cref{sss:advanced_dts} model either of the 3 elements. + In order to adhere to the holistic view of \gls{dcdt}s, and to fulfill the \gls{nasem}'s definition, the system must contain all 3 of these elements. +\end{enumerate} %To summarize, many \gls{dcdt}'s model the cooling systems inside the warehouse, because in a typical datacenter cooling accounts for more than 40\% of total electricity usage~\cite{DBLP:conf/AppliedEnergy/Zhao20}. %Since the cooling subsystem is mainly airflow-based, \gls{dt} designers often opt for a \gls{cfd} approach to model the facility. @@ -254,4 +346,4 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su %In your work, consider adding such an endpoint, albeit explain in future work that you envision \emph{implementing} this endpoint in the future. -\section{Discussion}\label{ss:background_discussion} +%\section{Discussion}\label{ss:background_discussion} |