diff options
Diffstat (limited to 'content')
| -rw-r--r-- | content/background.tex | 253 | ||||
| -rw-r--r-- | content/conclusion.tex | 6 | ||||
| -rw-r--r-- | content/design.tex | 144 | ||||
| -rw-r--r-- | content/evaluation.tex | 301 | ||||
| -rw-r--r-- | content/implementation.tex | 97 | ||||
| -rw-r--r-- | content/intro.tex | 119 | ||||
| -rw-r--r-- | content/preamble/abstract.tex | 4 | ||||
| -rw-r--r-- | content/preamble/acknowledgement.tex | 4 |
8 files changed, 666 insertions, 262 deletions
diff --git a/content/background.tex b/content/background.tex index 8e63f53..74083e5 100644 --- a/content/background.tex +++ b/content/background.tex @@ -1,37 +1,97 @@ \chapter{Background}\label{s:background} - \section{Overview}\label{ss:background_overview} \begin{mynote} - The contribution in this chapter is three-fold: + Addressing RQ$_1$, the contribution in this chapter is three-fold: \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt] - \item We provide a brief overview on datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning (\Cref{sss:what_is_digital_twinning}). - \item We survey the state-of-the-art concerning datacenter digital twinning (\Cref{sss:digital_twins_for_datacenters}). - \item We construct a system model for existing datacenter digital twins (\Cref{ss:system_model_for_dcdts}) + \item We provide a brief overview on datacenters (\Cref{ss:datacenters}) datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning~ (\Cref{ss:digital-twinning}). + \item We survey the state-of-the-art concerning datacenter digital twinning~ (\Cref{sss:advanced_dts}). + \item We propose a system model (architecture) for datacenter digital twins, including a novel Digital Thread component for the operational needs linking the digital with the physical worlds~(\Cref{ss:system_model_for_dcdts}). \end{enumerate} \end{mynote} \section{Datacenters}\label{ss:datacenters} -In this section we provide a short background on datacenter simulation and hardware failures. -We find it useful to provide a brief introduction to both topics so as to ensure reader's fullest understanding of subsequent chapters. -Since datacenters are important building blocks of the digital society, reliable warehouse management is a key priority for datacenter operators. -Incorrect management decisions can lead to missed \gls{sla}s~\cite{DBLP:journals/corr/IosupKLVG22} and even large financial penalties~\cite{DBLP:report/AnnualOutageAnalysis2025}. -However, efficient and timely management is a difficult challenge, because datacenters are extremely complex facilities. -To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. - -\subsection{Datacenter Simulation}\label{sss:simulation} +In this section we provide a short background on datacenters, datacenter simulation and compute failures. +We find it useful to provide a brief introduction to these topics so as to ensure reader's fullest understanding of subsequent chapters. +%What are the parts of a data center? + +A datacenter is ``a physical room, building, or facility for the purpose of the storage, management, and dissemination of data and information, including training artificial intelligence, housing IT infrastructure, computer systems, and associated components.''~\cite{Wikipedia:article/Datacenter}. +In essence, datacenters contain a large amount servers, and everything that is needed to maintain them. +Most often servers are specially-designed motherboards with a (multicore) \gls{cpu}, \gls{ram} and storage. +More diverse servers include a \gls{cpu}, \gls{tpu}, or \gls{npu}. +To efficiently organize the datacenter, servers are placed within server \emph{racks}. +To maintain a large number of server racks, datacenters contain a cooling system to control the heat transfer and temperature of both the hardware and the entire facility. +Additionally, datacenters consume vast amounts of electricity~\cite{Wikipedia:article/Datacenter}. +Because of this, the datacenter power supply play a critical role in keeping the services running on the servers always available. +An example datacenter in \gls{cern}, is depicted on \Cref{fig:datacenter}. + +\begin{figure}[t] + \centering + \includegraphics[width=0.9\linewidth]{images/datacenter.jpg} + \caption[Datacenter in CERN.]{Example of a datacenter in \gls{cern}, Switzerland (2010)~\cite{Wikipedia:article/Datacenter}. The figure depicts servers within servers racks, and network cables interconnecting them.} + \label{fig:datacenter} +\end{figure} + +%Who are the stakeholders? +Datacenters form the backbone of the digital society. +The main stakeholders, besides the companies in the \gls{it} sector, are intelligent healthcare, remote work, online gaming, digital government and education, banking and finance, transport and logistics~\cite{DBLP:journals/corr/IosupKLVG22}. +All of the above industries need reliable datacenters to work well in the 21\textsuperscript{st} century. + +%Where does the actual complexity come from? +The high demand for online services drives datacenter complexity. +Moreover, due to the Jevon's paradox of Computer Systems~\cite{Wikipedia:article/JevonsParadox}, improved availability increases the demand. +As a result, datacenters contain hundreds, or even thousands of hardware components. +Every device may have a different vendor, new configuration, unusual interface \etc +Because of this, datacenter operators are often faced with difficult operational and architectural challenges~\cite{Wiley:book/Condor2005,DBLP:conf/ccgrid/MastenbroekAJLB21}, which span software and performance engineering. +Making sure that all the parts of the datacenter work together is a tough task. +What drives datacenter complexity even further is that sophisticated systems are not merely a sum of their parts~\cite{Wikipedia:article/Systems_Thinking}. +The combination of the above factors makes datacenter management a difficult, non-trivial challenge. + +\section{Compute Failures}\label{sss:failures} +A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}. +A simple example of a failure is when an old hard drive stops working. +Data on the disk is lost, and services running on the respective server are disrupted. +In reality, problems with the power supply account for most failures (54\%). +The runner-ups are problems with cooling (13\%), and \gls{it}/software (12\%)~\cite{DBLP:report/AnnualOutageAnalysis2025}. +Power related failures may stem from software/firmware issues, battery degradation, overheating, power generator failure, mechanical problems, faulty control logic \etc~\cite{DBLP:report/AnnualOutageAnalysis2025}. + +Failure-caused outages are costly. +According to the Uptime Institute, 20\% of all outages cost more than 1 million +USD\$~\cite{DBLP:report/AnnualOutageAnalysis2025}. +Moreover, failures in datacenters result in service downtime, missed \gls{sla} and user inconvenience~\cite{DBLP:conf/acsos/TalluriOVTI21, DBLP:journals/jpdc/JavadiKIE13}. +Industries that rely on 24 hour access suffer the most from datacenter outages. +The impact of failures on medical informatics, nuclear power-plants, banks and financial institutions, airlines, and e-commerce is the most severe~\cite{Wikipedia:article/Downtime}. +Because of this, it is important to prevent failures. + +OpenDC uses the notion of a \emph{failure model} to simulate failures, alongside \emph{failure traces}. +In OpenDC, a failure constitutes a full host crash, regardless of whether the cause of the failure is a hardware of software problem. +A result of a failure in OpenDC, all tasks running on the given host are killed, and need to be rescheduled. +A failure model consists of two statistical distributions: +\begin{enumerate*}[label=(\arabic*)] + \item to model service unavailability + \item to model service availability. +\end{enumerate*} +A failure trace is defined by an interval, duration, and intensity of several failures, which are later looped throughout the simulated workload~\cite{GitHub:software/OpenDC}. +In summary OpenDC enables experimentation with failures that enables insights that are not provided by other state-of-the-art software. +However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter. +Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures. +\section{Datacenter Simulation}\label{sss:simulation} \input{sources/simulator_comparison.tex} + +Efficient and timely datacenter management is a difficult challenge, because datacenters are extremely complex facilities. +They require deep understanding to operate properly. +However, running real-world experiments is costly in both time and resources. +Additionally, experimentation \emph{in situ} is unsustainable and difficult to reproduce. +Alternatives to real-world experiments include simulation and mathematical analysis. +Because mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}, in this project we only consider simulation as a foundation for the \gls{dcdt}. +%To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. + Simulation empowers better design, testing and management of datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. A well-designed datacenter simulator can estimate a months-long workload in a few minutes or hours. To simulate is to ``imitate of real-world process or system over time, enabling the study of, and experimentation with the internal interactions of complex systems''~\cite{DBLP:books/daglib/0034857} In this project we only consider \emph{discrete-event simulation}. - Discrete-event simulation represents system operations as a sequence of events over time, with an assumption that no changes occur between the events. Due to the scale and complexity of datacenters, most simulators use discrete-event simulation~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. - -Alternatives to simulation include real-world experimentation and mathematical analysis. -However, experimentation \emph{in situ} is unsustainable, expensive and difficult to reproduce and mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -Therefore, in this project we only consider simulation as a foundation for the \gls{dcdt}. There exist many datacenter simulation tools, for example DGSim~\cite{DBLP:conf/europar/IosupSE08}, CloudSim~\cite{DBLP:journals/spe/CalheirosRBRB11}, SimGrid~\cite{DBLP:journals/corr/CasanovaGLQS13}, iCanCloud~\cite{DBLP:journals/grid/NunezVCCCL12}, GroudSim~\cite{DBLP:conf/europar/OstermannPPF10} and OpenDC~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. See \Cref{tab:datacenter_simulator_comparison} for a comparison of selected datacenter simulators, combined by Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. In order to narrow the scope of the project, we only consider {OpenDC} as a simulator for the digital twin design. @@ -39,52 +99,32 @@ We decided to use {OpenDC}, because we find it important for a simulator to mode \emph{Failure models} are a carefully calibrated, advanced feature of {OpenDC}. Further details about {OpenDC} can be referred to in the linked literature \cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -\subsection{Compute Failures}\label{sss:failures} -A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}. -We distinguish 2 failure types: \begin{enumerate*} - \item software failures - \item hardware failures. -\end{enumerate*} -For example, a hypervisor crash a software failure. -Each \gls{vm} within the crashed hypervisor is killed as a result. -An example of a hardware failure is a host crash, where a single server stops working (\eg as a result of a disk fault, or faulty power supply cable). -Hardware and software failures in datacenters result in service downtime, missed \gls{sla} and user inconvenience~\cite{DBLP:conf/acsos/TalluriOVTI21, DBLP:journals/jpdc/JavadiKIE13}. -OpenDC uses the notion of a \emph{failure model} to simulate failures, alongside \emph{failure traces}. -A failure model consists of two statistical distributions: -\begin{enumerate*}[label=(\arabic*)] - \item to model service unavailability - \item to model service availability. -\end{enumerate*} -A failure trace is defined by an interval, duration, and intensity of several failures, which are later looped throughout the simulated workload (source \url{opendc.org}). -In summary OpenDC enables experimentation with failures that enables insights that are not provided by other state-of-the-art software. -However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter. -Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures. -Digital twinning is an improvement upon pure simulation. +\begin{figure}[t] + \centering + \includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.png} + \caption[A basic framework for the Digital Twin.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} + %Fei Tao is a renowned figure with over 62k citations. He is a figure of authority on digital twins.% + \label{fig:five_dimensional_dt} +\end{figure} \section{Digital Twinning}\label{ss:digital-twinning} In this section we explore how the datacenter management can be improved using a novel modelling technique, digital twinning. We present the generic, field-agnostic \gls{dt} definition and investigate how \emph{datacenter} digital twinning applies the definition in practice. \subsection{What is Digital Twinning?}\label{sss:what_is_digital_twinning} -\begin{figure}[t] - \centering - \includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.pdf} - \caption[A basic framework for the \gls{dt}.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} - %Fei Tao is a renowned figure with over 62k citations. He is a figure of authority on digital twins.% - \label{fig:five_dimensional_dt} -\end{figure} + % Here talk a bit about different types of data analytics that are performed in a digital twin. ``A \emph{digital twin} is a set of virtual information constructs that mimics the structure, context and behaviour of a natural, engineered or social system, is dynamically updated with data from its physical twin, has predictive capability, and informs decisions that realize value''~\cite{DBLP:usdoe/report/AP26894}. A crucial characteristic that differentiates digital twinning from simulation and statistical modelling is the \emph{digital thread}: a bi-directional channel that enables continuous interaction between the virtual and physical entities. The longer the \gls{dt} is working, the more accurate its predictions, because a holistic twin aggregates historical patterns together with up-to-date monitoring data. -A generic \gls{dt} architecture is depicted in Figure \ref{fig:five_dimensional_dt} Section \ref{s:intro} from Tao \etal~\cite{DBLP:conf/cirp/TAO2018169}. +A generic \gls{dt} architecture is depicted in Figure \ref{fig:five_dimensional_dt} from Tao \etal~\cite{DBLP:conf/cirp/TAO2018169}. % Why has not anyone done this before? -Digital twinning has only recently become feasible because of the developments in \gls{hpc}. -Between 2003 and 2011 the compute needed to run a \gls{dt} was simply not present. +Although digital twinning through simplified models exists since the 1960s, detailed and accurate digital twinning of complex and realistic scenarios has only recently become feasible because of developments in \gls{hpc}. +Between 2003 and 2011, the compute needed to run a \gls{dt} was simply not present or was (largely) unaffordable. As such, while the concept existed, the hardware did not catch up yet. However, in the last decade, multicore computing paradigms and the advent of GPU computing has finally enabled computation needed to run digital twins. -As a result, digital twins have become more relevant today than 10 years ago~\cite{DBLP:conf/cirp/TAO2018169}. +As a result, digital twins have become more relevant today than a decade ago~\cite{DBLP:conf/cirp/TAO2018169}. A crucial part any of any \gls{dt} is \emph{predictive modelling}, which drives actionable insights~\cite{DBLP:usdoe/report/AP26894} (see \Cref{fig:predictive_analytics}). \begin{figure}[t] @@ -93,10 +133,10 @@ A crucial part any of any \gls{dt} is \emph{predictive modelling}, which drives \label{fig:predictive_analytics} \end{figure} Predictive modelling uses statistics to predict outcomes. -When deployed commercially, for example in datacenters, predictive modelling is often referred to as predictive analytics~\cite{Wikipedia:PredictiveModelling}. -Almost any statistical model can be used for prediction purposes, but nowadays predictive analysis is synonymous with machine learning. +When deployed commercially, for example in datacenters, predictive modelling is often referred to as predictive analytics~\cite{Wikipedia:article/PredictiveModelling}. +Almost any statistical model can be used for prediction purposes, but nowadays predictive analysis is synonymous with \gls{ml}. A primary example of popular analysis type is linear regression. -However, any modelling technique, \eg \emph{discrete-event simulation} can be used to make the predictions. +However, any modelling technique (\eg \emph{discrete-event simulation}) can be used to make the predictions. Predictive analysis belongs to a larger domain of \gls{oda}. \gls{oda} is the ``use of operational data instrumentation, analysis, integration, and archiving, towards effective design, commissioning, and optimization of datacenter operations'' \cite{DBLP:conf/icppw/BourassaJBCJVS19}. @@ -108,15 +148,53 @@ There exist several \gls{oda} frameworks, for example OMNI \cite{DBLP:conf/icppw A major limitation of predictive analytics is that history cannot always predict the future. Using historical data to predict outcomes works only under the assumption that there are certain long lasting patterns in the system. -Additionally, no matter how extensive is the training data, there is always the possibility of new variables that have not been considered or even defined, yet are critical to the outcome of the prediction~\cite{Wikipedia:PredictiveModelling}. +Additionally, no matter how extensive is the training data, there is always the possibility of new variables that have not been considered or even defined, yet are critical to the outcome of the prediction~\cite{Wikipedia:article/PredictiveModelling}. -\subsection{Digital Twins for Datacenters}\label{sss:digital_twins_for_datacenters} +\section{Literature Survey of Digital Twins for Datacenters}\label{ss:digital_twins_for_datacenters} In this section, we survey the work related to datacenter digital twinning. We summarize our results in Table \ref{tab:dt_features_comparison} to compare and contrast the features of existing datacenter digital twins. We select only the digital twins that adhere closest to the \gls{nasem} definition~\cite{DBLP:usdoe/report/AP26894}. +\subsection{Methodology}\label{sss:method} +The aim of this survey is to search and organize the field of \gls{dcdt}s. +In this subsection, we describe the methods for collecting relevant scientific articles and present the design of the system model for generic \gls{dcdt}s. + +\begin{enumerate}[label=\textbf{\arabic*.}] + \item \textbf{Review Strategy}\\ + According to Suman \etal~\cite{DBLP:conf/wosp/SumanCNTMI24}, the most common methods for conducting literature surveys are \begin{enumerate*}[label=(\arabic*)] + \item random traversal of the related literature, + \item snowballing~\cite{ACM:article/Webster2002}, + \item systematic literature survey as proposed by Kitchenham \etal~\cite{DBLP:journals/infsof/KitchenhamPBBTNL10} + \end{enumerate*}. + Random traversal encompasses surveying the field by following suggestions from portals like Google Scholar and randomly querying the different databases. + It is an unstructured way to conduct the literature review, and requires little effort. + Snowballing is similar to random traversal, but it is more structured. + The surveyor follows references from the relevant articles, and there is a depth limit~\cite{DBLP:conf/wosp/SumanCNTMI24}. + Systematic literature survey is a rigorous, fully-structured process to searching for literature, and it follows the method devised by Barbara Kitchenham~\cite{DBLP:journals/infsof/KitchenhamPBBTNL10}. + In our work, to scope down the project we chose a mix of (1) and (2), with some elements of (3) instead of following solely the systematic literature review process of Kitchenham \etal. + Therefore, our literature cannot be regarded as systematic, instead we can refer to it as comprehensive or semi-structured. + \item \textbf{Analysis of Selected Material}\\ + We borrow the process of Suman \etal conducted during his MSc thesis for a literature survey of \gls{oda}~\cite{DBLP:conf/wosp/SumanCNTMI24}. + The process can be described as follows. + \begin{enumerate*}[label=(\arabic*)] + \item first, search given queries followed by a manual inspection of the context of the article + \item then scan each article, by reading over the abstract, introduction and conclusion and decide whether it applies to \gls{dcdt}s. + \item after selection, extract the details of the \gls{dcdt} from the publication by reading carefully over the article. + \item lastly, interpret the functionality of the \gls{dcdt}s and systematically organize them. + \end{enumerate*} + \item \textbf{Design of the System Model}\\ + Based on the findings of the literature survey, we create a conceptual model of the \gls{dcdt} field. + We decided to create a system model, as the field of \gls{dcdt}s is still under development, and does not include many digital twin deployments. + An alternative to a system model would be a taxonomy. + To create the system model, we first gathered the functionality present in all the \gls{dcdt}s. + For each \gls{dcdt} feature in every article, we evaluated whether this feature is present in other deployments and how important it is for the \gls{dt}. + The result of this process is a set of \gls{dcdt} features that belong to the largest proportion of all \gls{dcdt}s. + Afterwards, for each of the features, we decide how it is interconnected. +\end{enumerate} + \input{sources/dt_features_comparison.tex} +\subsection{Advanced Digital Twins for Datacenters}\label{sss:advanced_dts} ExaDigiT~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24} is an open-source framework for developing digital twins of supercomputers. It consists of 3 modules: \begin{enumerate*}[label=(\arabic*)] @@ -182,18 +260,69 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su % What is more, Microsoft already offers digital twinning as a service https://azure.microsoft.com/en-us/products/digital-twins/ % Documentation: https://learn.microsoft.com/en-us/azure/digital-twins/ % Moreover, NVIDIA is doing too as well https://www.nvidia.com/en-sg/omniverse/ - -\section{System Model for Datacenter Digital Twinning} -\label{ss:system_model_for_dcdts} - \begin{figure}[t] \centering - \includegraphics[width=0.95\linewidth]{images/system_model.pdf} + \includegraphics[width=0.95\linewidth]{images/system_model.png} \caption[A system model for datacenter digital twins.]{A generic system model for datacenter digital twin deployments. The design of DyTwin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24} indirectly incorporates in its architecture a ``virtual-to-virtual`` digital thread between different digital twins. Zhao \etal likewise present key elements to the digital thread in their architecture~\cite{DBLP:conf/AppliedEnergy/Zhao20}. We add the \emph{Digital Thread} to our model explicitly.} \label{fig:system_model} \end{figure} +\subsection{System Model for Datacenter Digital Twinning} +\label{ss:system_model_for_dcdts} +In \Cref{fig:system_model} we propose a new, holistic model of \gls{dcdt}s from \Cref{sss:advanced_dts}. +The figure includes the functionality present in the majority of \gls{dcdt}s, combined together into a unified model. +We distinguish 3 core elements of every \gls{dcdt}: +\begin{enumerate*}[label=(\arabic*)] + \item the virtual world + \item digital thread + \item the physical world +\end{enumerate*}. +\begin{enumerate}[label=\textbf{\arabic*.}] + \item \textbf{Virtual World} contains the \gls{dcdt}. + It represents all the components that exist in software. + Every \gls{dcdt} model can be categorized into two sub-categories: + \begin{enumerate*}[label=(\arabic*)] + \item infrastructure model + \item operations model + \end{enumerate*}. + Each \gls{dcdt} from \Cref{sss:advanced_dts} contains a model of the infrastructure within the datacenter. + This includes virtual replicas of the hardware elements (\eg servers, networking, server racks, rooms). + These elements have varying degrees of fidelity. + For example, NetGraph models the datacenter interconnect using purely configuration files. + On the contrary ExaDigiT models the datacenter hardware fully in 3D. + Both offer virtual infrastructure models as a part of the \gls{dcdt}. + + The operations model is likewise present in all deployments. + It models the \emph{behaviour} of the datacenter, \ie the data flow, the different workloads running on the compute, the amount of data stored in each hosts \etc. + Both the infrastructure model and the operations model are part of all \gls{dcdt} deployments from \Cref{sss:advanced_dts}. + A digital twin that contains only the infrastructure model, cannot enable insights into the real-time operation of the datacenter. + Likewise, a \gls{dcdt} containing just the operations model does not possess a capability to \eg simulate the datacenter. + Only both, combined together enable the insights envisioned by the \gls{nasem} \gls{dt} definition~\cite{DBLP:usdoe/report/AP26894}. + \item \textbf{(novel) Digital Thread} connects the virtual world to the physical world. + This is a novel contribution of our thesis. + The digital thread is a \emph{conceptual} element that unites the components which do not belong in either of the worlds. + All \gls{dcdt} programs from \Cref{sss:advanced_dts} contain elements that are ``in-between'' the physical and virtual twin. + After comparing and corroborating these components across deployments, we find 4 that prevail the most: + \begin{enumerate*}[label=(\arabic*)] + \item the visualization interface + \item the message broker + \item the monitoring system + \item the system knobs + \end{enumerate*}. + These elements \emph{connect} facilitate the connection between the physical and the virtual. + For example, the visualization interface provides insights from the metrics collected by the \gls{dcdt} (virtual world) to the datacenter operators (physical world). + The message broker transfers the data from the real datacenter (physical world) to the digital twin (virtual world). + \item \textbf{Physical World} models the real datacenter. + All deployments in \Cref{sss:advanced_dts} contain this element. + Moreover, within the datacenter, we distinguish between 3 core elements that are necessary to model the datacenter faithfully\begin{enumerate*}[label=(\arabic*)] + \item the \gls{it} equipment + \item cooling subsystem + \item power supply + \end{enumerate*}. + All of the aforementioned systems from \Cref{sss:advanced_dts} model either of the 3 elements. + In order to adhere to the holistic view of \gls{dcdt}s, and to fulfill the \gls{nasem}'s definition, the system must contain all 3 of these elements. +\end{enumerate} %To summarize, many \gls{dcdt}'s model the cooling systems inside the warehouse, because in a typical datacenter cooling accounts for more than 40\% of total electricity usage~\cite{DBLP:conf/AppliedEnergy/Zhao20}. %Since the cooling subsystem is mainly airflow-based, \gls{dt} designers often opt for a \gls{cfd} approach to model the facility. @@ -210,13 +339,11 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su %Include something about data-preprocessing in the pipeline. %See the article by Fei Tao - %Moreover, a crucial parallel between the work of Zhao \etal and ExaDigiT is the concept of multiple models within a single digital twin. %Brewer \etal argue ExaDigiT is compromised of 5 ``smaller'' twins too. %In Zhang \etal the digital twin can communicate with different other digital twins, as in the work of Taheri \etal. %To do this, the working program has an API, with a specific API endpoint to communicate with other Digital Twins. - %In your work, consider adding such an endpoint, albeit explain in future work that you envision \emph{implementing} this endpoint in the future. -\section{Discussion}\label{ss:background_discussion} +%\section{Discussion}\label{ss:background_discussion} diff --git a/content/conclusion.tex b/content/conclusion.tex index 1df3d95..fa90b3e 100644 --- a/content/conclusion.tex +++ b/content/conclusion.tex @@ -37,7 +37,7 @@ We answer the main research question by addressing each sub-research question. \item \emph{How to validate and evaluate a datacenter digital twin architecture in relation to system requirements?}\\ To answer the last research question we crated a prototype. - During the prototype design, we used state-of-the-practice software, such as \code{Confluent Kafka}, \code{Redis} and \code{PostgreSQL} (see \Cref{ss:implementation_overview}). + During the prototype design, we used state-of-the-practice software, such as \code{Confluent Kafka}, \code{Redis} and \code{PostgreSQL} (see \Cref{ss:detailed_implementation_overview}). However, as it turns out, evaluating \gls{dcdt}s is not a trivial task. Lacking the physical datacenter to experiment with, we came up with a novel digital twin evaluation method. Our method, relies solely on discrete-event simulation to model the physical datacenter, overcoming the problems of real-world experimentation (\eg sustainability, costliness, reproducibility). @@ -48,7 +48,7 @@ We answer the main research question by addressing each sub-research question. \begin{figure}[ht] \centering \includegraphics[width=0.8\textwidth]{images/48_years.pdf} - \caption[48 years of microprocessor trend data.]{48 years of microprocessor trend data. Legend: \textcolor{Orange}{$\blacktriangle$ Transistors (thousands)}, \textcolor{Blue}{$\lgblkcircle$ Single Thread Performance (SpecINT $\times 10^3$)}, \textcolor{Green}{$\lgblksquare$ Frequency (MHz)}, \textcolor{Maroon}{$\blacktriangledown$ Typical Power (Watts)}, $\mdlgblkdiamond$ Number of Logical Cores~\cite{DBLP:image/48Microprocessor/Rupp}.} + \caption[48 years of microprocessor trend data.]{48 years of microprocessor trend data. Legend: \textcolor{Orange}{$\blacktriangle$ Transistors (thousands)}, \textcolor{Blue}{$\lgblkcircle$ Single Thread Performance (SpecINT $\times 10^3$)}, \textcolor{Green}{$\lgblksquare$ Frequency (MHz)}, \textcolor{Maroon}{$\blacktriangledown$ Typical Power (Watts)}, $\mdlgblkdiamond$ Number of Logical Cores. (Source:~\cite{DBLP:image/48Microprocessor/Rupp}.)} \label{fig:rupp_48_years_microprocessor_data} \end{figure} @@ -70,7 +70,7 @@ To achieve the \gls{nasem} goals of digital twinning~\cite{DBLP:usdoe/report/AP2 A \gls{dcdt} must posses predictive capabilities, by definition~\cite{DBLP:usdoe/report/AP26894}. \item \textbf{What Is Missing?}\\ To power the predictions, we envision an \gls{ml}-based inference engine as a necessary component of digital twinning. - The need for \gls{ml} arises naturally in scenarios where large volumes of data, requiring little to no preprocessing meet the demand for estimating future facility behaviour~\cite{Wikipedia:PredictiveModelling,CambridgeUniversityPress:book/Deisenroth}. + The need for \gls{ml} arises naturally in scenarios where large volumes of data, requiring little to no preprocessing meet the demand for estimating future facility behaviour~\cite{Wikipedia:article/PredictiveModelling,CambridgeUniversityPress:book/Deisenroth}. However, currently there are no \gls{dcdt} deployments that model the warehouse using an \gls{ml} approach to predict events (see \Cref{tab:dt_features_comparison}). \item \textbf{The Next Steps}\\ In short, we stipulate \gls{dcdt}s should include \gls{ml} in their \gls{oda} analysis. diff --git a/content/design.tex b/content/design.tex index 2eb195d..72414a3 100644 --- a/content/design.tex +++ b/content/design.tex @@ -2,17 +2,19 @@ \chapter{Design of \mysystem, a Digital Twin For Predictive Analysis of Datacenters}\label{s:design} \section{Overview}\label{ss:design_overview} \begin{mynote} - Our contribution in this chapter is three-fold: + Addressing RQ$_2$, our contribution in this chapter is three-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt] - \item We analyze the requirements for \mysystem (\Cref{ss:requirements_analysis}). - \item We propose a conceptual design for \mysystem's architecture (\Cref{ss:design_of_mysystem}) - \item We describe how \mysystem enables predictive analytics through digital twinning (\Cref{ss:design_discussion}) + \item We analyze the requirements for digital twins enabling predictive analysis of datacenters~(\Cref{ss:requirements_analysis}). + + \item We design the \mysystem architecture as a digital twin enabling predictive analysis of datacenters through discrete-event simulation and ODA~(\Cref{ss:design_of_mysystem}). + + \item We analyze how \mysystem fulfills the functional and non-functional requirements~(\Cref{ss:requirement_validation}). \end{enumerate} \end{mynote} \section{Requirements Analysis}\label{ss:requirements_analysis} In this section we determine the requirements that should be fulfilled by \mysystem. -We present here the stakeholders identified by our literature survey (see \Cref{sss:digital_twins_for_datacenters}) and the relevant use-cases. +We present here the stakeholders identified by our literature survey (see \Cref{ss:digital_twins_for_datacenters}) and the relevant use-cases. Afterwards, we list the functional and non-functional requirements for \mysystem. \subsection{Stakeholders}\label{sss:stakeholders} @@ -21,10 +23,11 @@ We identify four main stakeholders of a predictive datacenter digital twin: \item \textbf{Datacenter Managers}\\ Responsible for maintenance and operation of the warehouse, operators manage the datacenter daily. They interact with the servers, bring downed hosts up and ensure customers' services run smoothly at all times. + Importantly, datacenter operators follow a strict set of policies for taking different action. Datacenter operators need to ensure different \gls{sla}s are met, energy costs are balanced and carbon emission quota is maintained. - \item \textbf{Datacenter Technicians}\\ + \item \textbf{Datacenter Engineers}\\ The term datacenter engineers encompasses datacenter architects and technicians alike. - From the moment the datacenter layout is determined, to the physical process of booting the server racks for the first time, datacenter engineers help build and maintain the datacenter. + From the moment datacenter architects determine the warehouse layout, to the physical process of booting the server racks for the first time by the technicians, datacenter engineers help build and maintain the datacenter. They must continuously adapt to changing requirements and ensure everything goes smoothly~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. \item \textbf{Scientists and Academia}\\ Digital twinning generates unprecedented amount of data. @@ -64,9 +67,10 @@ Based on the identified stakeholders we list 6 potential use-cases for a predict \subsection{Functional Requirements}\label{sss:functional_requirements} Based on a subset of the above use-cases, we formulate the functional and non-functional requirements for \mysystem: \begin{enumerate}[label=\textbf{FR\arabic* --},align=left] - \item \textbf{The system should be able to handle workloads of arbitrary size.} \\ - Existing systems range from Cloud through the Edge to HPC digital twins. - Therefore, \mysystem must support workloads similar in length and type to the commercial setting. + \item \textbf{The system should be able to handle workload size representative of its modelling technique.} \\ + Existing \gls{dt}s range from Cloud through the Edge to HPC. + Therefore, \mysystem must not impose any significant limitations on the size or type of the workload it runs. + In particular, \mysystem must be able to handle workloads as large as the underlying modelling technique permits. Without \textbf{FR1}, \mysystem will be incomplete, and like the majority of the \Cref{tab:dt_features_comparison} systems, its use-case will be niche. \textbf{FR1} is necessary to avoid overly-specializing the \gls{dcdt}. \item \textbf{The system should support failure detection.}\\ @@ -102,8 +106,12 @@ Based on a subset of the above use-cases, we formulate the functional and non-fu \subsection{Non-functional Requirements}\label{sss:non_functional_requirements} In addition to the functional requirements, we also present non-functional requirements for \mysystem: \begin{enumerate}[label=\textbf{NFR\arabic* --},align=left] - \item \textbf{The system should enable real-time insights and visualizations.} \\ + \item \textbf{Using \mysystem should not introduce any delays in visualizations longer than 1 second.}\\ The system must work in real-time, without significant delay. + We impose 1 second delay as acceptable to datacenter engineers. + We have arrived at the 1 second threshold due to the fact that datacenter operators need time to react to sudden changes in the facility behaviour. + To expect a sub-1-second performance from human employees is unjustified. + At the same time, any delay $>1$ second can already negatively impact engineer's ability to react to different events at the same time in the correct order (\ie $>1$ delay hinders the ability of technicians to follow the specific datacenter protocols and policies during an event). The system must support datacenter operators with insights at fine-grained granularity, so that insights derived from data analysis remain accurate upon reception by datacenter operators. Without \textbf{NFR1}, \mysystem's insights will not be timely, and will be useless to datacenter operators. \item \textbf{The system should log the ingestion and processing of metrics.} \\ @@ -116,14 +124,6 @@ In addition to the functional requirements, we also present non-functional requi Without \textbf{NFR4}, the system will present overwhelming amount of information to its users, rendering it unusable. \end{enumerate} -\begin{figure}[ht] - \centering - \includegraphics[width=0.75\linewidth]{images/ref_architecture.png} - \caption[The predictive datacenter digital twin architecture.]{The predictive datacenter digital twin reference architecture. - We call the system \emph{Sunfish}. - The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months.} - \label{fig:reference_architecture} -\end{figure} \section{Overview of \mysystem Architecture}\label{ss:design_of_mysystem} As a result of the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} designed a reference architecture for a predictive datacenter digital twin. @@ -135,15 +135,28 @@ As a result of the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTET \item predictive analytics. \end{enumerate*} -The physical datacenter (I) encompasses 3 core elements important to digital twinning. +\subsection{The Physical Datacenter}\label{sss:physical_datacenter} +The Physical Datacenter (I) encompasses 3 core elements important to digital twinning. Workloads (\myCircled{1a}) include the hardware requirements of each datacenter job and the submission time. They are executed on the datacenter compute (\myCircled{1b}), which is controlled partly by the Datacenter Operators (\myCircled{1c}). Component (\myCircled{1c}), while seemingly unimportant, is crucial to the digital twin design. We envision \gls{dcdt}s as systems that contain a human-in-the-loop, which can control and overwrite the system's autonomous decisions. Datacenter Operators (\myCircled{1c}) interact with both the Servers (\myCircled{1b}), and have the ability to overwrite the potential autonomous digital twin decisions, stemming from component (\myCircled{2c}), the System Knobs. +\begin{figure}[t!] + \centering + \includegraphics[width=0.75\linewidth]{images/ref_architecture.png} + \caption[The predictive datacenter digital twin architecture.]{The predictive datacenter digital twin reference architecture. + We call the system \emph{Sunfish}. + The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months.} + \label{fig:reference_architecture} +\end{figure} +\subsection{The Digital Thread}\label{sss:digital_thread} The Digital Thread (II) is a novel contribution from \Cref{s:background}. +It is a crucial element of our architecture. It separates the physical world from the virtual world, and contains components that do not belong to either twin, or belong to both twins. +Otherwise, it would be unclear how to model the data flow between the physical and the digital twin (\ie which elements take part in the information exchange). +Conceptually separating the physical and virtual twin makes it easier to comprehend and compartmentalize the design. It contains the Interactive Dashboard (\myCircled{2a}), the Message Broker (\myCircled{2b}), and System Knobs (\myCircled{2c}). To fulfill the functional requirements of our system, we incorporate element (\myCircled{2a}), the Interactive Dashboard to our system. @@ -152,12 +165,21 @@ The Interactive Dashboard (\myCircled{2a}) allows datacenter operators to see th The Message Broker (\myCircled{2b}) is a crucial component to the reference architecture, because it facilities the physical twin $\rightarrow$ virtual twin connection. A low-latency, high-throughput message broker partly meets our functional requirements to enable arbitrary amounts of telemetry data transfer. -We elaborate on the specific components that make up the message broker (\myCircled{2b}) in \Cref{sss:message_broker}. +The Message Broker (\myCircled{2b}) is a component is slightly more complex, and necessities a separate diagram. +In \Cref{fig:message_broker} we present the composite elements that make up the Message Broker. +In particular, the \emph{schema registry} and the \emph{connector manager} play a crucial part in fulfilling the functional requirements of our system. +The \emph{schema registry} allows the telemetry producer to submit \emph{any} data format for sending (and storing) the telemetry data.The registry is responsible for detecting what kind of format is the data sent in, and automatically adjusting the schema within the \emph{data pipeline}. +The connector manager is responsible for joining multiple distinct services to a single data pipeline. +In the reference architecture, the consumers would constitute the Database (\myCircled{3a}), the Cache (\myCircled{3b}), and the Interactive Dashboard (\myCircled{2a}). +What is remarkable about the connector manager is the ability to swiftly connect more consumers to the system. +This way, the predictive digital twin can facilitate multiple different types of analytics engines or techniques. +Additionally, the setup in \Cref{fig:message_broker} is currently standard industry practice for large software deployments. The System Knobs (\myCircled{2c}) represent the different cogs within the datacenter software and hardware (\myCircled{1b}) that can be adjusted during runtime (\eg to optimize \gls{pue}, change cooling strategy, allocate compute resources). For example, System Knobs (\myCircled{2c}) within the datacenter scheduler can be tuned to schedule jobs on Servers (\myCircled{1b}) that are least likely to experience future downtime. The autonomous actions of the digital twin (the tuning of the System Knobs (\myCircled{2c})) can be further adjusted by Datacenter Operators (\myCircled{1c}). +\subsection{The Digital Twin}\label{sss:the_digital_twin_archi} In our design, we explicitly differentiate between the physical and virtual space by including the Digital Twin (III) in a separate box. The Digital Twin (III) constitutes of the long-term storage (\myCircled{3a}), short-term storage (\myCircled{3b}), the API Server (\myCircled{3c}) and the Predictive Analytics (IV) module. @@ -176,6 +198,7 @@ The API Server (\myCircled{3c}) communicates directly with the System Knobs (\my Additionally, the Physical Twin (III) can query the API Server (\myCircled{3c}) for one-shot requests (\eg to create a new datacenter prototype configuration, to request special data analysis). Moreover, the Datacenter Operators (\myCircled{1c}) can query the API Server (\myCircled{3c}) for extra insights, when necessary. +\subsection{The Predictive Analytics Engine}\label{sss:predictive_analytics_engine} The Predictive Analytics (IV) module is an extensible part of the reference architecture, enabling different kinds of predictive analysis. In our design, to facilitate meaningful predictions we incorporate an Event-driven Simulator (\myCircled{4a}), Analytics Engine (\myCircled{4b}), and a Monitoring Service (\myCircled{4c}). @@ -197,15 +220,70 @@ Any discrepancies are communicated to the Analytics Engine (\myCircled{4b}) for \caption[The detailed view of the Message Broker.]{The detailed view of the Message Broker (\myCircled{2b}) from \Cref{fig:reference_architecture}.} \label{fig:message_broker} \end{figure} -\section{The Digital Thread and Predictive Analytics}\label{ss:detailed_design} -\subsection{Message Broker}\label{sss:message_broker} -The Message Broker (\myCircled{2b}) is a component is slightly more complex, and necessities a separate diagram. -In \Cref{fig:message_broker} we present the composite elements that make up the Message Broker. -In particular, the \emph{schema registry} and the \emph{connector manager} play a crucial part in fulfilling the functional requirements of our system. -The \emph{schema registry} allows the telemetry producer to submit \emph{any} data format for sending (and storing) the telemetry data.The registry is responsible for detecting what kind of format is the data sent in, and automatically adjusting the schema within the \emph{data pipeline}. -The connector manager is responsible for joining multiple distinct services to a single data pipeline. -In the reference architecture, the consumers would constitute the Database (\myCircled{3a}), the Cache (\myCircled{3b}), and the Interactive Dashboard (\myCircled{2a}). -What is remarkable about the connector manager is the ability to swiftly connect more consumers to the system. -This way, the predictive digital twin can facilitate multiple different types of analytics engines or techniques. -Additionally, the setup is \Cref{fig:message_broker} is currently standard industry practice for large software deployments. -\section{Discussion}\label{ss:design_discussion} + +\section{Requirement Validation}\label{ss:requirement_validation} +%How does this model satisfy the requirements? +%How did you arrive at this specific model? +In this section we describe the rationale behind the different design decisions of \mysystem. +To show that our model satisfies the functional and non-functional requirements, we map each component of \mysystem onto the requirements it fulfills. +\input{sources/requirements_validation.tex} + +The Interactive Dashboard (\myCircled{2a}) supports (\textbf{FR2}), as it allows to display the failures detected to datacenter operators; it is the core of (\textbf{FR4}) because descriptive analytics are synonymous to visualizations. +It enables (\textbf{FR5}) because the predictive insights can be displayed directly through the dashboard to the datacenter operators. +Operators can then overwrite the autonomous decisions performed by the \gls{dt}. +Dashboards are the current industry practice for clear data visualization (\textbf{FR7}). +In our design, we ensure (\textbf{NFR1}) is in a two-fold way. +Firstly, we ensure a direct connection between the Interactive Dashboard (\myCircled{2a}) and the Message Broker (\myCircled{2b}). +As the Message Broker (\myCircled{2b}) is capable of accommodating multiple consumers, the Interactive Dashboard (\myCircled{2b}) receives the same telemetry at the same rate as the live prediction engine. +Secondly, through a direct connection to the Database (\myCircled{3a}), the Interactive Dashboard (\myCircled{2a}) can instantly retrieve all historical patterns. +Thus, we meet (\textbf{NFR1}). +(\textbf{NFR2}) and (\textbf{NFR3}) are met by including in the system implementation a dashboard that enables logging of metrics. + +The Message Broker (\myCircled{2b}) indirectly fulfills (\textbf{FR1}). +A good and robust message broker alleviates potential bottlenecks that might stem from large data transfer. +My including a dedicated message broker, we ensure the bottle neck when twinning is not \mysystem. +As the Message Broker (\myCircled{2b}) is a vital component in the communication between the datacenter operators and the physical twin, it indirectly enables (\textbf{FR2}), (\textbf{FR3}), (\textbf{FR4}), (\textbf{FR5}). +In the other direction, to ensure that the \gls{dcdt} can ingest any amount of data, the Message Broker (\myCircled{2b}) is needed for (\textbf{FR6}). +The non-functional requirements are met by including a state-of-the-art message broker from either academia or the industry in the implementation. + +The inclusion of System Knobs (\myCircled{2c}) in the reference architecture fulfills (\textbf{FR2}). +With autonomous actions, the \gls{dt} can alter the operation of the datacenter based on the detected failures. +Failure detection must support datacenter operators in meeting \gls{sla}s. +Autonomous actions, enabled by System Knobs (\myCircled{2c}) help achieve that. +(\textbf{FR4}) and (\textbf{FR5}) are met by ensuring the insights that come from the \gls{dt} can be automated with the system knobs. +Supporting both predictive and descriptive analytics is core to the system knobs. +(\textbf{NFR4}) is met as the existing digital twin deployments often contain (indirectly) the System Knobs (\myCircled{2c}) element. +Including it in our design adheres to best design practice. +Importantly, (\textbf{NFR5}) is fulfilled together by creating a predictive analytics engine that is capable of intelligent insights and system knobs. +In order for the predictive insights to be realized, systems knobs are needed. + +The Database (\myCircled{2c}) meets (\textbf{FR3}), (\textbf{FR4}), (\textbf{FR5}), (\textbf{FR6}) and (\textbf{NFR2}). +Trivially, including a simple \gls{dbms} in the architecture fulfills (\textbf{FR3}). +For the different types of analytics (\textbf{FR4}), (\textbf{FR5}), the Database (\myCircled{2c}) enables insights from historical patterns. +Database are the current state-of-the-art in storing telemetry (\textbf{FR6}). +They are designed to hold arbitrary volume of data. +Lastly, (\textbf{NFR2}) is met by storing in the database the logs from both the \gls{dcdt} and the physical datacenter. + +The API Server (\myCircled{3c}) ensures (\textbf{FR4}), (\textbf{FR5}), (\textbf{FR2}) and (\textbf{NFR3}). +The server, which facilitates the insights that come into the System Knobs (\myCircled{2c}), by proxy enables (\textbf{FR4}) and (\textbf{FR5}). +To detect failures, some sort of notification daemon is needed for the datacenter operators, not just the dashboard. +The API Server (\myCircled{3c}) fulfills that role, ensuring (\textbf{FR2}). +The API server (\myCircled{3c}), specifically a server using the \gls{http} protocol is a state-of-the-art approach to existing digital twin communication (\textbf{NFR3}). + +The Cache (\myCircled{3b}) ensures the same requirements as the Database (\myCircled{3a}), with a couple of exceptions. +Namely, it also ensures (\textbf{NFR1}) and (\textbf{NFR3}). +Caching is needed for sub 1-second visualizations, and the inclusion of a cache for in-band data analytics is the current community practice. + +In our design, we include an Event-driven Simulator (\myCircled{4a}). +This ensures (\textbf{FR1}), (\textbf{FR2}), (\textbf{FR4}), (\textbf{FR5}) and all non-functional requirements except (\textbf{NFR2}). +Using a discrete-event simulator ensures that our system can handle workloads that are representative of the simulator at use. +Simulation, a method that is widely accepted by the scientific community, is one way \mysystem can ensure (\textbf{FR1}). +Detecting failures, enabling both descriptive and predictive analytics depends on how good of a simulator (\myCircled{4a}) is. +In principle, a robust, holistic simulator incorporated into the predictive engine is capable of meeting (\textbf{FR4}), (\textbf{FR2}) and (\textbf{FR5}). +Simulation also allows to model large workloads and to interact with the simulation in real-time (\textbf{NFR1}), it is considered a state-of-the-art approach to datacenter modelling (\textbf{NFR3}) and, with good data analytics provides enough data for insights at varying levels of confidence (\textbf{NFR4}). + +The Analytics Engine component (\myCircled{4b}) and the Monitoring Service (\myCircled{4c}) fulfill the same set of requirements, namely (\textbf{FR1}), (\textbf{FR2}), (\textbf{FR4}) and (\textbf{FR5}). +Additionally the Analytics Engine (\myCircled{4b}) also fulfills (\textbf{NFR4}), as it provides the capability to differentiate between different levels of confidence. +Both components are working together in unison, closely watching and reacting to the metrics from the real datacenter. +Hence, they fulfill (\textbf{FR2}), (\textbf{FR4}) and (\textbf{FR5}) and are crucial to the \gls{dcdt} operation. +(\textbf{FR1}) is fulfilled by ensuring both the Analytics Engine and the Monitoring Service are not the source of any bottlenecks during the system implementation. diff --git a/content/evaluation.tex b/content/evaluation.tex index b17fd9d..4978206 100644 --- a/content/evaluation.tex +++ b/content/evaluation.tex @@ -2,11 +2,13 @@ \section{Overview}\label{ss:evaluation_overview} \begin{mynote} - The contribution of this chapter is two-fold: + Addressing RQ$_3$ together with the previous chapter, the contribution of this chapter is two-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] \item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}. - \item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}. + + \item We provide a comprehensive set of experiments to evaluate \mysystem~(\Cref{ss:experiment1,ss:experiment2}). + \end{enumerate} Our findings indicate: \vspace{-0.2cm} @@ -14,59 +16,69 @@ \item Digital twinning can be used for failure detection to the benefit of datacenter operators. \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime. \item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime. - \item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated tasks. + \item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated during a workload. + \item \mysystem can estimate both the best-case and worst-case number of total tasks terminated due to failures during a datacenter workload. \end{enumerate} \end{mynote} -\section{Experimental Setup}\label{ss:experimental_setup} - +\section{Novel Evaluation Technique}\label{ss:experimental_setup} \begin{figure}[t] \centering - \includegraphics[width=0.8\linewidth]{images/novel_eval_method.png} - \caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.} + \includegraphics[width=0.74\linewidth]{images/novel_eval_method_improved.png} + \caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. In this approach, both the physical and virtual datacenter are emulated using discrete-event simulation.} \label{fig:novel_eval_method} \end{figure} +In the this section, we evaluate \mysystem using \code{OpenDC} and all the necessary software (\ie \code{Grafana}, \code{PostgreSQL}, \code{Confluent Kafka}. +A na\"ive experimental approach would consist of connecting \mysystem to a physical datacenter. +However, as explained earlier in this work, running large experiments on real-world data centers is expensive and time-consuming~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +Additionally, we do not have access to a datacenter. +To alleviate this problem, we again utilize simulation for the benefit of evaluating \mysystem, and \gls{dcdt}s in general. -In this section we describe the technical setup used to evaluate \mysystem. -However, \Cref{fig:reference_architecture} assumes the system designer is capable of connecting the digital twin directly to the datacenter. -This raises a problem, we cannot just go and test digital twins on large systems, because we do not have large systems at hand. -Moreover, real-world experimentation is costly and unsustainable in the long run~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -To overcome this problem, we present a novel datacenter digital twin method capable of evaluating a \gls{dcdt} without the physical datacenter. -\Cref{fig:novel_eval_method} details our approach. - -In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. -In our implementation, this is a second \code{OpenDC} -process (see \Cref{fig:implementation}). -The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. -For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}. +In this non-trivial approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. +For our implementation, we decided to use a second \code{OpenDC} +process. +A schematic overview of this experimental technique can be seen in \Cref{fig:novel_eval_method}, where the \textcolor{Red}{\ding{110} \textbf{red}} box highlights our initial approach, and the \textcolor{Green}{\ding{110} \textbf{green}} box shows the novel evaluation method. +The advantages of this approach are as follows: +\begin{enumerate*}[label=(\arabic*)] + \item The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. + \item The environmental footprint is smaller. As a result, \mysystem is more sustainable. + \item Datacenter engineers can fine-tune the digital twin using this setup before pairing it with the production environment. + In result, engineers can effectively use our setup to test and debug the digital twin before installation. + \item With this method, \gls{dcdt} research becomes more accessible both to the scientific community and higher education. +\end{enumerate*} -The technical setup used for all experiment adheres to the \code{OpenDC} -documentation (see Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}). -The workload trace used for all experiments comes from BitBrains~\cite{DBLP:conf/ccgrid/ShenBI15}. +\section{Experiment Parameters}\label{ss:experiment_parameters} +In this section we detail the experimental setup that follows most of the experiments. +The workload trace used for all experiments is based on a one-month SURF workload trace~\cite{DBLP:conf/ccgrid/ShenBI15}. In the experiments we model a Dutch SURF datacenter for scientific computing. The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}. -The scheduling policy for all experiments is the \code{FilterScheduler} which considers the RAM and CPU capacity for choosing hosts to run tasks on. -This scheduling policy is also used by \code{SmartScheduler}, as outlined in \Cref{ss:programming}, albeit with modifications to enable the system knobs to take autonomous action. +This setup ensures a realistic experiment scenario that is representative of a real-world setting. + +The \code{OpenDC} that models the physical datacenter uses the \code{SmartScheduler}. +The event driven simulator within the predictive analytics engine of \mysystem uses the \code{FilterScheduler} by default, unless otherwise specified. -In all experiments we use either \emph{failure traces} or \emph{failure models}. -For a brief explanation on the differences between the two, consult \Cref{sss:failures}. +For the experiments that model datacenter compute failures, we use \emph{failure traces} or \emph{failure models}. +For a detailed explanation on the differences between the two, consult \Cref{sss:failures}. +In short, failure traces contain the data necessary to reproduce the outages experienced by the real datacenter during the simulation. +A trace contains 3 elements, failure interval, failure duration and failure intensity (see \Cref{tab:failure_traces_overview}). +A failure model is characterized as as a statistical distribution of failures based on scientific research. +It is akin to the output of an algorithm that determines when failures should happen, rather than a real-world trace~\cite{DBLP:journals/jpdc/JavadiKIE13}. In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -We chose a diverse range of failure models, based on the mean failure intensity in each trace (indicated in parentheses). -As a result, we chose the traces from: \begin{enumerate*}[label=(\Roman*)] - \item Gmail (53.26\%), - \item WhatsApp (57.97\%), - \item YouTube (62.1\%), - \item Twitter (65\%), - \item Facebook (64\%). -\end{enumerate*} +We chose a diverse range of failure traces, based on the mean failure intensity in each trace (see \Cref{tab:failure_traces_overview}). +Mean failure intensity is a metric that shows best how severe the failures are. +The higher the failure intensity, the more hosts go down at once, resulting in more serious downtime and missed \gls{sla}s. +According to the mean failure intensity, we chose the 5 traces in \Cref{tab:failure_traces_overview} for our experiments. +\input{sources/failure_model_statistics.tex} +\input{sources/failure_models.tex} +\input{sources/failure_characteristics.tex} In \Cref{ss:experiment2} we used a failure trace from Skype. -This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}). +This is the only trace that can be paired with a corresponding failure model (see \Cref{tab:failure_models_table}). Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures. For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5. -Importantly, in our figures we do not report the standard deviation of our experiments. -This is due to the fact that \code{OpenDC} is a fully deterministic simulator, and on each simulation run, given the same random seed will produce exactly the same results. -We believe the deviation in the results of the experiments stemming only from the random number generator is not meaningful, therefore none of the figures contain the standard deviation bars. + +All the experiments were run on a commodity laptop, with an AMD Ryzen 7840U CPU containing 16, double-threaded cores and maximum frequency of 5.13 GHz, 32GB of DDR5 RAM. +By showing that \mysystem works using this setup, we encourage scientists and academia (stakeholder \textbf{S3}) to freely experiment with \mysystem. \section{Experiment 1: Failure Detection}\label{ss:experiment1} \begin{figure}[t] @@ -75,107 +87,198 @@ We believe the deviation in the results of the experiments stemming only from th \caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.} \label{fig:red_yellow_alarms} \end{figure} +For the first experiment we adopt the idea introduced by Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +We show 2 different ways a \gls{dcdt} can notify datacenter engineers of detected failures. + +\subsection{Context}\label{sss:failure_detection_context} +In this section we try to provide the rationale behind this experiment. +Failure detection is a primary use-case for \gls{dcdt}s~\cite{DBLP:journals/computer/AthavaleBBMMPS24,DBLP:usdoe/report/AP26894,DBLP:conf/sc/TaheriBPRHDEWPM24,DBLP:journal/IJAE/Teugel2011}. +Failures in a datacenter can arise from a number of problems, \eg software configuration issues, power outage, network congestion~\cite{DBLP:report/AnnualOutageAnalysis2025}, and must be immediately detected to ensure their negative impact is minimized. +Some failures can have negligible consequences (around 38\% of all failures are negligible~\cite{DBLP:report/AnnualOutageAnalysis2025}), whilst other can cause severe downtime and loss of millions of \$USD~\cite{DBLP:report/AnnualOutageAnalysis2025}. +By differentiating between insignificant and severe failures, datacenter operators can focus their efforts on addressing the most impactful problems, minimizing missed \gls{sla} and ensuring swift datacenter operation. -The purpose of this experiment is two fold: \begin{enumerate*} - \item to show our system works correctly +In \Cref{ss:experiment1} we model such a scenario. +For example, imagine the following sequence of events: +\begin{enumerate}[label=\textbf{\arabic*.},align=left] + \item The datacenter receives a scheduled workload. + \item To predict what kind of failures might occur during the workload, we ask the digital twin to run the workload first. + \item We have no \emph{a priori} knowledge of the workload type, so we can only estimate the distribution of failures. To do this, the \gls{dcdt} will run the workload, assuming failures follow \eg a normal distribution. (Unlike traditional analytical tools, the distribution can be selected arbitrarily, even changed during the run, and the predictive results do not rely on underlying statistical assumptions.) + \item After running the simulation, the \gls{dcdt} can analyze the results and compare them with the running workload. + \item If the running workload has more failures than what the \gls{dcdt} predicted, with the latter representing the known errors inherent in typical datacenter operations, the discrepancy is used to notify the datacenter operators -- + something is wrong. + Through the real-time feedback loop, the \gls{dcdt} not only notifies the datacenter operators, but can also help understand where and how what is happening in reality is different from the simulation. +\end{enumerate} + +In our experiment, we notify the datacenter operators in two cases: +(1) we send a yellow alarm if the 80\% threshold is reached, that is, the simulated results get within 80\% of the predicted threshold for the number of failures; +(2) we send a red alarm if the 90\% threshold is reached. + +The purpose of this experiment is two-fold: \begin{enumerate*}[label=(\arabic*)] + \item to show our system works correctly, and \item to show our system fulfills the functional and non-functional requirements. \end{enumerate*} -To this end, we replicate an experiment from Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system. -The experimental setup is as defined in \Cref{ss:experimental_setup}. -The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)] +The technical experimental setup can be described as follows: \begin{enumerate*}[label=(\arabic*)] \item firstly, we use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload. \item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm) \item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached. \item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold. \end{enumerate*} -The results are in \Cref{fig:red_yellow_alarms}. -\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. -What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems. -Importantly, the more failure-intense the trace, the more alarms are raised on behalf of the digital twin. +The results are in \Cref{fig:red_yellow_alarms,fig:alarms_vs_failures,fig:failure_detecton_rate}. \begin{figure}[t] \centering \includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf} - \caption[Total number of failures versus numbe rof alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a BitBrains workload in the SURF-SARA cluster. The failure traced used in this experiment models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.} + \caption[Total number of failures versus number of alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a SURF month workload in the SURF-SARA cluster. The failure trace used in this plot models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.} \label{fig:alarms_vs_failures} \end{figure} -Additionally \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. -In the figure we can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. + +\subsection{Discussion}\label{sss:experiment1_discussion} + +\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. +In the figure, we can see that as the mean failure intensity rises, so does the total number of failures detected. +This behaviour is expected -- the higher the mean failure intensity, the more severe the failures, the more notifications to datacenter operators are sent. +The more failure-intense the trace, the more alarms are raised on behalf of the digital twin. +What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems (\ie the number of yellow alarms is smaller than the number of red alarms). + + +To offer a second perspective, we also provide the results depicted in \Cref{fig:alarms_vs_failures}. +In this figure, we show the total number of alarms raised and the ground truth (how many failures occurred in reality.) +We can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. For this visualization, we combined both the red and yellow alarms into a single metric. +In short, \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. -However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. -The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate. -What is surprising is Taheri \etal report almost negligible false positive rate and of their system. -Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate. -In our experiment, the numbers differ significantly. -\begin{figure}[t] +\begin{figure}[!t] \centering - \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf} + \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate2.pdf} \caption[Failure detection rate overview.]{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}). Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. We believe this is due to the irreconcilable differences between our experimental setups.} \label{fig:failure_detecton_rate} \end{figure} -\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%. -Compared to the DyTwin deployment, the difference is staggering. -However, the discrepancy stems from the fact in our setup we differentiate between different types of failures. -This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe. -Unusually, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace. + +However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. +The rate is simply calculated as the anomalies detected correctly over the true anomaly count~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. (Note: The correct term for this should be the \emph{anomaly detection ratio}.) +Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection ratio, defined as the sum of Red and Yellow alarm-counts, over the Total Failure count. +What is surprising is Taheri \etal report almost negligible false positive rate and of their system. +Moreover, after testing with DyTwin in a specific experimental setup, Taheri \etal conclude their proposed method achieves 100\% anomaly detection rate. +In our experiment, the numbers differ significantly; we attribute this to irreconcilable differences between our experimental setups. + + +% Our results are also different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +% We believe this is due to the irreconcilable differences between our experimental setups. + + + +\Cref{fig:failure_detecton_rate} shows the mean failure detection ratio to be around 12\%. +Compared to the DyTwin results reported by Taheri~\etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}, the difference is significant. +We stipulate that the root of the discrepancy is that our setup differentiates between different types of failures, a capability that is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +As a new result, we interpret \Cref{fig:failure_detecton_rate} as showing that on average 12\% of failures in the workload are severe. +Surprisingly, the WhatsApp failure detection ratio is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2\textsuperscript{nd} least failure-intensive trace. \section{Experiment 2: Failure Prediction}\label{ss:experiment2} -In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics. +In \Cref{ss:experiment1} we showed \mysystem is capable of incorporating descriptive analytics. Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures. -In this section we try to show \mysystem can additionally work well together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. - -\begin{figure}[ht] - \hspace{-0.8cm} - \begin{minipage}[b]{0.45\textwidth} - \centering - \includegraphics[width=1.2\linewidth]{images/failure_likelihood.pdf} - \end{minipage} - \hspace{1.2cm} - \begin{minipage}[b]{0.45\textwidth} - \centering - \includegraphics[width=1.2\linewidth]{images/conceptual_experiment.pdf} - \end{minipage} - \caption[The results of Experiment 2.]{Left figure shows the potential failure distribution likelihood to approximate the true failure distribution. - Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.} +In this section, we aim to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. + +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/failure_likelihood.png} + \caption[The failure model likelihood over time.]{How closely a \emph{failure model} approximates the real failure distribution (the Skype failure trace) during runtime. The vertical axis is linear but does \textit{not} start at 0.} \label{fig:failure_likelihood} \end{figure} -\input{sources/failure_models.tex} \subsection{Context}\label{sss:context_experiment2} -In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods. -Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions. -For the task at hand, we chose the Skype trace, because it is supported by 4 different failure models, based on past Skype workload data. -These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}. -The Skype trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -The goal was to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes. - +To predict when a host failure might occur, a straightforward approach is to use long-established statistical methods. +Our goal is to approximate the real failure distribution of a workload, using past data and relevant statistical distributions. +For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models~(see \Cref{tab:failure_models_table}, sourced from~\cite{DBLP:journals/jpdc/JavadiKIE13}), based on past Skype workload data. +%These 4 statistical distributions, published in a peer-reviewed journal are in . +The Skype Failure Trace model was sourced from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines \emph{before} it crashes. Initial experiment results were unpromising. Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly). To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution. Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time. -The results are in \Cref{fig:failure_likelihood}. +The results are shown in \Cref{fig:failure_likelihood}. -In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$. -Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. -Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. -This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis. +% In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +% Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. +% Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +% \Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. + +% \begin{figure}[!t] +% \centering +% \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} +% \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} +% \label{fig:conceptual_experiment} +% \end{figure} -Undeterred, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. +%Despite the aforementioned results, we set out for a different solution to show +We also show \mysystem is capable of incorporating a predictive analytics engine. Instead, we designed a \emph{conceptual experiment}. In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy. Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment. -The results are in \Cref{fig:failure_likelihood} on the right side. -\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly. +The results are shown in \Cref{fig:conceptual_experiment}. + +\subsection{Discussion}\label{sss:experiment2_discussion} + +In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +For any failure model, at any time interval, it is almost as likely to accurately model the actual failures as the other models. +Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +\Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. + + +\begin{figure}[!t] + \centering + \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} + \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} + \label{fig:conceptual_experiment} +\end{figure} + +\Cref{fig:conceptual_experiment} shows that using a perfect predictive-analytics engine, \mysystem is capable of lowering the total number of failures significantly. +In this experiment, we used only 3 failure traces from user reports of Twitter, WhatsApp, and Gmail. +Importantly, we assume in this experiment a perfect precognition of what failures might occur and when. +While unrealistic in a practical scenario, this assumption serves to approximate the potential gains from incorporating a predictive analytics engine with \mysystem. +The results are only indicative. +In the figure we can see that the mean number of failures in a ``physical datacenter'' (\ie \code{OpenDC}) is much higher without the insights of \mysystem. + +\section{Experiment 3: Failure Exploration}\label{ss:experiment3} + +For this experiment, we adapt the idea of using multi-modal simulation from Nicolae \etal~\cite{DBLP:conf/cf/NicolaeNTI26} to estimate failures with \gls{dcdt}. + +\subsection{Context}\label{sss:context_experiment3} +In this experiment we use the \gls{dcdt} to explore the number of terminated tasks when different failure injection algorithms are used. +The primary objective of this experiment is to identify the worst-case scenario and best-case scenario of running a workload prone to failures. + +Nicolae \etal explore how multi-modal simulation can provide additional insight into datacenter operation and can potentially halve the error of singular models~\cite{DBLP:conf/cf/NicolaeNTI26}. +For our work, we adopt a similar approach to failure prediction. +Imagine the following sequence of events: +\begin{enumerate}[label=\textbf{\arabic*.}, align=left] + \item The datacenter receives a scheduled workload, and the operators want to know the worst-case scenario and best-case scenario of how severe failures they can expect. + \item Datacenter engineers ask the digital twin to run the workload first with different failure injection algorithms. + \item After running the simulation, the \gls{dcdt} can provide the maximum and minimum number of expected failures. + \item By continuously monitoring the datacenter during the runtime of the real workload, the operators can use the estimate to make more informative decisions. +\end{enumerate} +\begin{figure} + \centering + \includegraphics[width=0.85\linewidth]{images/victim_selector_comparison.pdf} + \caption[Comparison of different victim selector algorithms.]{Comparison of different algorithms used for injecting failures into the simulated digital twin to explore the worst-case and best-case scenarios.} + \label{fig:victim_selector_comparison} +\end{figure} +For this experiment we used a SURF month trace running on the SURF-SARA cluster, as explained in \Cref{ss:experiment_parameters}. +Additionally, we use the Gmail failure trace to inject failures, and use 3 different algorithms for selecting the hosts to fail: \code{CpuUtilSelector}, \code{RoundRobinSelector}, \code{RandomVictimSelector}. +The results are depicted in \Cref{fig:victim_selector_comparison}. + +\subsection{Discussion}\label{sss:discussion_experiment3} +\Cref{fig:victim_selector_comparison} shows using different victim selection algorithms can effectively present the worst-case and best-case number of tasks terminated before running the real workload. +The victim selector that achieves the most tasks terminated is the \code{CpuUtilSelector}, which stops the hosts with the highest \gls{cpu} utilization. +It reaches over 700 terminated tasks (709) in total. +The \code{Random} selector, which randomly picks the hosts to stop, achieves the second-highest number of tasks terminated, at around 450. -\section{Experiment 3: Additional Experiment}\label{ss:additional_experiment} +The most optimistic scenario for datacenter operators tested here is the \code{RoundRobin} selector, which picks the hosts to terminate in a round-robin fashion. +No more than 130 tasks are terminated using this fault injection algorithm. -\section{Discussion}\label{ss:discussion_evaluation} +To summarize, this experiment shows that \mysystem can be used to explore different failure algorithms for the benefit of datacenter operators. +Using \mysystem, on-site engineers can effectively estimate the best-case and worst-case number of terminated tasks during a workload. diff --git a/content/implementation.tex b/content/implementation.tex index 8db19cf..79c5191 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -1,16 +1,27 @@ \chapter{Implementation of \mysystem}\label{s:implementation} -In this chapter we describe the implementation of \gls{my_system}. -The main contribution of this chapter towards answering \emph{RQ3} is the prototype of \gls{my_system}. -After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. + +\section{Overview}\label{ss:overview_implementation} + +\begin{mynote} + Addressing RQ$_3$ together with the next chapter, the contribution of this chapter is two-fold: + \vspace{-0.2cm} + \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] + \item We implement the real-world prototype of \mysystem (see \Cref{ss:detailed_implementation_overview}) realizing key features of the design. + + \item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming}). + + \end{enumerate} +\end{mynote} + +\section{Detailed Implementation of \mysystem}\label{ss:detailed_implementation_overview} +In this section, we describe the detailed implementation of \gls{my_system}. +After completing it, the reader should understand the technical decisions, choice of tools, and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}. -To understand why \gls{my_system} it is crucial to provide a holistic view on the prototype. -Therefore, the rest of the chapter is structured in a top-down approach: \Cref{ss:implementation_overview} +To understand \gls{my_system} it is crucial to provide a holistic view on the prototype. +Therefore, the rest of this chapter is structured in a top-down approach: \Cref{ss:detailed_implementation_overview} presents the rationale for using the specific software packages, \Cref{ss:data_flow} shows the flow of data within the system, and \Cref{ss:extensions} details the different modifications and new software extensions to \code{OpenDC}. Lastly, \Cref{ss:programming} carefully explains the design decisions behind the major Python modules. - -\section{Overview}\label{ss:implementation_overview} - \begin{figure}[t] \centering \includegraphics[width=0.85\linewidth]{images/implementation.pdf} @@ -21,14 +32,21 @@ Lastly, \Cref{ss:programming} carefully explains the design decisions behind the At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications. The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}. -In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. +To facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}. To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}. To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}. The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}. -\code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards. -We posit it is crucial to include a user-friendly \gls{ui}. +\begin{figure}[t] + \centering + {\includegraphics[width=0.9\linewidth]{images/grafana_panel.pdf}} + \caption[Example Grafana dashboard.]{Example Grafana Dashboard with downtime plotted against time.} + \label{fig:example_grafana_dashboard} +\end{figure} + +\code{Grafana} (\myCircled{2a}) is a state-of-the-art industry tool to visualize dashboards. +We posit it is crucial to include a user-friendly performance dashboard. A number of previous publications on \gls{dt}s find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}. We chose \code{Grafana} (\myCircled{2a}) instead of other software packages because of its seamless integration with \code{PostgreSQL} (\myCircled{3a}). \code{Grafana} (\myCircled{2a}) provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself. @@ -61,13 +79,13 @@ For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a t \begin{figure}[t] \centering \includegraphics[width=\linewidth]{images/flow_diagram.png} - \caption{The data flow within \gls{my_system}.} + \caption{The data flow within \mysystem.} \label{fig:flow_diagram} \end{figure} -\section{Data Flow}\label{ss:data_flow} +\section{Main Data Flow}\label{ss:data_flow} -In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram. +In this section, we describe the data flow within \Cref{fig:implementation} using a separate diagram. Efficient data flow is of utmost importance to \gls{dt}s. In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}. In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data. @@ -95,7 +113,6 @@ Should anything unusual occur, the \code{Monitoring Service} notifies the \code{ Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights. All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}). - \begin{figure}[t] \input{sources/listing_schema.tex} \end{figure} @@ -105,20 +122,44 @@ Due to the massive volume of data incoming from the physical datacenter, the \co \code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}). On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}). This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out. - \begin{figure}[t] \input{sources/listing_sinks.tex} \end{figure} + +\section{The \code{OpenDC} Scheduling Paradigm}\label{ss:opendc_scheduling} +In this section we introduce the scheduling paradigm of \code{OpenDC}. +\code{OpenDC}, a robust datacenter simulator uses discrete-event simulation. +``Discrete-event simulation models the operation of a system as a (discrete) sequence of events in time''~\cite{Wikipedia:article/DiscreteEventSimulation}. +Colloquially, it is akin to calling an \code{Update()} method on a set of objects to model changes in the simulator. +Scheduling in \code{OpenDC} also works using this method. +\Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation. + +A task (\myCircled{1}) is represented by its submission time, duration and computational requirements. +To be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). +The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time. +\code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. +In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}. +In the \code{FilterPolicy}, a series of \code{HostFilter}s (\myCircled{3}) and \code{Host Weighter}s (\myCircled{5}) are used to find the matching host for the task. +In this project, the different filters and weighters take into account the default metrics (\ie \gls{cpu} capacity, \gls{ram} capacity, number of \gls{cpu} cores). +The host itself is represented as a \code{HostView} class (\myCircled{6}). +Importantly, the \code{HostView} class does not serve to simulate the behaviour of the host, but to provide a interface for the \emph{current state} of the host. +For actual computation, the \code{SimHost} (\myCircled{8}) class is used. +The \code{SimHost} object is created via the \code{HostProvisioningStep} class (\myCircled{7})~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. + + \section{Extensions to \code{OpenDC}}\label{ss:extensions} \code{OpenDC} is a state-of-the-art datacenter simulator. -In order to turn it into a \gls{dt}, we have made several design decisions and extensions. +To turn it into a \gls{dt}, we have made several design decisions and extensions. \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{SmartScheduler}}\\ The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions. It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights. For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}). + It \emph{replaces} the \code{FilterScheduler} (\myCircled{4}) in \Cref{fig:scheduling_in_opendc}. + Importantly, it is functionally almost exactly the same as the \code{FilterScheduler}, with the exception that it can change the scheduling outcome based on the information from the \gls{dt}. + It is a \code{FilterScheduler} with an attached network socket. \item \textbf{\code{KafkaMonitor}}\\ The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}). We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic. @@ -126,7 +167,24 @@ In order to turn it into a \gls{dt}, we have made several design decisions and e \item \textbf{\code{HTTPClient}}\\ The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter. We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice. + \item \textbf{\code{CpuUtilVictimSelector}}\\ + To ensure \code{OpenDC} can model different failure injection algorithms, we added new methods for selecting the hosts to be stopped. + The \code{CpuUtilVictimSelector} selects hosts based on their current \gls{cpu} utilization (\ie the hosts with the highest utilization are stopped first). + \item \textbf{\code{RoundRobinVictimSelector}}\\ + The \code{RoundRobinVictimSelector} is another algorithm to used to inject failures into hosts. + This method works in a round robin fashion, using the alphabetical order of the hosts (\ie by name). + For example, in a scenario hosts \code{H-01} to \code{H-20} are stopped first, and then \code{H-20} to \code{H-40} are stopped next, \etc + \item \textbf{\code{RandomVictimSelector}}\\ + This is the default \code{OpenDC} victim selection algorithm. + It randomly chooses which host to stop when injecting failures. + This component is left unmodified, but we include it here for the purposes of listing all the failure injection algorithms we use together. \end{enumerate} +\begin{figure}[t] + \centering + \includegraphics[width=\linewidth]{images/scheduling_opendc.png} + \caption[OpenDC scheduling paradigm.]{The scheduling paradigm in \code{OpenDC}. Adapted from Musc{\u a} \etal~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. The highlighted \textcolor{Orchid}{\ding{110} \textbf{Filter Scheduler}} is the component that is substituted by the \code{SmartScheduler}.} + \label{fig:scheduling_in_opendc} +\end{figure} \section{Python Modules}\label{ss:programming} @@ -137,12 +195,12 @@ For future work, we envision a system that implements the reference architecture \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{AnalyticsEngine}}\\ - The \code{AnalyticsEngine} module is necessary in order to encapsulate the logic of data preprocessing and analysis from monitoring. + The \code{AnalyticsEngine} module is necessary to encapsulate the logic of data preprocessing and analysis from monitoring. This component can contain capabilities for different statistical metrics, subject to \gls{dt}s focus. In \gls{my_system} \code{AnalyticsEngine} continuously checks whether the incoming datacenter sensor readings exceed different thresholds. For example, the \code{AnalyticsEngine} is capable of calculating a similarity score $S$ between potential failure distributions and the true failure distribution. \item \textbf{\code{HTTPServer}}\\ - The \code{HTTPServer} is crucial for interrupting the operation of the datacenter in order to adjust its operation or offer insights. + The \code{HTTPServer} is crucial for interrupting the operation of the datacenter to adjust its operation or offer insights. It maintains a python \code{Queue} structure. The \code{Queue} \emph{producer} is the \code{AnalyticsEngine} (\grayCircled{4}). The \emph{consumer} is the \code{HTTPClient} within \code{OpenDC} (\ie the real datacenter, (\myCircled{2}), (\myCircled{1})). @@ -152,3 +210,4 @@ For future work, we envision a system that implements the reference architecture It contains a \code{while True} Python loop which contains the function call to fetch the latest changes to the \code{Redis} stream. Upon update, the \code{MonitoringService} informs the \code{AnalyticsEngine} that new data is awaiting \code{AnalyticsEngine} (\grayCircled{6}). \end{enumerate} + diff --git a/content/intro.tex b/content/intro.tex index 69322d4..85ae205 100644 --- a/content/intro.tex +++ b/content/intro.tex @@ -1,7 +1,9 @@ \chapter{Introduction}\label{s:intro} Currently, computer and network systems play a crucial part in the digital industry. -The transport, education and government sectors largely depend on digital services, which are hosted in datacenters~\cite{DBLP:journals/corr/IosupKLVG22}. -To address the recent rise in demand for computation, due to the advancements in Artificial Intelligence, managers expand datacenters with new components and more heterogeneous architectures (\eg GPUs, NPUs)~\cite{DBLP:conf/date/MilojicicFDR21}. +For example, the transport, education and government sectors largely depend on digital services, which are hosted in datacenters~\cite{DBLP:journals/corr/IosupKLVG22}. +Datacenters are complex facilities, housing thousands of servers. +Moreover, the advancements in Artificial Intelligence have sped up data datacenter expansion. +To address the recent rise in demand for computation, datacenter managers add new components and more heterogeneous architectures (\eg GPUs, NPUs)~\cite{DBLP:conf/date/MilojicicFDR21} to the already complex warehouses. However, in return datacenter complexity increases significantly. To make better operational decisions despite the massive scale, promising technologies arise such as Datacenter Digital Twins~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. @@ -12,50 +14,60 @@ Since many public services continue to move online (\eg online administration an % What is changing? In the modern \gls{ai} economy, datacenters need diverse and scalable server architectures, because inference-based workloads require more heterogeneous server components (GPUs, TPUs, NPUs \etc) to perform well. -Nowadays, datacenter operators try to meet AI-customer expectations by adding more specialized hardware~\cite{DBLP:conf/date/MilojicicFDR21}, at the cost of increased system complexity. +%Nowadays, datacenter operators try to meet AI-customer expectations by adding more specialized hardware~\cite{DBLP:conf/date/MilojicicFDR21}, at the cost of increased system complexity. In return, operating a modern datacenter warehouse with thousands of diversified servers presents a difficult challenge that requires fast and well-informed decisions from on-site engineers. -The computational requirements of \gls{ai} are expected to increase in the future~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. -Datacenter complexity will continue to grow, and it will become more difficult to manage~\cite{DBLP:conf/icdcs/IosupUVAEHTBT18}. +The computational requirements of \gls{ai} are expected to increase in the future. +\Cref{fig:explosive_growth_in_ai_computational_requirements} presents the speed and scale which drives the compute requirements of \gls{ai} models. +From 2012 to 2024, the compute power needed to train the models grew by $10^{7}$, a trend that is expected to continue~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +Because of this, datacenter complexity will continue to grow, and it will become more difficult to manage~\cite{DBLP:conf/icdcs/IosupUVAEHTBT18}. Future servers and software related services to them will include even more specialized hardware, which, while improving datacenter performance, will exhibit behaviour that is harder to predict. Already the rapid expansion of datacenters has increased the presence of service failures across all cloud services~\cite{DBLP:conf/acsos/TalluriOVTI21}. Preventing failure-caused outages in advance could help datacenter operators reduce operational costs, as over 20\% of all reported outages amount to more than 1 million US\$~\cite{DBLP:report/AnnualOutageAnalysis2025}. %Moreover, datacenter outages can have catastrophic consequences, cite Fabian. -In short, the high computational demand of \gls{ai} and the end of Dennard's scaling have resulted in the rise of larger and more heterogeneous datacenter architectures~\cite{DBLP:conf/date/MilojicicFDR21}. +%In short, the +The high computational demand of \gls{ai} and the end of Dennard's scaling have resulted in the rise of larger and more heterogeneous datacenter architectures~\cite{DBLP:conf/date/MilojicicFDR21}. Both events create a need for more careful datacenter management to tackle the unprecedented complexity and ensure availability of all cloud services. To address this new problem a concept of a datacenter \gls{dt} was proposed~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +% Explain DT here, very briefly +A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. +The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{Wikipedia:article/DigitalTwin}. +% +This thesis explores the use of digital twins to cope with increasing operational complexity in datacenters, by enabling both historical and online data to pass through the digital twin's analytical and predictive loops. -\begin{figure} +\begin{figure}[t] + \hspace{-1.75cm} \centering - \includegraphics[width=0.8\linewidth]{images/simple_dt.pdf} - \caption[Elements of the digital twin ecosystem.]{Elements of the digital twin ecosystem~\cite{DBLP:modsim24/presentation/Iosup2024} include: the insights and decisions coming from the digital twin (\myCircled{A}), the physical infrastructure (\myCircled{B}), the data coming from the physical twin telemetry (\myCircled{C}), and the digital counterpart to the physical twin (\myCircled{D}). - This thesis focuses on components (\myCircled{A}), (\myCircled{C}), and (\myCircled{D}) in this ecosystem, proposing design improvements to (\myCircled{D}, \myCircled{C}), and the feedback loop (\myCircled{A}). - } - \label{fig:simple_dt} -\end{figure} + \includegraphics[width=0.8\linewidth]{images/ai_timeline.pdf} + \caption[Explosive growth in AI computational requirements.]{Explosive growth in AI computational requirements drives datacenter upgrades (source: NVIDIA Analysis: reproduction with NVIDIA permission by~\cite{DBLP:journals/computer/AthavaleBBMMPS24}). \emph{X}-axis presents the year, and the \emph{Y}-axis presents the training compute requirement (in petaFLOPs). The plot shows the one of the reasons behind why the complexity of datacenters is rising.} + \label{fig:explosive_growth_in_ai_computational_requirements} +\end{figure} \section{Context}\label{s:context} % A digital twin is often called a virtual twin. % The communication between a physical entity and the digital twin is referred to as a digital thread. -A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. -The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{WIKI:page/DigitalTwin}. +% +% A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. +% The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{Wikipedia:article/DigitalTwin}. -Most modern \gls{dt} usages are related to prognostics and system health management~\cite{DBLP:conf/cirp/TAO2018169}. +Much modern \gls{dt} usages are related to prognostics and system health management~\cite{DBLP:conf/cirp/TAO2018169}. For example, in aerospace engineering, the \gls{dt} analyzes operational data (\eg temperature, vibration) to predict when a airplane component is likely to fail. The \gls{dt} can reliably manage the health of the physical entity by detecting fatigue cracks on aircraft wings or damage to the wind turbine blades~\cite{DBLP:journal/IJAE/Teugel2011}. This allows maintenance to be scheduled proactively, reducing unplanned downtime and preventing catastrophic failures. Forecasting future maintenance and managing the physical health of an object or facility are the prime purpose of many \gls{dt}s used in practice~\cite{DBLP:conf/AIAA/Teugel2012}. -The first mention of a \gls{dt} dates back to 2003, when Dr. Michael Grieves of Dassault Syst\'emes introduced the 3 core components of a \gls{dt}: the virtual entity, physical entity and the two-way connection (see Figure \ref{fig:simple_dt}). +The concept of a \gls{dt} began in 1960s, at the \gls{nasa}~\cite{Nature:article/Görtz2026}. +\gls{nasa} pioneered the concept to debug issues with its spacecraft. +However, the term ``digital-twin'' dates back to 2003, when Dr. Michael Grieves of Dassault Syst\'emes introduced the 3 core components of a \gls{dt}: the virtual entity, physical entity and the two-way connection (see Figure \ref{fig:simple_dt}). Due to insufficient technological foundations, little work is available on \gls{dt}s between 2003 and 2018, and it is only with the rapid growth of cloud computing, \gls{iot} and Big Data analytics that \gls{dt}s have re-emerged. Today, research is focused on bridging the gap between the long-established foundations of \gls{dt}s and new, novel applications in academia and industry, such as the \gls{dcdt}~\cite{DBLP:conf/cirp/TAO2018169, DBLP:journals/computer/AthavaleBBMMPS24}. -A \gls{dcdt} mirrors the structure, context and behaviour of a datacenter~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +A \gls{dcdt} mirrors the structure, context, and behaviour of a datacenter~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. The foundation to any digital twin is good monitoring and sensing capabilities in the physical entity. Datacenters, meet this requirement easily because they already connect hundreds of monitoring sensors. -With hundreds of gigabytes of useful information coming from distributed \gls{iot} sensors inside the warehouse, we can gain insight into failure patterns, energy usage, heat dissipation \etc +With hundreds of gigabytes of useful information coming from distributed \gls{iot} sensors inside the warehouse, we can gain insight into failure patterns, energy usage, heat dissipation, \etc What remains challenging is to connect the physical and virtual spaces with a bi-directional connection and to use the monitoring insights and data analysis results for autonomous decision-making. Crucial to \gls{dcdt} operation are predictive capabilities and the continuous interaction with the real-world datacenter. @@ -63,7 +75,7 @@ Crucial to \gls{dcdt} operation are predictive capabilities and the continuous i There already exist \gls{dcdt} deployments. For example, ExaDigiT~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24} is a framework for digital twin development of supercomputers. It has been demonstrated at the Frontier supercomputer and it facilitates virtual prototyping and system optimization. - +% Nonetheless, existing \gls{dcdt}'s are still very limited in their capabilities as the definition and scope of a \gls{dcdt} concept is shallow and unclear. After all, only recently did the hardware capabilities needed to continuously simulate a datacenter become available~\cite{DBLP:conf/cirp/TAO2018169}. Many \gls{dcdt} frameworks still lack critical data analysis components, fault detection mechanisms, profiling techniques \etc~\cite{DBLP:conf/wosp/SumanCNTMI24}, rendering them unusable in large-scale systems. @@ -74,33 +86,47 @@ Downtime, which is the result of failures, disturbs the users and produces unful % DT's merge both simulation and telemetry to develop a holistic virtual representation of the system, bridging both the physical and virtual worlds. However, predicting datacenter behaviour quickly and reliably is a non-trivial problem that remains insufficiently unaddressed in the existing \gls{dcdt} architectures ~\cite{DBLP:conf/wosp/SumanCNTMI24, DBLP:journals/computer/AthavaleBBMMPS24} and deployments~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24}. +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/simple_dt.pdf} + \caption[Elements of the digital twin ecosystem.]{Elements of the digital twin ecosystem~\cite{DBLP:modsim24/presentation/Iosup2024} include: the insights and decisions coming from the digital twin (\myCircled{A}), the physical infrastructure (\myCircled{B}), the data coming from the physical twin telemetry (\myCircled{C}), and the digital counterpart to the physical twin (\myCircled{D}). + This thesis focuses on components (\myCircled{A}), (\myCircled{C}), and (\myCircled{D}) in this ecosystem, proposing design improvements to (\myCircled{D}, \myCircled{C}), and the feedback loop (\myCircled{A}). + } + \label{fig:simple_dt} +\end{figure} \section{Problem statement}\label{s:problem-statement} -We envision \gls{dcdt}'s as systems indispensable in future datacenters, actively interacting with the real-world facility, lowering operational costs and predicting hardware failure and software faults. In this work, we address the lack of a unified \gls{dcdt} system model and the absence of predictive capabilities in existing \gls{dcdt} system designs. We argue that the current state-of-the-art \gls{dcdt}'s lack sufficient predictive capabilities that are essential to real-time facility management of a modern datacenter. -A \gls{dt} without predictive capabilities cannot maintain the health of the datacenter effectively. +Because the main purpose of many \gls{dt}s is to forecast the short and long-term facility behaviour, \gls{dcdt} without predictive capabilities cannot maintain the health of the datacenter effectively. We posit that including holistic predictive analysis in \gls{dcdt} design can aid in efficient datacenter management and prevent missing \gls{sla}'s. -We propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. +% +For example, preventing compute failures could greatly benefit datacenter operators. + + +%To enable insights from both historical data and immediate telemetry, we propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. +We envision \gls{dcdt}'s as systems indispensable in future datacenters, actively interacting with the real-world facility, lowering operational costs and predicting hardware failure and software faults. +In particular, to enable insights from both historical data and immediate telemetry, we propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. +Our solution to this problem encompasses different levels of \gls{oda} (\eg in-band analytics, out-of-band analytics) for holistic datacenter modelling. +Together with a unified \gls{dcdt} system model and a revolutionary evaluation method of \gls{dcdt}'s, we hope to bring the modern vision of \gls{dt}'s to datacenters. \section{Research Questions}\label{s:research-questions} \emph{Main Research Question:} How to enable predictive analytics in datacenters through digital twinning?\\ -\noindent We divide the problem of designing a predictive \gls{dcdt} into three research questions: +\noindent We scope the problem of designing a predictive \gls{dcdt} by proposing three research questions: \begin{enumerate}[label=\emph{RQ\textsubscript{\arabic*}}, align=left, itemsep=0pt] % First research question stolen from Capelin by Georgios Andreadis and adapted to my work. \item \emph{How to assess the current state-of-the-art of digital twinning for datacenters?}\\ There is currently a lack of a unified system model of what constitutes a \gls{dcdt}, and the differences between existing \gls{dcdt} deployments. - Thus, it is necessary that we establish a common model of a \gls{dcdt} in the research community. - We must develop a holistic \gls{dcdt} model that factors in the necessary components of a \gls{dt}. - This is very challenging, because the \gls{dcdt} system model must address many kinds of operational and technical requirements, compatible with the existing background on \gls{dt}s. + To alleviate this problem, we aim to develop a holistic \gls{dcdt} model that factors in the necessary components of a \gls{dt}. + This is a challenging, because the \gls{dcdt} system model must address many kinds of operational and technical requirements, compatible with the existing background on \gls{dt}s. \item \emph{How to design a \gls{dcdt} reference architecture using discrete-event simulation and predictive data analysis?}\\ % You should start referring to my_system as a framework, rather than a standalone system. Existing \gls{dcdt} frameworks lack the necessary predictive capabilities to prevent unplanned behaviour in datacenters~\cite{DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/sc/BrewerMKWBHSGGW24, DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:journals/computer/AthavaleBBMMPS24}. In this work, we aim to explore the design space of a predictive \gls{dcdt} and the different design trade-offs. Through discrete-event simulation, we aim provide the foundation for the system to interact with a physical datacenter. - This is a very challenging task, because there are many functional and non-functional requirements of a \gls{dcdt} that need careful consideration. + What makes the task challenging are the many functional and non-functional requirements of a \gls{dcdt} that need careful consideration. The architecture must comply with the generic \gls{dt} model and address the non-trivial challenges in operating a modern datacenter. \item \emph{How to evaluate and validate a \gls{dcdt} reference architecture in relation to system requirements}?\\ To understand the operation of the proposed system and whether it meets its design goals we need to measure it's performance. @@ -111,9 +137,9 @@ We propose that digital twinning can be enhanced by integrating predictive analy \section{Research Methodology}\label{s:research-methodology} To answer \emph{RQ\textsubscript{1}} we conduct a literature review as proposed by \textit{Kitchenham et al.} \cite{DBLP:journals/infsof/KitchenhamPBBTNL10} along with the guidance of the supervisor. Firstly, we determine the right review method. -Secondly, we identify the various works related to \gls{dcdt}'s using different search strings +Secondly, we identify the various works related to \gls{dcdt}'s using different search strings~ (\eg ``Datacenter Digital Twinning'', ``ICT Virtual Twin'') and query combinations (``Datacenter \code{AND} Maintenance''). -To search for the results we use the digital libraries of Google Scholar, DBLP, ACM Digital Library, IEEExplore, Springer \etc +To search for the results we use the digital libraries of Google Scholar, DBLP, ACM Digital Library, IEEExplore, Springer, \etc Thirdly, we select work relevant to our research and organize the details of each article into a table. A potential outcome of this could be a system model for \gls{dcdt}'s. We envision the literature review supplying us with potential use-cases for the predictive \gls{dcdt}. @@ -132,35 +158,41 @@ We first gather a set of questions worth asking about the performance and impact We define the correct experiment setup(s) and perform the experiments on a specified hardware, considering different usage scenarios. \section{Thesis Contributions}\label{s:thesis-contributions} -\begin{enumerate} +\begin{enumerate}[align=left, labelsep=1pt] \item \textbf{Conceptual}: \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, align=left, labelsep=0pt] \item We conduct a comprehensive literature review and detailed analysis of existing works on digital twinning in the scientific research community. We collect and organize the \gls{dcdt}'s characteristics and based on our findings we propose a unified system model of the design space. - \item We propose the design of \gls{my_system}, a discrete-event \gls{dcdt} for reliable and timely failure prediction in datacenters. - \gls{my_system} includes a set of novel system components which leverage \gls{oda} and discrete-event simulation. + \item We propose the design of \mysystem, a discrete-event \gls{dcdt} for reliable and timely failure prediction in datacenters. + \mysystem includes a set of novel system components which leverage \gls{oda} and discrete-event simulation. - \item We evaluate \gls{my_system} using a novel experimentation technique and datacenter workload traces from the industry. + \item We evaluate \mysystem using a novel experimentation technique and datacenter workload traces from the industry. We design a method to evaluate \gls{dcdt}s without expensive and costly real-world experimentation. We conduct a set of exhaustive experiments and analyse the results. \end{enumerate} \item \textbf{Technical:} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, align=left, labelsep=0pt] - \item We prototype \gls{my_system} following the established \gls{dt} design principles using discrete-event simulation and \gls{oda}. + \item We prototype \mysystem following the established \gls{dt} design principles using discrete-event simulation and \gls{oda}. We include the code as an Open Science artifact and ensure the prototype remains accessible to the broader scientific community including detailed project documentation. - \item We provide the experiment setup, validation and evaluation of \gls{my_system} for detecting and predicting datacenter failures in real-time as an Open Science artifact. + \item We provide the experiment setup, validation and evaluation of \mysystem for detecting and predicting datacenter failures in real-time as an Open Science artifact. \end{enumerate} \end{enumerate} + \section{Academic Integrity Declaration}\label{s:academic_integrity_declaration} + +The statements in this section address an ongoing and seemingly growing problem in academia, that of integrity breaches primarily through the generation of material using AI techniques, such as LLM services. + \subsection{Non-Plagiarism Declaration}\label{ss:plagiarism-declaraion} I hereby declare that this thesis is my own independent work and writing. The thesis does not contain any material copied from other sources (person, Internet, or \gls{ai}), and has not been submitted for assessment elsewhere. I acknowledge that the usage of material from other works or paraphrase of such material without proper citations or credit will be treated as plagiarism. I declare that this thesis is free from \gls{ai} generated content and has been written without the help of any \gls{ai} tools. -To order to adhere to the strictest restrictions on AI-usage in higher education, this work follows the Berkley School of Law Artificial Intelligence Policy, as stated in \url{https://www.law.berkeley.edu/wp-content/uploads/2026/05/AI-Final-Policy-26.pdf}. + +To adhere to the current (strictest) restrictions on AI-usage in higher education, this work follows the Berkley School of Law Artificial Intelligence Policy, as stated in \url{https://www.law.berkeley.edu/wp-content/uploads/2026/05/AI-Final-Policy-26.pdf}. \subsection{Preventing Reference Fraud}\label{ss:plagiarism_references} -I hereby declare that all the references in this thesis refer to genuine scientific work published in peer-reviewed journals or other sources of reliable and safe online information (\eg Wikipedia articles) and have been used in accordance to the article authors' wishes. +I hereby declare that all the references in this thesis refer to genuine scientific work published in peer-reviewed journals or other sources of reliable and safe online information~ (\eg Wikipedia articles) and have been used in accordance to the article authors' wishes. + Additionally, under the guidance of the supervisor this work adheres to the strictest rules for referencing and to prove the originality of all references, each \BibTeX citation contains a \texttt{note} field with the following comment: \emph{This BibTeX citation comes from:} followed by the URL leading directly to the citation source. In case of citations not formatted in \BibTeX, the same format follows but with adequate reference-style name (\eg APA, Chicago, MLA). @@ -177,14 +209,18 @@ This work addresses the four grand societal challenges related to this goal: \be \item sustainability \item usability \end{enumerate*}~\cite{DBLP:journals/corr/IosupKLVG22}. -\gls{my_system} addresses (1) directly by making large-scale datacenter management easier. + +\mysystem addresses goal (1) directly by making large-scale datacenter management easier. + We address (2) by ensuring our work adheres to the \gls{fair} principles of Open Science. Moreover, in this thesis we try to make \gls{dcdt} systems more understandable to the broader scientific community by providing a unified system model. Additionally, we contribute to responsible software design by adhering to best software engineering practices in the design of the prototype. -(3) is addressed indirectly, as the consequences of the insights provided by a holistic, \gls{oda} powered \gls{dcdt} can help datacenter managers make decisions that are more sustainable in the future. + +Goal (3) is addressed indirectly, as the consequences of the insights provided by a holistic, \gls{oda} powered \gls{dcdt} can help datacenter managers make decisions that are more sustainable in the future. + We contribute to (4) by helping predict unexpected failures and lowering operational costs, ensuring datacenters can continue to be usable in the future. We believe this work has a strong societal impact due to addressing the four grand societal challenges described by Iosup \etal and we hope through this work we can advance the scientific research community towards a more sustainable future. -Abiding the FAIR data principles, the entire source code of the prototype and related work has been made available at the \url{https://git.denounce.ai/opendc.git} repository. +Abiding the FAIR data principles, the entire source code of the prototype and related work has been made available at the \url{https://github.com/M-J-Kwiatkowski/opendc} and the \url{https://github.com/M-J-Kwiatkowski/sunfish} repositories. The reuse and reproduction of experiments is explained in a detailed guide at the root of the repository, along with the necessary dependencies and experimental setup. % Comment from Alexandru is to merge the above repository into the AtLarge repository, with the proper attribution to you. % I will do this once the project is over. @@ -196,7 +232,6 @@ In Chapter \ref{s:design}, we present the design of \mysystem. In \Cref{s:implementation} we present the technical details of \mysystem prototype. In Chapter \ref{s:evaluation} we evaluate the prototype of the system and validate it against the set of functional and non-functional requirements. In Chapter \ref{s:conclusion} we conclude the thesis with a summary of contributions and potential future work. -\newpage \begin{figure}[t!] \centering \includegraphics[width=\linewidth]{images/thesis_structure.png} diff --git a/content/preamble/abstract.tex b/content/preamble/abstract.tex index c8078b4..9c2247a 100644 --- a/content/preamble/abstract.tex +++ b/content/preamble/abstract.tex @@ -12,6 +12,8 @@ Without predictive maintenance and forecasts, system administrators cannot make In this work, we propose to enable predictive analytics for datacenters using digital twinning. We survey the datacenter digital twinning field, and organize our findings into a system model. -Additionally, we design \mysystem~-- a novel reference architecture for predictive datacenter digital twins, and evaluate it through prototype-based experiments. +Based on real-world use-cases we design \mysystem~-- a novel reference architecture for predictive datacenter digital twins, and evaluate it through prototype-based experiments. Our results indicate \mysystem is capable of reliably differentiating between mild and severe compute failures, and can successfully incorporate a predictive analytics engine to the benefit of datacenter managers. +The main findings are backed-up by experiments which replicate a peer-reviewed publication evaluation approach to show that \mysystem can notify datacenter operators of compute failures. +Moreover, to show that the system can incorporate a predictive analytics-engine, we have conducted a novel, conceptual experiment to exhibit the potential future gains of a holistic Datacenter Digital Twin. diff --git a/content/preamble/acknowledgement.tex b/content/preamble/acknowledgement.tex index df5ea7c..81660b5 100644 --- a/content/preamble/acknowledgement.tex +++ b/content/preamble/acknowledgement.tex @@ -4,9 +4,9 @@ \end{center} I wish to express my thanks to the many people who have helped me in preparation of this thesis. -Dante Niewehuis, my daily supervisor, with patience and understanding mentored me during our collaboration, which led to the timely and successful completion of this thesis. +Dante Niewenhuis, my daily supervisor, with patience and understanding mentored me during our collaboration, which led to the timely and successful completion of this thesis. -Alexandru Iosup, my 1\textsuperscript{st} supervisor, provided invaluable feedback and suggestions that helped immensely during writing. +Alexandru Iosup, my 1\textsuperscript{st} supervisor, provided invaluable feedback and suggestions that helped immensely during the design and writing. Jesse Donkervliet guided the thesis timeline and greatly assisted in preparation of the final defence during the weekly thesis meetings. |