diff options
Diffstat (limited to 'content')
| -rw-r--r-- | content/background.tex | 73 | ||||
| -rw-r--r-- | content/conclusion.tex | 4 | ||||
| -rw-r--r-- | content/design.tex | 19 | ||||
| -rw-r--r-- | content/evaluation.tex | 301 | ||||
| -rw-r--r-- | content/implementation.tex | 37 | ||||
| -rw-r--r-- | content/intro.tex | 66 | ||||
| -rw-r--r-- | content/preamble/abstract.tex | 4 | ||||
| -rw-r--r-- | content/preamble/acknowledgement.tex | 2 |
8 files changed, 332 insertions, 174 deletions
diff --git a/content/background.tex b/content/background.tex index 97dc6aa..74083e5 100644 --- a/content/background.tex +++ b/content/background.tex @@ -1,12 +1,12 @@ \chapter{Background}\label{s:background} \section{Overview}\label{ss:background_overview} \begin{mynote} - The contribution in this chapter is three-fold: + Addressing RQ$_1$, the contribution in this chapter is three-fold: \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt] - \item We provide a brief overview on datacenters (\Cref{ss:datacenters}) datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning (\Cref{ss:digital-twinning}). - \item We survey the state-of-the-art concerning datacenter digital twinning (\Cref{sss:advanced_dts}). - \item We construct a system model for existing datacenter digital twins (\Cref{ss:system_model_for_dcdts}) + \item We provide a brief overview on datacenters (\Cref{ss:datacenters}) datacenter simulation (\Cref{sss:simulation}), compute failures (\Cref{sss:failures}), and digital twinning~ (\Cref{ss:digital-twinning}). + \item We survey the state-of-the-art concerning datacenter digital twinning~ (\Cref{sss:advanced_dts}). + \item We propose a system model (architecture) for datacenter digital twins, including a novel Digital Thread component for the operational needs linking the digital with the physical worlds~(\Cref{ss:system_model_for_dcdts}). \end{enumerate} \end{mynote} @@ -28,7 +28,7 @@ An example datacenter in \gls{cern}, is depicted on \Cref{fig:datacenter}. \begin{figure}[t] \centering \includegraphics[width=0.9\linewidth]{images/datacenter.jpg} - \caption[Datacenter in CERN.]{Example of a datacenter in \gls{cern}, Switzerland (2010)~\cite{Wikipedia:article/Datacenter}. In the figure we can see servers within servers racks, and the network cables.} + \caption[Datacenter in CERN.]{Example of a datacenter in \gls{cern}, Switzerland (2010)~\cite{Wikipedia:article/Datacenter}. The figure depicts servers within servers racks, and network cables interconnecting them.} \label{fig:datacenter} \end{figure} @@ -47,31 +47,7 @@ Making sure that all the parts of the datacenter work together is a tough task. What drives datacenter complexity even further is that sophisticated systems are not merely a sum of their parts~\cite{Wikipedia:article/Systems_Thinking}. The combination of the above factors makes datacenter management a difficult, non-trivial challenge. -\subsection{Datacenter Simulation}\label{sss:simulation} -\input{sources/simulator_comparison.tex} - -Efficient and timely datacenter management is a difficult challenge, because datacenters are extremely complex facilities. -They require deep understanding to operate properly. -However, running real-world experiments is costly in both time and resources. -Additionally, experimentation \emph{in situ} is unsustainable and difficult to reproduce. -Alternatives to real-world experiments include simulation and mathematical analysis. -Because mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}, in this project we only consider simulation as a foundation for the \gls{dcdt}. -%To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. - -Simulation empowers better design, testing and management of datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -A well-designed datacenter simulator can estimate a months-long workload in a few minutes or hours. -To simulate is to ``imitate of real-world process or system over time, enabling the study of, and experimentation with the internal interactions of complex systems''~\cite{DBLP:books/daglib/0034857} -In this project we only consider \emph{discrete-event simulation}. -Discrete-event simulation represents system operations as a sequence of events over time, with an assumption that no changes occur between the events. -Due to the scale and complexity of datacenters, most simulators use discrete-event simulation~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -There exist many datacenter simulation tools, for example DGSim~\cite{DBLP:conf/europar/IosupSE08}, CloudSim~\cite{DBLP:journals/spe/CalheirosRBRB11}, SimGrid~\cite{DBLP:journals/corr/CasanovaGLQS13}, iCanCloud~\cite{DBLP:journals/grid/NunezVCCCL12}, GroudSim~\cite{DBLP:conf/europar/OstermannPPF10} and OpenDC~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -See \Cref{tab:datacenter_simulator_comparison} for a comparison of selected datacenter simulators, combined by Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -In order to narrow the scope of the project, we only consider {OpenDC} as a simulator for the digital twin design. -We decided to use {OpenDC}, because we find it important for a simulator to model hardware failures well. -\emph{Failure models} are a carefully calibrated, advanced feature of {OpenDC}. -Further details about {OpenDC} can be referred to in the linked literature \cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. - -\subsection{Compute Failures}\label{sss:failures} +\section{Compute Failures}\label{sss:failures} A failure is defined as ``an event that makes a system fail to operate according to its specifications``~\cite{DBLP:journals/jpdc/JavadiKIE13}. A simple example of a failure is when an old hard drive stops working. Data on the disk is lost, and services running on the respective server are disrupted. @@ -99,10 +75,35 @@ A failure trace is defined by an interval, duration, and intensity of several fa In summary OpenDC enables experimentation with failures that enables insights that are not provided by other state-of-the-art software. However, the fidelity of failure modeling inside a datacenter simulation is still insufficient to predict in failures in real-time, as they happen in a physical datacenter. Since a datacenter simulator is quite different from a digital twin, we cannot use the same computation methods from simulation to predict real-time failures. +\section{Datacenter Simulation}\label{sss:simulation} +\input{sources/simulator_comparison.tex} + +Efficient and timely datacenter management is a difficult challenge, because datacenters are extremely complex facilities. +They require deep understanding to operate properly. +However, running real-world experiments is costly in both time and resources. +Additionally, experimentation \emph{in situ} is unsustainable and difficult to reproduce. +Alternatives to real-world experiments include simulation and mathematical analysis. +Because mathematical analysis is not scalable to modern datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}, in this project we only consider simulation as a foundation for the \gls{dcdt}. +%To help datacenter operators, the scientific community proposes to simulate datacenters to make more informed decisions. + +Simulation empowers better design, testing and management of datacenters~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +A well-designed datacenter simulator can estimate a months-long workload in a few minutes or hours. +To simulate is to ``imitate of real-world process or system over time, enabling the study of, and experimentation with the internal interactions of complex systems''~\cite{DBLP:books/daglib/0034857} +In this project we only consider \emph{discrete-event simulation}. +Discrete-event simulation represents system operations as a sequence of events over time, with an assumption that no changes occur between the events. +Due to the scale and complexity of datacenters, most simulators use discrete-event simulation~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +There exist many datacenter simulation tools, for example DGSim~\cite{DBLP:conf/europar/IosupSE08}, CloudSim~\cite{DBLP:journals/spe/CalheirosRBRB11}, SimGrid~\cite{DBLP:journals/corr/CasanovaGLQS13}, iCanCloud~\cite{DBLP:journals/grid/NunezVCCCL12}, GroudSim~\cite{DBLP:conf/europar/OstermannPPF10} and OpenDC~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +See \Cref{tab:datacenter_simulator_comparison} for a comparison of selected datacenter simulators, combined by Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +In order to narrow the scope of the project, we only consider {OpenDC} as a simulator for the digital twin design. +We decided to use {OpenDC}, because we find it important for a simulator to model hardware failures well. +\emph{Failure models} are a carefully calibrated, advanced feature of {OpenDC}. +Further details about {OpenDC} can be referred to in the linked literature \cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. + \begin{figure}[t] \centering \includegraphics[width=0.95\linewidth]{images/five_dimensional_dt.png} - \caption[A basic framework for the \gls{dt}.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} + \caption[A basic framework for the Digital Twin.]{A basic framework for the \gls{dt}. Four core elements of a \gls{dt} are defined: The physical entity (\myCircled{1}) and the simulated virtual twin (\myCircled{2}). A service for out-of-band data analytics (\myCircled{3}) and a persistent storage of historical data (\myCircled{4}) are crucial to the \gls{dt} because they are necessary to gain meaningful monitoring insights. Adapted from Tao \etal ~\cite{DBLP:conf/cirp/TAO2018169}.} + %Fei Tao is a renowned figure with over 62k citations. He is a figure of authority on digital twins.% \label{fig:five_dimensional_dt} \end{figure} @@ -119,11 +120,11 @@ The longer the \gls{dt} is working, the more accurate its predictions, because a A generic \gls{dt} architecture is depicted in Figure \ref{fig:five_dimensional_dt} from Tao \etal~\cite{DBLP:conf/cirp/TAO2018169}. % Why has not anyone done this before? -Digital twinning has only recently become feasible because of the developments in \gls{hpc}. -Between 2003 and 2011 the compute needed to run a \gls{dt} was simply not present. +Although digital twinning through simplified models exists since the 1960s, detailed and accurate digital twinning of complex and realistic scenarios has only recently become feasible because of developments in \gls{hpc}. +Between 2003 and 2011, the compute needed to run a \gls{dt} was simply not present or was (largely) unaffordable. As such, while the concept existed, the hardware did not catch up yet. However, in the last decade, multicore computing paradigms and the advent of GPU computing has finally enabled computation needed to run digital twins. -As a result, digital twins have become more relevant today than 10 years ago~\cite{DBLP:conf/cirp/TAO2018169}. +As a result, digital twins have become more relevant today than a decade ago~\cite{DBLP:conf/cirp/TAO2018169}. A crucial part any of any \gls{dt} is \emph{predictive modelling}, which drives actionable insights~\cite{DBLP:usdoe/report/AP26894} (see \Cref{fig:predictive_analytics}). \begin{figure}[t] @@ -269,7 +270,7 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su \end{figure} \subsection{System Model for Datacenter Digital Twinning} \label{ss:system_model_for_dcdts} -In \Cref{fig:system_model} we present a holistic model of \gls{dcdt}s from \Cref{sss:advanced_dts}. +In \Cref{fig:system_model} we propose a new, holistic model of \gls{dcdt}s from \Cref{sss:advanced_dts}. The figure includes the functionality present in the majority of \gls{dcdt}s, combined together into a unified model. We distinguish 3 core elements of every \gls{dcdt}: \begin{enumerate*}[label=(\arabic*)] @@ -298,7 +299,7 @@ We distinguish 3 core elements of every \gls{dcdt}: A digital twin that contains only the infrastructure model, cannot enable insights into the real-time operation of the datacenter. Likewise, a \gls{dcdt} containing just the operations model does not possess a capability to \eg simulate the datacenter. Only both, combined together enable the insights envisioned by the \gls{nasem} \gls{dt} definition~\cite{DBLP:usdoe/report/AP26894}. - \item \textbf{Digital Thread} connects the virtual world to the physical world. + \item \textbf{(novel) Digital Thread} connects the virtual world to the physical world. This is a novel contribution of our thesis. The digital thread is a \emph{conceptual} element that unites the components which do not belong in either of the worlds. All \gls{dcdt} programs from \Cref{sss:advanced_dts} contain elements that are ``in-between'' the physical and virtual twin. diff --git a/content/conclusion.tex b/content/conclusion.tex index 86d9f68..fa90b3e 100644 --- a/content/conclusion.tex +++ b/content/conclusion.tex @@ -48,7 +48,7 @@ We answer the main research question by addressing each sub-research question. \begin{figure}[ht] \centering \includegraphics[width=0.8\textwidth]{images/48_years.pdf} - \caption[48 years of microprocessor trend data.]{48 years of microprocessor trend data. Legend: \textcolor{Orange}{$\blacktriangle$ Transistors (thousands)}, \textcolor{Blue}{$\lgblkcircle$ Single Thread Performance (SpecINT $\times 10^3$)}, \textcolor{Green}{$\lgblksquare$ Frequency (MHz)}, \textcolor{Maroon}{$\blacktriangledown$ Typical Power (Watts)}, $\mdlgblkdiamond$ Number of Logical Cores~\cite{DBLP:image/48Microprocessor/Rupp}.} + \caption[48 years of microprocessor trend data.]{48 years of microprocessor trend data. Legend: \textcolor{Orange}{$\blacktriangle$ Transistors (thousands)}, \textcolor{Blue}{$\lgblkcircle$ Single Thread Performance (SpecINT $\times 10^3$)}, \textcolor{Green}{$\lgblksquare$ Frequency (MHz)}, \textcolor{Maroon}{$\blacktriangledown$ Typical Power (Watts)}, $\mdlgblkdiamond$ Number of Logical Cores. (Source:~\cite{DBLP:image/48Microprocessor/Rupp}.)} \label{fig:rupp_48_years_microprocessor_data} \end{figure} @@ -70,7 +70,7 @@ To achieve the \gls{nasem} goals of digital twinning~\cite{DBLP:usdoe/report/AP2 A \gls{dcdt} must posses predictive capabilities, by definition~\cite{DBLP:usdoe/report/AP26894}. \item \textbf{What Is Missing?}\\ To power the predictions, we envision an \gls{ml}-based inference engine as a necessary component of digital twinning. - The need for \gls{ml} arises naturally in scenarios where large volumes of data, requiring little to no preprocessing meet the demand for estimating future facility behaviour~\cite{Wikipedia:PredictiveModelling,CambridgeUniversityPress:book/Deisenroth}. + The need for \gls{ml} arises naturally in scenarios where large volumes of data, requiring little to no preprocessing meet the demand for estimating future facility behaviour~\cite{Wikipedia:article/PredictiveModelling,CambridgeUniversityPress:book/Deisenroth}. However, currently there are no \gls{dcdt} deployments that model the warehouse using an \gls{ml} approach to predict events (see \Cref{tab:dt_features_comparison}). \item \textbf{The Next Steps}\\ In short, we stipulate \gls{dcdt}s should include \gls{ml} in their \gls{oda} analysis. diff --git a/content/design.tex b/content/design.tex index 7d4a73a..72414a3 100644 --- a/content/design.tex +++ b/content/design.tex @@ -2,12 +2,14 @@ \chapter{Design of \mysystem, a Digital Twin For Predictive Analysis of Datacenters}\label{s:design} \section{Overview}\label{ss:design_overview} \begin{mynote} - Our contribution in this chapter is three-fold: + Addressing RQ$_2$, our contribution in this chapter is three-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}, itemsep=0.2pt] - \item We analyze the requirements for \mysystem (\Cref{ss:requirements_analysis}). - \item We propose a conceptual design for \mysystem's architecture (\Cref{ss:design_of_mysystem}) - \item We describe how \mysystem fulfills the functional and non-functional requirements in \Cref{ss:requirement_validation}. + \item We analyze the requirements for digital twins enabling predictive analysis of datacenters~(\Cref{ss:requirements_analysis}). + + \item We design the \mysystem architecture as a digital twin enabling predictive analysis of datacenters through discrete-event simulation and ODA~(\Cref{ss:design_of_mysystem}). + + \item We analyze how \mysystem fulfills the functional and non-functional requirements~(\Cref{ss:requirement_validation}). \end{enumerate} \end{mynote} \section{Requirements Analysis}\label{ss:requirements_analysis} @@ -107,6 +109,9 @@ In addition to the functional requirements, we also present non-functional requi \item \textbf{Using \mysystem should not introduce any delays in visualizations longer than 1 second.}\\ The system must work in real-time, without significant delay. We impose 1 second delay as acceptable to datacenter engineers. + We have arrived at the 1 second threshold due to the fact that datacenter operators need time to react to sudden changes in the facility behaviour. + To expect a sub-1-second performance from human employees is unjustified. + At the same time, any delay $>1$ second can already negatively impact engineer's ability to react to different events at the same time in the correct order (\ie $>1$ delay hinders the ability of technicians to follow the specific datacenter protocols and policies during an event). The system must support datacenter operators with insights at fine-grained granularity, so that insights derived from data analysis remain accurate upon reception by datacenter operators. Without \textbf{NFR1}, \mysystem's insights will not be timely, and will be useless to datacenter operators. \item \textbf{The system should log the ingestion and processing of metrics.} \\ @@ -227,7 +232,11 @@ The Interactive Dashboard (\myCircled{2a}) supports (\textbf{FR2}), as it allows It enables (\textbf{FR5}) because the predictive insights can be displayed directly through the dashboard to the datacenter operators. Operators can then overwrite the autonomous decisions performed by the \gls{dt}. Dashboards are the current industry practice for clear data visualization (\textbf{FR7}). -For our implementation, we ensure (\textbf{NFR1}) is met by including a state-of-the-art dashboard. +In our design, we ensure (\textbf{NFR1}) is in a two-fold way. +Firstly, we ensure a direct connection between the Interactive Dashboard (\myCircled{2a}) and the Message Broker (\myCircled{2b}). +As the Message Broker (\myCircled{2b}) is capable of accommodating multiple consumers, the Interactive Dashboard (\myCircled{2b}) receives the same telemetry at the same rate as the live prediction engine. +Secondly, through a direct connection to the Database (\myCircled{3a}), the Interactive Dashboard (\myCircled{2a}) can instantly retrieve all historical patterns. +Thus, we meet (\textbf{NFR1}). (\textbf{NFR2}) and (\textbf{NFR3}) are met by including in the system implementation a dashboard that enables logging of metrics. The Message Broker (\myCircled{2b}) indirectly fulfills (\textbf{FR1}). diff --git a/content/evaluation.tex b/content/evaluation.tex index b17fd9d..4978206 100644 --- a/content/evaluation.tex +++ b/content/evaluation.tex @@ -2,11 +2,13 @@ \section{Overview}\label{ss:evaluation_overview} \begin{mynote} - The contribution of this chapter is two-fold: + Addressing RQ$_3$ together with the previous chapter, the contribution of this chapter is two-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] \item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}. - \item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}. + + \item We provide a comprehensive set of experiments to evaluate \mysystem~(\Cref{ss:experiment1,ss:experiment2}). + \end{enumerate} Our findings indicate: \vspace{-0.2cm} @@ -14,59 +16,69 @@ \item Digital twinning can be used for failure detection to the benefit of datacenter operators. \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime. \item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime. - \item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated tasks. + \item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated during a workload. + \item \mysystem can estimate both the best-case and worst-case number of total tasks terminated due to failures during a datacenter workload. \end{enumerate} \end{mynote} -\section{Experimental Setup}\label{ss:experimental_setup} - +\section{Novel Evaluation Technique}\label{ss:experimental_setup} \begin{figure}[t] \centering - \includegraphics[width=0.8\linewidth]{images/novel_eval_method.png} - \caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.} + \includegraphics[width=0.74\linewidth]{images/novel_eval_method_improved.png} + \caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. In this approach, both the physical and virtual datacenter are emulated using discrete-event simulation.} \label{fig:novel_eval_method} \end{figure} +In the this section, we evaluate \mysystem using \code{OpenDC} and all the necessary software (\ie \code{Grafana}, \code{PostgreSQL}, \code{Confluent Kafka}. +A na\"ive experimental approach would consist of connecting \mysystem to a physical datacenter. +However, as explained earlier in this work, running large experiments on real-world data centers is expensive and time-consuming~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +Additionally, we do not have access to a datacenter. +To alleviate this problem, we again utilize simulation for the benefit of evaluating \mysystem, and \gls{dcdt}s in general. -In this section we describe the technical setup used to evaluate \mysystem. -However, \Cref{fig:reference_architecture} assumes the system designer is capable of connecting the digital twin directly to the datacenter. -This raises a problem, we cannot just go and test digital twins on large systems, because we do not have large systems at hand. -Moreover, real-world experimentation is costly and unsustainable in the long run~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -To overcome this problem, we present a novel datacenter digital twin method capable of evaluating a \gls{dcdt} without the physical datacenter. -\Cref{fig:novel_eval_method} details our approach. - -In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. -In our implementation, this is a second \code{OpenDC} -process (see \Cref{fig:implementation}). -The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. -For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}. +In this non-trivial approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. +For our implementation, we decided to use a second \code{OpenDC} +process. +A schematic overview of this experimental technique can be seen in \Cref{fig:novel_eval_method}, where the \textcolor{Red}{\ding{110} \textbf{red}} box highlights our initial approach, and the \textcolor{Green}{\ding{110} \textbf{green}} box shows the novel evaluation method. +The advantages of this approach are as follows: +\begin{enumerate*}[label=(\arabic*)] + \item The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. + \item The environmental footprint is smaller. As a result, \mysystem is more sustainable. + \item Datacenter engineers can fine-tune the digital twin using this setup before pairing it with the production environment. + In result, engineers can effectively use our setup to test and debug the digital twin before installation. + \item With this method, \gls{dcdt} research becomes more accessible both to the scientific community and higher education. +\end{enumerate*} -The technical setup used for all experiment adheres to the \code{OpenDC} -documentation (see Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}). -The workload trace used for all experiments comes from BitBrains~\cite{DBLP:conf/ccgrid/ShenBI15}. +\section{Experiment Parameters}\label{ss:experiment_parameters} +In this section we detail the experimental setup that follows most of the experiments. +The workload trace used for all experiments is based on a one-month SURF workload trace~\cite{DBLP:conf/ccgrid/ShenBI15}. In the experiments we model a Dutch SURF datacenter for scientific computing. The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}. -The scheduling policy for all experiments is the \code{FilterScheduler} which considers the RAM and CPU capacity for choosing hosts to run tasks on. -This scheduling policy is also used by \code{SmartScheduler}, as outlined in \Cref{ss:programming}, albeit with modifications to enable the system knobs to take autonomous action. +This setup ensures a realistic experiment scenario that is representative of a real-world setting. + +The \code{OpenDC} that models the physical datacenter uses the \code{SmartScheduler}. +The event driven simulator within the predictive analytics engine of \mysystem uses the \code{FilterScheduler} by default, unless otherwise specified. -In all experiments we use either \emph{failure traces} or \emph{failure models}. -For a brief explanation on the differences between the two, consult \Cref{sss:failures}. +For the experiments that model datacenter compute failures, we use \emph{failure traces} or \emph{failure models}. +For a detailed explanation on the differences between the two, consult \Cref{sss:failures}. +In short, failure traces contain the data necessary to reproduce the outages experienced by the real datacenter during the simulation. +A trace contains 3 elements, failure interval, failure duration and failure intensity (see \Cref{tab:failure_traces_overview}). +A failure model is characterized as as a statistical distribution of failures based on scientific research. +It is akin to the output of an algorithm that determines when failures should happen, rather than a real-world trace~\cite{DBLP:journals/jpdc/JavadiKIE13}. In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -We chose a diverse range of failure models, based on the mean failure intensity in each trace (indicated in parentheses). -As a result, we chose the traces from: \begin{enumerate*}[label=(\Roman*)] - \item Gmail (53.26\%), - \item WhatsApp (57.97\%), - \item YouTube (62.1\%), - \item Twitter (65\%), - \item Facebook (64\%). -\end{enumerate*} +We chose a diverse range of failure traces, based on the mean failure intensity in each trace (see \Cref{tab:failure_traces_overview}). +Mean failure intensity is a metric that shows best how severe the failures are. +The higher the failure intensity, the more hosts go down at once, resulting in more serious downtime and missed \gls{sla}s. +According to the mean failure intensity, we chose the 5 traces in \Cref{tab:failure_traces_overview} for our experiments. +\input{sources/failure_model_statistics.tex} +\input{sources/failure_models.tex} +\input{sources/failure_characteristics.tex} In \Cref{ss:experiment2} we used a failure trace from Skype. -This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}). +This is the only trace that can be paired with a corresponding failure model (see \Cref{tab:failure_models_table}). Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures. For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5. -Importantly, in our figures we do not report the standard deviation of our experiments. -This is due to the fact that \code{OpenDC} is a fully deterministic simulator, and on each simulation run, given the same random seed will produce exactly the same results. -We believe the deviation in the results of the experiments stemming only from the random number generator is not meaningful, therefore none of the figures contain the standard deviation bars. + +All the experiments were run on a commodity laptop, with an AMD Ryzen 7840U CPU containing 16, double-threaded cores and maximum frequency of 5.13 GHz, 32GB of DDR5 RAM. +By showing that \mysystem works using this setup, we encourage scientists and academia (stakeholder \textbf{S3}) to freely experiment with \mysystem. \section{Experiment 1: Failure Detection}\label{ss:experiment1} \begin{figure}[t] @@ -75,107 +87,198 @@ We believe the deviation in the results of the experiments stemming only from th \caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.} \label{fig:red_yellow_alarms} \end{figure} +For the first experiment we adopt the idea introduced by Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +We show 2 different ways a \gls{dcdt} can notify datacenter engineers of detected failures. + +\subsection{Context}\label{sss:failure_detection_context} +In this section we try to provide the rationale behind this experiment. +Failure detection is a primary use-case for \gls{dcdt}s~\cite{DBLP:journals/computer/AthavaleBBMMPS24,DBLP:usdoe/report/AP26894,DBLP:conf/sc/TaheriBPRHDEWPM24,DBLP:journal/IJAE/Teugel2011}. +Failures in a datacenter can arise from a number of problems, \eg software configuration issues, power outage, network congestion~\cite{DBLP:report/AnnualOutageAnalysis2025}, and must be immediately detected to ensure their negative impact is minimized. +Some failures can have negligible consequences (around 38\% of all failures are negligible~\cite{DBLP:report/AnnualOutageAnalysis2025}), whilst other can cause severe downtime and loss of millions of \$USD~\cite{DBLP:report/AnnualOutageAnalysis2025}. +By differentiating between insignificant and severe failures, datacenter operators can focus their efforts on addressing the most impactful problems, minimizing missed \gls{sla} and ensuring swift datacenter operation. -The purpose of this experiment is two fold: \begin{enumerate*} - \item to show our system works correctly +In \Cref{ss:experiment1} we model such a scenario. +For example, imagine the following sequence of events: +\begin{enumerate}[label=\textbf{\arabic*.},align=left] + \item The datacenter receives a scheduled workload. + \item To predict what kind of failures might occur during the workload, we ask the digital twin to run the workload first. + \item We have no \emph{a priori} knowledge of the workload type, so we can only estimate the distribution of failures. To do this, the \gls{dcdt} will run the workload, assuming failures follow \eg a normal distribution. (Unlike traditional analytical tools, the distribution can be selected arbitrarily, even changed during the run, and the predictive results do not rely on underlying statistical assumptions.) + \item After running the simulation, the \gls{dcdt} can analyze the results and compare them with the running workload. + \item If the running workload has more failures than what the \gls{dcdt} predicted, with the latter representing the known errors inherent in typical datacenter operations, the discrepancy is used to notify the datacenter operators -- + something is wrong. + Through the real-time feedback loop, the \gls{dcdt} not only notifies the datacenter operators, but can also help understand where and how what is happening in reality is different from the simulation. +\end{enumerate} + +In our experiment, we notify the datacenter operators in two cases: +(1) we send a yellow alarm if the 80\% threshold is reached, that is, the simulated results get within 80\% of the predicted threshold for the number of failures; +(2) we send a red alarm if the 90\% threshold is reached. + +The purpose of this experiment is two-fold: \begin{enumerate*}[label=(\arabic*)] + \item to show our system works correctly, and \item to show our system fulfills the functional and non-functional requirements. \end{enumerate*} -To this end, we replicate an experiment from Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system. -The experimental setup is as defined in \Cref{ss:experimental_setup}. -The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)] +The technical experimental setup can be described as follows: \begin{enumerate*}[label=(\arabic*)] \item firstly, we use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload. \item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm) \item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached. \item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold. \end{enumerate*} -The results are in \Cref{fig:red_yellow_alarms}. -\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. -What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems. -Importantly, the more failure-intense the trace, the more alarms are raised on behalf of the digital twin. +The results are in \Cref{fig:red_yellow_alarms,fig:alarms_vs_failures,fig:failure_detecton_rate}. \begin{figure}[t] \centering \includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf} - \caption[Total number of failures versus numbe rof alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a BitBrains workload in the SURF-SARA cluster. The failure traced used in this experiment models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.} + \caption[Total number of failures versus number of alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a SURF month workload in the SURF-SARA cluster. The failure trace used in this plot models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.} \label{fig:alarms_vs_failures} \end{figure} -Additionally \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. -In the figure we can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. + +\subsection{Discussion}\label{sss:experiment1_discussion} + +\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. +In the figure, we can see that as the mean failure intensity rises, so does the total number of failures detected. +This behaviour is expected -- the higher the mean failure intensity, the more severe the failures, the more notifications to datacenter operators are sent. +The more failure-intense the trace, the more alarms are raised on behalf of the digital twin. +What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems (\ie the number of yellow alarms is smaller than the number of red alarms). + + +To offer a second perspective, we also provide the results depicted in \Cref{fig:alarms_vs_failures}. +In this figure, we show the total number of alarms raised and the ground truth (how many failures occurred in reality.) +We can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. For this visualization, we combined both the red and yellow alarms into a single metric. +In short, \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. -However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. -The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate. -What is surprising is Taheri \etal report almost negligible false positive rate and of their system. -Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate. -In our experiment, the numbers differ significantly. -\begin{figure}[t] +\begin{figure}[!t] \centering - \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf} + \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate2.pdf} \caption[Failure detection rate overview.]{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}). Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. We believe this is due to the irreconcilable differences between our experimental setups.} \label{fig:failure_detecton_rate} \end{figure} -\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%. -Compared to the DyTwin deployment, the difference is staggering. -However, the discrepancy stems from the fact in our setup we differentiate between different types of failures. -This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe. -Unusually, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace. + +However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. +The rate is simply calculated as the anomalies detected correctly over the true anomaly count~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. (Note: The correct term for this should be the \emph{anomaly detection ratio}.) +Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection ratio, defined as the sum of Red and Yellow alarm-counts, over the Total Failure count. +What is surprising is Taheri \etal report almost negligible false positive rate and of their system. +Moreover, after testing with DyTwin in a specific experimental setup, Taheri \etal conclude their proposed method achieves 100\% anomaly detection rate. +In our experiment, the numbers differ significantly; we attribute this to irreconcilable differences between our experimental setups. + + +% Our results are also different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +% We believe this is due to the irreconcilable differences between our experimental setups. + + + +\Cref{fig:failure_detecton_rate} shows the mean failure detection ratio to be around 12\%. +Compared to the DyTwin results reported by Taheri~\etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}, the difference is significant. +We stipulate that the root of the discrepancy is that our setup differentiates between different types of failures, a capability that is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +As a new result, we interpret \Cref{fig:failure_detecton_rate} as showing that on average 12\% of failures in the workload are severe. +Surprisingly, the WhatsApp failure detection ratio is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2\textsuperscript{nd} least failure-intensive trace. \section{Experiment 2: Failure Prediction}\label{ss:experiment2} -In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics. +In \Cref{ss:experiment1} we showed \mysystem is capable of incorporating descriptive analytics. Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures. -In this section we try to show \mysystem can additionally work well together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. - -\begin{figure}[ht] - \hspace{-0.8cm} - \begin{minipage}[b]{0.45\textwidth} - \centering - \includegraphics[width=1.2\linewidth]{images/failure_likelihood.pdf} - \end{minipage} - \hspace{1.2cm} - \begin{minipage}[b]{0.45\textwidth} - \centering - \includegraphics[width=1.2\linewidth]{images/conceptual_experiment.pdf} - \end{minipage} - \caption[The results of Experiment 2.]{Left figure shows the potential failure distribution likelihood to approximate the true failure distribution. - Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.} +In this section, we aim to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. + +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/failure_likelihood.png} + \caption[The failure model likelihood over time.]{How closely a \emph{failure model} approximates the real failure distribution (the Skype failure trace) during runtime. The vertical axis is linear but does \textit{not} start at 0.} \label{fig:failure_likelihood} \end{figure} -\input{sources/failure_models.tex} \subsection{Context}\label{sss:context_experiment2} -In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods. -Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions. -For the task at hand, we chose the Skype trace, because it is supported by 4 different failure models, based on past Skype workload data. -These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}. -The Skype trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -The goal was to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes. - +To predict when a host failure might occur, a straightforward approach is to use long-established statistical methods. +Our goal is to approximate the real failure distribution of a workload, using past data and relevant statistical distributions. +For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models~(see \Cref{tab:failure_models_table}, sourced from~\cite{DBLP:journals/jpdc/JavadiKIE13}), based on past Skype workload data. +%These 4 statistical distributions, published in a peer-reviewed journal are in . +The Skype Failure Trace model was sourced from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines \emph{before} it crashes. Initial experiment results were unpromising. Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly). To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution. Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time. -The results are in \Cref{fig:failure_likelihood}. +The results are shown in \Cref{fig:failure_likelihood}. -In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$. -Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. -Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. -This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis. +% In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +% Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. +% Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +% \Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. + +% \begin{figure}[!t] +% \centering +% \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} +% \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} +% \label{fig:conceptual_experiment} +% \end{figure} -Undeterred, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. +%Despite the aforementioned results, we set out for a different solution to show +We also show \mysystem is capable of incorporating a predictive analytics engine. Instead, we designed a \emph{conceptual experiment}. In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy. Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment. -The results are in \Cref{fig:failure_likelihood} on the right side. -\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly. +The results are shown in \Cref{fig:conceptual_experiment}. + +\subsection{Discussion}\label{sss:experiment2_discussion} + +In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +For any failure model, at any time interval, it is almost as likely to accurately model the actual failures as the other models. +Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +\Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. + + +\begin{figure}[!t] + \centering + \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} + \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} + \label{fig:conceptual_experiment} +\end{figure} + +\Cref{fig:conceptual_experiment} shows that using a perfect predictive-analytics engine, \mysystem is capable of lowering the total number of failures significantly. +In this experiment, we used only 3 failure traces from user reports of Twitter, WhatsApp, and Gmail. +Importantly, we assume in this experiment a perfect precognition of what failures might occur and when. +While unrealistic in a practical scenario, this assumption serves to approximate the potential gains from incorporating a predictive analytics engine with \mysystem. +The results are only indicative. +In the figure we can see that the mean number of failures in a ``physical datacenter'' (\ie \code{OpenDC}) is much higher without the insights of \mysystem. + +\section{Experiment 3: Failure Exploration}\label{ss:experiment3} + +For this experiment, we adapt the idea of using multi-modal simulation from Nicolae \etal~\cite{DBLP:conf/cf/NicolaeNTI26} to estimate failures with \gls{dcdt}. + +\subsection{Context}\label{sss:context_experiment3} +In this experiment we use the \gls{dcdt} to explore the number of terminated tasks when different failure injection algorithms are used. +The primary objective of this experiment is to identify the worst-case scenario and best-case scenario of running a workload prone to failures. + +Nicolae \etal explore how multi-modal simulation can provide additional insight into datacenter operation and can potentially halve the error of singular models~\cite{DBLP:conf/cf/NicolaeNTI26}. +For our work, we adopt a similar approach to failure prediction. +Imagine the following sequence of events: +\begin{enumerate}[label=\textbf{\arabic*.}, align=left] + \item The datacenter receives a scheduled workload, and the operators want to know the worst-case scenario and best-case scenario of how severe failures they can expect. + \item Datacenter engineers ask the digital twin to run the workload first with different failure injection algorithms. + \item After running the simulation, the \gls{dcdt} can provide the maximum and minimum number of expected failures. + \item By continuously monitoring the datacenter during the runtime of the real workload, the operators can use the estimate to make more informative decisions. +\end{enumerate} +\begin{figure} + \centering + \includegraphics[width=0.85\linewidth]{images/victim_selector_comparison.pdf} + \caption[Comparison of different victim selector algorithms.]{Comparison of different algorithms used for injecting failures into the simulated digital twin to explore the worst-case and best-case scenarios.} + \label{fig:victim_selector_comparison} +\end{figure} +For this experiment we used a SURF month trace running on the SURF-SARA cluster, as explained in \Cref{ss:experiment_parameters}. +Additionally, we use the Gmail failure trace to inject failures, and use 3 different algorithms for selecting the hosts to fail: \code{CpuUtilSelector}, \code{RoundRobinSelector}, \code{RandomVictimSelector}. +The results are depicted in \Cref{fig:victim_selector_comparison}. + +\subsection{Discussion}\label{sss:discussion_experiment3} +\Cref{fig:victim_selector_comparison} shows using different victim selection algorithms can effectively present the worst-case and best-case number of tasks terminated before running the real workload. +The victim selector that achieves the most tasks terminated is the \code{CpuUtilSelector}, which stops the hosts with the highest \gls{cpu} utilization. +It reaches over 700 terminated tasks (709) in total. +The \code{Random} selector, which randomly picks the hosts to stop, achieves the second-highest number of tasks terminated, at around 450. -\section{Experiment 3: Additional Experiment}\label{ss:additional_experiment} +The most optimistic scenario for datacenter operators tested here is the \code{RoundRobin} selector, which picks the hosts to terminate in a round-robin fashion. +No more than 130 tasks are terminated using this fault injection algorithm. -\section{Discussion}\label{ss:discussion_evaluation} +To summarize, this experiment shows that \mysystem can be used to explore different failure algorithms for the benefit of datacenter operators. +Using \mysystem, on-site engineers can effectively estimate the best-case and worst-case number of terminated tasks during a workload. diff --git a/content/implementation.tex b/content/implementation.tex index 571afd3..79c5191 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -3,17 +3,19 @@ \section{Overview}\label{ss:overview_implementation} \begin{mynote} - The contribution of this chapter is two-fold: + Addressing RQ$_3$ together with the next chapter, the contribution of this chapter is two-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] \item We implement the real-world prototype of \mysystem (see \Cref{ss:detailed_implementation_overview}) realizing key features of the design. - \item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming}) + + \item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming}). + \end{enumerate} \end{mynote} \section{Detailed Implementation of \mysystem}\label{ss:detailed_implementation_overview} -In this section we describe the detailed implementation of \gls{my_system}. -After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. +In this section, we describe the detailed implementation of \gls{my_system}. +After completing it, the reader should understand the technical decisions, choice of tools, and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}. To understand \gls{my_system} it is crucial to provide a holistic view on the prototype. @@ -30,7 +32,7 @@ Lastly, \Cref{ss:programming} carefully explains the design decisions behind the At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications. The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}. -In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. +To facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}. To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}. To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}. @@ -77,13 +79,13 @@ For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a t \begin{figure}[t] \centering \includegraphics[width=\linewidth]{images/flow_diagram.png} - \caption{The data flow within \gls{my_system}.} + \caption{The data flow within \mysystem.} \label{fig:flow_diagram} \end{figure} -\section{Data Flow}\label{ss:data_flow} +\section{Main Data Flow}\label{ss:data_flow} -In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram. +In this section, we describe the data flow within \Cref{fig:implementation} using a separate diagram. Efficient data flow is of utmost importance to \gls{dt}s. In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}. In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data. @@ -133,7 +135,7 @@ Scheduling in \code{OpenDC} also works using this method. \Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation. A task (\myCircled{1}) is represented by its submission time, duration and computational requirements. -In order to be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). +To be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time. \code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}. @@ -148,7 +150,7 @@ The \code{SimHost} object is created via the \code{HostProvisioningStep} class ( \section{Extensions to \code{OpenDC}}\label{ss:extensions} \code{OpenDC} is a state-of-the-art datacenter simulator. -In order to turn it into a \gls{dt}, we have made several design decisions and extensions. +To turn it into a \gls{dt}, we have made several design decisions and extensions. \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{SmartScheduler}}\\ @@ -165,6 +167,17 @@ In order to turn it into a \gls{dt}, we have made several design decisions and e \item \textbf{\code{HTTPClient}}\\ The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter. We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice. + \item \textbf{\code{CpuUtilVictimSelector}}\\ + To ensure \code{OpenDC} can model different failure injection algorithms, we added new methods for selecting the hosts to be stopped. + The \code{CpuUtilVictimSelector} selects hosts based on their current \gls{cpu} utilization (\ie the hosts with the highest utilization are stopped first). + \item \textbf{\code{RoundRobinVictimSelector}}\\ + The \code{RoundRobinVictimSelector} is another algorithm to used to inject failures into hosts. + This method works in a round robin fashion, using the alphabetical order of the hosts (\ie by name). + For example, in a scenario hosts \code{H-01} to \code{H-20} are stopped first, and then \code{H-20} to \code{H-40} are stopped next, \etc + \item \textbf{\code{RandomVictimSelector}}\\ + This is the default \code{OpenDC} victim selection algorithm. + It randomly chooses which host to stop when injecting failures. + This component is left unmodified, but we include it here for the purposes of listing all the failure injection algorithms we use together. \end{enumerate} \begin{figure}[t] \centering @@ -182,12 +195,12 @@ For future work, we envision a system that implements the reference architecture \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{AnalyticsEngine}}\\ - The \code{AnalyticsEngine} module is necessary in order to encapsulate the logic of data preprocessing and analysis from monitoring. + The \code{AnalyticsEngine} module is necessary to encapsulate the logic of data preprocessing and analysis from monitoring. This component can contain capabilities for different statistical metrics, subject to \gls{dt}s focus. In \gls{my_system} \code{AnalyticsEngine} continuously checks whether the incoming datacenter sensor readings exceed different thresholds. For example, the \code{AnalyticsEngine} is capable of calculating a similarity score $S$ between potential failure distributions and the true failure distribution. \item \textbf{\code{HTTPServer}}\\ - The \code{HTTPServer} is crucial for interrupting the operation of the datacenter in order to adjust its operation or offer insights. + The \code{HTTPServer} is crucial for interrupting the operation of the datacenter to adjust its operation or offer insights. It maintains a python \code{Queue} structure. The \code{Queue} \emph{producer} is the \code{AnalyticsEngine} (\grayCircled{4}). The \emph{consumer} is the \code{HTTPClient} within \code{OpenDC} (\ie the real datacenter, (\myCircled{2}), (\myCircled{1})). diff --git a/content/intro.tex b/content/intro.tex index 3f23487..85ae205 100644 --- a/content/intro.tex +++ b/content/intro.tex @@ -17,41 +17,57 @@ In the modern \gls{ai} economy, datacenters need diverse and scalable server arc %Nowadays, datacenter operators try to meet AI-customer expectations by adding more specialized hardware~\cite{DBLP:conf/date/MilojicicFDR21}, at the cost of increased system complexity. In return, operating a modern datacenter warehouse with thousands of diversified servers presents a difficult challenge that requires fast and well-informed decisions from on-site engineers. -The computational requirements of \gls{ai} are expected to increase in the future~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +The computational requirements of \gls{ai} are expected to increase in the future. +\Cref{fig:explosive_growth_in_ai_computational_requirements} presents the speed and scale which drives the compute requirements of \gls{ai} models. +From 2012 to 2024, the compute power needed to train the models grew by $10^{7}$, a trend that is expected to continue~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. Because of this, datacenter complexity will continue to grow, and it will become more difficult to manage~\cite{DBLP:conf/icdcs/IosupUVAEHTBT18}. Future servers and software related services to them will include even more specialized hardware, which, while improving datacenter performance, will exhibit behaviour that is harder to predict. Already the rapid expansion of datacenters has increased the presence of service failures across all cloud services~\cite{DBLP:conf/acsos/TalluriOVTI21}. Preventing failure-caused outages in advance could help datacenter operators reduce operational costs, as over 20\% of all reported outages amount to more than 1 million US\$~\cite{DBLP:report/AnnualOutageAnalysis2025}. %Moreover, datacenter outages can have catastrophic consequences, cite Fabian. -In short, the high computational demand of \gls{ai} and the end of Dennard's scaling have resulted in the rise of larger and more heterogeneous datacenter architectures~\cite{DBLP:conf/date/MilojicicFDR21}. +%In short, the +The high computational demand of \gls{ai} and the end of Dennard's scaling have resulted in the rise of larger and more heterogeneous datacenter architectures~\cite{DBLP:conf/date/MilojicicFDR21}. Both events create a need for more careful datacenter management to tackle the unprecedented complexity and ensure availability of all cloud services. To address this new problem a concept of a datacenter \gls{dt} was proposed~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +% Explain DT here, very briefly +A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. +The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{Wikipedia:article/DigitalTwin}. +% +This thesis explores the use of digital twins to cope with increasing operational complexity in datacenters, by enabling both historical and online data to pass through the digital twin's analytical and predictive loops. +\begin{figure}[t] + \hspace{-1.75cm} + \centering + \includegraphics[width=0.8\linewidth]{images/ai_timeline.pdf} + \caption[Explosive growth in AI computational requirements.]{Explosive growth in AI computational requirements drives datacenter upgrades (source: NVIDIA Analysis: reproduction with NVIDIA permission by~\cite{DBLP:journals/computer/AthavaleBBMMPS24}). \emph{X}-axis presents the year, and the \emph{Y}-axis presents the training compute requirement (in petaFLOPs). The plot shows the one of the reasons behind why the complexity of datacenters is rising.} + \label{fig:explosive_growth_in_ai_computational_requirements} +\end{figure} \section{Context}\label{s:context} % A digital twin is often called a virtual twin. % The communication between a physical entity and the digital twin is referred to as a digital thread. -A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. -The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{Wikipedia:article/DigitalTwin}. +% +% A \gls{dt} is a virtual model of an intended or actual real-world system that serves as its counterpart for purposes such as simulation, integration, testing, monitoring and maintenance. +% The digital twin replicates the physical system to predict failures, prescribe real-time actions for mitigating unexpected events, observing and evaluating the behaviour of the system~\cite{Wikipedia:article/DigitalTwin}. -Most modern \gls{dt} usages are related to prognostics and system health management~\cite{DBLP:conf/cirp/TAO2018169}. +Much modern \gls{dt} usages are related to prognostics and system health management~\cite{DBLP:conf/cirp/TAO2018169}. For example, in aerospace engineering, the \gls{dt} analyzes operational data (\eg temperature, vibration) to predict when a airplane component is likely to fail. The \gls{dt} can reliably manage the health of the physical entity by detecting fatigue cracks on aircraft wings or damage to the wind turbine blades~\cite{DBLP:journal/IJAE/Teugel2011}. This allows maintenance to be scheduled proactively, reducing unplanned downtime and preventing catastrophic failures. Forecasting future maintenance and managing the physical health of an object or facility are the prime purpose of many \gls{dt}s used in practice~\cite{DBLP:conf/AIAA/Teugel2012}. The concept of a \gls{dt} began in 1960s, at the \gls{nasa}~\cite{Nature:article/Görtz2026}. -\gls{nasa} pioneered the concept in order to debug issues with its spacecraft. +\gls{nasa} pioneered the concept to debug issues with its spacecraft. However, the term ``digital-twin'' dates back to 2003, when Dr. Michael Grieves of Dassault Syst\'emes introduced the 3 core components of a \gls{dt}: the virtual entity, physical entity and the two-way connection (see Figure \ref{fig:simple_dt}). Due to insufficient technological foundations, little work is available on \gls{dt}s between 2003 and 2018, and it is only with the rapid growth of cloud computing, \gls{iot} and Big Data analytics that \gls{dt}s have re-emerged. Today, research is focused on bridging the gap between the long-established foundations of \gls{dt}s and new, novel applications in academia and industry, such as the \gls{dcdt}~\cite{DBLP:conf/cirp/TAO2018169, DBLP:journals/computer/AthavaleBBMMPS24}. -A \gls{dcdt} mirrors the structure, context and behaviour of a datacenter~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. +A \gls{dcdt} mirrors the structure, context, and behaviour of a datacenter~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. The foundation to any digital twin is good monitoring and sensing capabilities in the physical entity. Datacenters, meet this requirement easily because they already connect hundreds of monitoring sensors. -With hundreds of gigabytes of useful information coming from distributed \gls{iot} sensors inside the warehouse, we can gain insight into failure patterns, energy usage, heat dissipation \etc +With hundreds of gigabytes of useful information coming from distributed \gls{iot} sensors inside the warehouse, we can gain insight into failure patterns, energy usage, heat dissipation, \etc What remains challenging is to connect the physical and virtual spaces with a bi-directional connection and to use the monitoring insights and data analysis results for autonomous decision-making. Crucial to \gls{dcdt} operation are predictive capabilities and the continuous interaction with the real-world datacenter. @@ -59,7 +75,7 @@ Crucial to \gls{dcdt} operation are predictive capabilities and the continuous i There already exist \gls{dcdt} deployments. For example, ExaDigiT~\cite{DBLP:conf/sc/BrewerMKWBHSGGW24} is a framework for digital twin development of supercomputers. It has been demonstrated at the Frontier supercomputer and it facilitates virtual prototyping and system optimization. - +% Nonetheless, existing \gls{dcdt}'s are still very limited in their capabilities as the definition and scope of a \gls{dcdt} concept is shallow and unclear. After all, only recently did the hardware capabilities needed to continuously simulate a datacenter become available~\cite{DBLP:conf/cirp/TAO2018169}. Many \gls{dcdt} frameworks still lack critical data analysis components, fault detection mechanisms, profiling techniques \etc~\cite{DBLP:conf/wosp/SumanCNTMI24}, rendering them unusable in large-scale systems. @@ -84,9 +100,13 @@ In this work, we address the lack of a unified \gls{dcdt} system model and the a We argue that the current state-of-the-art \gls{dcdt}'s lack sufficient predictive capabilities that are essential to real-time facility management of a modern datacenter. Because the main purpose of many \gls{dt}s is to forecast the short and long-term facility behaviour, \gls{dcdt} without predictive capabilities cannot maintain the health of the datacenter effectively. We posit that including holistic predictive analysis in \gls{dcdt} design can aid in efficient datacenter management and prevent missing \gls{sla}'s. +% For example, preventing compute failures could greatly benefit datacenter operators. -To enable insights from both historical data and immediate telemetry, we propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. + + +%To enable insights from both historical data and immediate telemetry, we propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. We envision \gls{dcdt}'s as systems indispensable in future datacenters, actively interacting with the real-world facility, lowering operational costs and predicting hardware failure and software faults. +In particular, to enable insights from both historical data and immediate telemetry, we propose that digital twinning can be enhanced by integrating predictive analytics through \gls{oda}. Our solution to this problem encompasses different levels of \gls{oda} (\eg in-band analytics, out-of-band analytics) for holistic datacenter modelling. Together with a unified \gls{dcdt} system model and a revolutionary evaluation method of \gls{dcdt}'s, we hope to bring the modern vision of \gls{dt}'s to datacenters. @@ -94,7 +114,7 @@ Together with a unified \gls{dcdt} system model and a revolutionary evaluation m \emph{Main Research Question:} How to enable predictive analytics in datacenters through digital twinning?\\ -\noindent We divide the problem of designing a predictive \gls{dcdt} into three research questions: +\noindent We scope the problem of designing a predictive \gls{dcdt} by proposing three research questions: \begin{enumerate}[label=\emph{RQ\textsubscript{\arabic*}}, align=left, itemsep=0pt] % First research question stolen from Capelin by Georgios Andreadis and adapted to my work. \item \emph{How to assess the current state-of-the-art of digital twinning for datacenters?}\\ @@ -117,9 +137,9 @@ Together with a unified \gls{dcdt} system model and a revolutionary evaluation m \section{Research Methodology}\label{s:research-methodology} To answer \emph{RQ\textsubscript{1}} we conduct a literature review as proposed by \textit{Kitchenham et al.} \cite{DBLP:journals/infsof/KitchenhamPBBTNL10} along with the guidance of the supervisor. Firstly, we determine the right review method. -Secondly, we identify the various works related to \gls{dcdt}'s using different search strings +Secondly, we identify the various works related to \gls{dcdt}'s using different search strings~ (\eg ``Datacenter Digital Twinning'', ``ICT Virtual Twin'') and query combinations (``Datacenter \code{AND} Maintenance''). -To search for the results we use the digital libraries of Google Scholar, DBLP, ACM Digital Library, IEEExplore, Springer \etc +To search for the results we use the digital libraries of Google Scholar, DBLP, ACM Digital Library, IEEExplore, Springer, \etc Thirdly, we select work relevant to our research and organize the details of each article into a table. A potential outcome of this could be a system model for \gls{dcdt}'s. We envision the literature review supplying us with potential use-cases for the predictive \gls{dcdt}. @@ -157,16 +177,22 @@ We define the correct experiment setup(s) and perform the experiments on a speci \item We provide the experiment setup, validation and evaluation of \mysystem for detecting and predicting datacenter failures in real-time as an Open Science artifact. \end{enumerate} \end{enumerate} + \section{Academic Integrity Declaration}\label{s:academic_integrity_declaration} + +The statements in this section address an ongoing and seemingly growing problem in academia, that of integrity breaches primarily through the generation of material using AI techniques, such as LLM services. + \subsection{Non-Plagiarism Declaration}\label{ss:plagiarism-declaraion} I hereby declare that this thesis is my own independent work and writing. The thesis does not contain any material copied from other sources (person, Internet, or \gls{ai}), and has not been submitted for assessment elsewhere. I acknowledge that the usage of material from other works or paraphrase of such material without proper citations or credit will be treated as plagiarism. I declare that this thesis is free from \gls{ai} generated content and has been written without the help of any \gls{ai} tools. -To order to adhere to the strictest restrictions on AI-usage in higher education, this work follows the Berkley School of Law Artificial Intelligence Policy, as stated in \url{https://www.law.berkeley.edu/wp-content/uploads/2026/05/AI-Final-Policy-26.pdf}. + +To adhere to the current (strictest) restrictions on AI-usage in higher education, this work follows the Berkley School of Law Artificial Intelligence Policy, as stated in \url{https://www.law.berkeley.edu/wp-content/uploads/2026/05/AI-Final-Policy-26.pdf}. \subsection{Preventing Reference Fraud}\label{ss:plagiarism_references} -I hereby declare that all the references in this thesis refer to genuine scientific work published in peer-reviewed journals or other sources of reliable and safe online information (\eg Wikipedia articles) and have been used in accordance to the article authors' wishes. +I hereby declare that all the references in this thesis refer to genuine scientific work published in peer-reviewed journals or other sources of reliable and safe online information~ (\eg Wikipedia articles) and have been used in accordance to the article authors' wishes. + Additionally, under the guidance of the supervisor this work adheres to the strictest rules for referencing and to prove the originality of all references, each \BibTeX citation contains a \texttt{note} field with the following comment: \emph{This BibTeX citation comes from:} followed by the URL leading directly to the citation source. In case of citations not formatted in \BibTeX, the same format follows but with adequate reference-style name (\eg APA, Chicago, MLA). @@ -183,14 +209,18 @@ This work addresses the four grand societal challenges related to this goal: \be \item sustainability \item usability \end{enumerate*}~\cite{DBLP:journals/corr/IosupKLVG22}. -\mysystem addresses (1) directly by making large-scale datacenter management easier. + +\mysystem addresses goal (1) directly by making large-scale datacenter management easier. + We address (2) by ensuring our work adheres to the \gls{fair} principles of Open Science. Moreover, in this thesis we try to make \gls{dcdt} systems more understandable to the broader scientific community by providing a unified system model. Additionally, we contribute to responsible software design by adhering to best software engineering practices in the design of the prototype. -(3) is addressed indirectly, as the consequences of the insights provided by a holistic, \gls{oda} powered \gls{dcdt} can help datacenter managers make decisions that are more sustainable in the future. + +Goal (3) is addressed indirectly, as the consequences of the insights provided by a holistic, \gls{oda} powered \gls{dcdt} can help datacenter managers make decisions that are more sustainable in the future. + We contribute to (4) by helping predict unexpected failures and lowering operational costs, ensuring datacenters can continue to be usable in the future. We believe this work has a strong societal impact due to addressing the four grand societal challenges described by Iosup \etal and we hope through this work we can advance the scientific research community towards a more sustainable future. -Abiding the FAIR data principles, the entire source code of the prototype and related work has been made available at the \url{https://git.denounce.ai/opendc.git} repository. +Abiding the FAIR data principles, the entire source code of the prototype and related work has been made available at the \url{https://github.com/M-J-Kwiatkowski/opendc} and the \url{https://github.com/M-J-Kwiatkowski/sunfish} repositories. The reuse and reproduction of experiments is explained in a detailed guide at the root of the repository, along with the necessary dependencies and experimental setup. % Comment from Alexandru is to merge the above repository into the AtLarge repository, with the proper attribution to you. % I will do this once the project is over. diff --git a/content/preamble/abstract.tex b/content/preamble/abstract.tex index c8078b4..9c2247a 100644 --- a/content/preamble/abstract.tex +++ b/content/preamble/abstract.tex @@ -12,6 +12,8 @@ Without predictive maintenance and forecasts, system administrators cannot make In this work, we propose to enable predictive analytics for datacenters using digital twinning. We survey the datacenter digital twinning field, and organize our findings into a system model. -Additionally, we design \mysystem~-- a novel reference architecture for predictive datacenter digital twins, and evaluate it through prototype-based experiments. +Based on real-world use-cases we design \mysystem~-- a novel reference architecture for predictive datacenter digital twins, and evaluate it through prototype-based experiments. Our results indicate \mysystem is capable of reliably differentiating between mild and severe compute failures, and can successfully incorporate a predictive analytics engine to the benefit of datacenter managers. +The main findings are backed-up by experiments which replicate a peer-reviewed publication evaluation approach to show that \mysystem can notify datacenter operators of compute failures. +Moreover, to show that the system can incorporate a predictive analytics-engine, we have conducted a novel, conceptual experiment to exhibit the potential future gains of a holistic Datacenter Digital Twin. diff --git a/content/preamble/acknowledgement.tex b/content/preamble/acknowledgement.tex index 180ffad..81660b5 100644 --- a/content/preamble/acknowledgement.tex +++ b/content/preamble/acknowledgement.tex @@ -4,7 +4,7 @@ \end{center} I wish to express my thanks to the many people who have helped me in preparation of this thesis. -Dante Niewehuis, my daily supervisor, with patience and understanding mentored me during our collaboration, which led to the timely and successful completion of this thesis. +Dante Niewenhuis, my daily supervisor, with patience and understanding mentored me during our collaboration, which led to the timely and successful completion of this thesis. Alexandru Iosup, my 1\textsuperscript{st} supervisor, provided invaluable feedback and suggestions that helped immensely during the design and writing. |