diff options
Diffstat (limited to 'content')
| -rw-r--r-- | content/background.tex | 2 | ||||
| -rw-r--r-- | content/conclusion.tex | 11 | ||||
| -rw-r--r-- | content/design.tex | 102 | ||||
| -rw-r--r-- | content/evaluation.tex | 190 | ||||
| -rw-r--r-- | content/implementation.tex | 2 | ||||
| -rw-r--r-- | content/intro.tex | 27 | ||||
| -rw-r--r-- | content/preamble/abstract.tex | 40 | ||||
| -rw-r--r-- | content/preamble/acknowledgement.tex | 18 | ||||
| -rw-r--r-- | content/preamble/dedication.tex | 18 |
9 files changed, 331 insertions, 79 deletions
diff --git a/content/background.tex b/content/background.tex index 6671537..76ce5b3 100644 --- a/content/background.tex +++ b/content/background.tex @@ -1,5 +1,6 @@ \chapter{Background}\label{s:background} +\section{Overview}\label{ss:background_overview} \begin{mynote} The contribution in this chapter is three-fold: @@ -218,3 +219,4 @@ Kalibre takes the best of both \gls{ml} and \gls{cfd} approaches and achieves su %In your work, consider adding such an endpoint, albeit explain in future work that you envision \emph{implementing} this endpoint in the future. +\section{Discussion}\label{ss:background_discussion} diff --git a/content/conclusion.tex b/content/conclusion.tex index 122b2be..83a499c 100644 --- a/content/conclusion.tex +++ b/content/conclusion.tex @@ -9,11 +9,11 @@ Starting from a thorough investigation into the new, emerging field of datacente We ended our project with a novel evaluation method used in a set of exhaustive experiments. We answer the main research question by addressing each sub-research question. -\section{Answers to Research Questions}\label{ss:answers_to_rqs} +\section{Answers to Each Research Question}\label{ss:answers_to_rqs} \begin{enumerate}[label=\emph{RQ\textsubscript{\arabic*}}] \item \emph{How to asses the current state-of-the-art of digital twinning for datacenters?}\\ - In order to answer this research question, we conducted a semi-structured literature review. + To answer this research question, we conducted a semi-structured literature review. Our findings indicate that the field of datacenter digital twinning is still under development, and there exist few \gls{dcdt} deployments. The current efforts in modelling datacenters focus on very specialized parts of datacenter management, \ie cooling and heat modelling, network mapping. Many crucial features, inherent to the \gls{dt} definition are still missing from current \gls{dcdt}s. @@ -38,10 +38,13 @@ We answer the main research question by addressing each sub-research question. \section{Future Work}\label{ss:future_work} +\subsection{A Strong, New Principle of \gls{dcdt} Design}\label{sss:future_work_in_analytics} We envision \gls{dcdt}s as systems that encompass features necessary to model the entire datacenter. -It came to our attention that with the explosive growth of \gls{ai} and the diversification of datacenters under way, \gls{dt}s will be indispensable in datacenter management. -To power the predictions we envision an \gls{ml}-based inference engine as a necessary component of digital twinning. +It came to our attention that with the growth of \gls{ai} and the diversification of datacenters under way, \gls{dt}s will be indispensable in datacenter management. +To power the predictions, we envision an \gls{ml}-based inference engine as a necessary component of digital twinning. The need for \gls{ml} arises naturally in scenarios where large volumes of data, requiring little to no preprocessing meet the demand for estimating future facility behaviour. + +\subsection{}\label{sss:future_work_in_failures} For future work in failure prediction, we envision an \gls{abc} approach to estimate the real failure distribution within the datacenter. Additionally, power usage optimization is a critical concern in datacenter management. We hope future attempts to enhance datacenter digital twinning can enable datacenter operators with actionable insights towards lowering the power consumption. diff --git a/content/design.tex b/content/design.tex index b14dfec..48d6ffd 100644 --- a/content/design.tex +++ b/content/design.tex @@ -1,4 +1,6 @@ -\chapter{Design}\label{s:design} +% Changed chapter name, as suggested by Alexandru +\chapter{Design of \mysystem, a Digital Twin For Predictive Analysis of Datacenters}\label{s:design} +\section{Overview}\label{ss:design_overview} \begin{mynote} Our contribution in this chapter is three-fold: \vspace{-0.2cm} @@ -106,30 +108,104 @@ In addition to the functional requirements, we also present non-functional requi Without \textbf{NFR1}, \mysystem's insights will not be timely, and will be useless to datacenter operators. \item \textbf{The system should log the ingestion and processing of metrics.} \\ The system should provide a log of the current network traffic to and from the digital twin. - \item \textbf{The system should adhere to modern software development standards.} \\ The system should follow modern coding principles and guidelines to ensure reproducibility and usefulness for future work. Without \textbf{NFR3}, it will be difficult for current and future digital twin developers to include predictive analytics using \mysystem in their deployments. - \item \textbf{The system should provide insights at varying levels of confidence.} \\ With the huge amount of telemetry data incoming from the physical twin, the digital counterpart must be able to filter and pre-process the telemetry. Without \textbf{NFR4}, the system will present overwhelming amount of information to its users, rendering it unusable. \end{enumerate} -\section{Design of \emph{Sunfish}}\label{ss:design_of_mysystem} - -\begin{figure}[t] - \includegraphics[width=\linewidth]{images/ref_architecture.png} - \caption{ The predictive datacenter digital twin reference architecture. +\begin{figure}[ht] + \centering + \includegraphics[width=0.75\linewidth]{images/ref_architecture.png} + \caption{The predictive datacenter digital twin reference architecture. We call the system \emph{Sunfish}. - The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months} + The architecture was designed with the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} over several iterations in the past months.} + \label{fig:reference_architecture} \end{figure} -\subsection{Digital Thread} +\section{Overview of \mysystem Architecture}\label{ss:design_of_mysystem} +As a result of the \emph{AtLarge Design Process}~\cite{DBLP:conf/icdcs/IosupVTETBFMT19} designed a reference architecture for a predictive datacenter digital twin. +\Cref{fig:reference_architecture} encompasses 4 main elements: +\begin{enumerate*}[label=(\Roman*)] + \item physical datacenter + \item digital thread + \item digital twin + \item predictive analytics. +\end{enumerate*} -\subsection{Data Storage} +The physical datacenter (I) encompasses 3 core elements important to digital twinning. +Workloads (\myCircled{1a}) include the hardware requirements of each datacenter job and the submission time. +They are executed on the datacenter compute (\myCircled{1b}), which is controlled partly by the Datacenter Operators (\myCircled{1c}). +Component (\myCircled{1c}), while seemingly unimportant, is crucial to the digital twin design. +We envision \gls{dcdt}s as systems that contain a human-in-the-loop, which can control and overwrite the system's autonomous decisions. +Datacenter Operators (\myCircled{1c}) interact with both the Servers (\myCircled{1b}), and have the ability to overwrite the potential autonomous digital twin decisions, stemming from component (\myCircled{2c}), the System Knobs. -\subsection{Predictive Analytics Module} +The Digital Thread (II) is a novel contribution from \Cref{s:background}. +It separates the physical world from the virtual world, and contains components that do not belong to either twin, or belong to both twins. +It contains the Interactive Dashboard (\myCircled{2a}), the Message Broker (\myCircled{2b}), and System Knobs (\myCircled{2c}). -\section{Discussion}\label{ss:design_discussion} +To fulfill the functional requirements of our system, we incorporate element (\myCircled{2a}), the Interactive Dashboard to our system. +This dashboard ingests data coming directly from the Message Broker (\myCircled{2b}) and from the long-term storage (\myCircled{3a}), the Database. +The Interactive Dashboard (\myCircled{2a}) allows datacenter operators to see the telemetry data arrive to the digital twin in real time. + +The Message Broker (\myCircled{2b}) is a crucial component to the reference architecture, because it facilities the physical twin $\rightarrow$ virtual twin connection. +A low-latency, high-throughput message broker partly meets our functional requirements to enable arbitrary amounts of telemetry data transfer. +We elaborate on the specific components that make up the message broker (\myCircled{2b}) in \Cref{sss:message_broker}. + +The System Knobs (\myCircled{2c}) represent the different cogs within the datacenter software and hardware (\myCircled{1b}) that can be adjusted during runtime (\eg to optimize \gls{pue}, change cooling strategy, allocate compute resources). +For example, System Knobs (\myCircled{2c}) within the datacenter scheduler can be tuned to schedule jobs on Servers (\myCircled{1b}) that are least likely to experience future downtime. +The autonomous actions of the digital twin (the tuning of the System Knobs (\myCircled{2c})) can be further adjusted by Datacenter Operators (\myCircled{1c}). + +In our design, we explicitly differentiate between the physical and virtual space by including the Digital Twin (III) in a separate box. +The Digital Twin (III) constitutes of the long-term storage (\myCircled{3a}), short-term storage (\myCircled{3b}), the API Server (\myCircled{3c}) and the Predictive Analytics (IV) module. + +Long-term storage, namely a Database (\myCircled{3a}) is crucial to enable real-time visualizations (\myCircled{2a}), and to model long-term system behaviour. +This fulfills the functional requirements for our system, as we set out to differentiate between in-band data analytics, and out-of-band data analysis. +The data is consumed by the Database (\myCircled{3a}) directly from the Message Broker (\myCircled{2b}). +In-between, simple data pre-processing can take place. +Short-term storage, in the form of a Cache (\myCircled{3b}) is essential for in-band data analytics, ``on-the-go''. +The Cache (\myCircled{3b}) retains the data consumed from the Message Broker (\myCircled{2b}), and for a short period of time keeps a copy. +The data analytics performed on the Database (\myCircled{3a}) dataset and the Cache (\myCircled{3b}) adhere to the use-cases for predictive (long-term) and descriptive (long-term, short-term) data analytics. + +The Message Broker (\myCircled{2b}) enables one-way connection (physical twin $\rightarrow$ digital twin). +The API Server (\myCircled{3c}) enables the digital twin $\rightarrow$ link. +The API Server (\myCircled{3c}) communicates directly with the System Knobs (\myCircled{2c}) in order to take meaningful action, based on the (predictive) insights generated by the Digital Twin (III). +Additionally, the Physical Twin (III) can query the API Server (\myCircled{3c}) for one-shot requests (\eg to create a new datacenter prototype configuration, to request special data analysis). +Moreover, the Datacenter Operators (\myCircled{1c}) can query the API Server (\myCircled{3c}) for extra insights, when necessary. + +The Predictive Analytics (IV) module is an extensible part of the reference architecture, enabling different kinds of predictive analysis. +In our design, to facilitate meaningful predictions we incorporate an Event-driven Simulator (\myCircled{4a}), Analytics Engine (\myCircled{4b}), and a Monitoring Service (\myCircled{4c}). + +We chose an Event-driven Simulator (\myCircled{4a}) as a core element of predictive analytics. +For the rationale behind this decision see \Cref{ss:datacenters}. +The predictions from the Event-driven Simulator (\myCircled{4a}) communicate directly with the Database (\myCircled{3a}) to store the simulation results, and with the Analytics Engine (\myCircled{4b}). + +The Analytics Engine (\myCircled{4b}) performs descriptive and predictive data analytics on the results from the Event-driven Simulator (\myCircled{4a}) and the telemetry data collected from the physical datacenter (stored in (\myCircled{3a}, \myCircled{3b})). +This component is highly extensible to accommodate the different types of analytics (\eg \gls{ml}, \gls{ai}, statistical methods). +The results of the data analysis are propagated to the API Server (\myCircled{3c}) to be queued for retrieval by the physical twin. + +The Monitoring Service (\myCircled{4c}) serves as a separate thread to watch over the collected telemetry in real-time. +It facilities detection of irregularities in collected data, which adheres to the functional requirements set out for the system. +Any discrepancies are communicated to the Analytics Engine (\myCircled{4b}) for further analysis, and potential insights. + +\begin{figure}[t] + \centering + \includegraphics[width=\linewidth]{images/message_broker.png} + \caption{The detailed view of the Message Broker (\myCircled{2b}) from \Cref{fig:reference_architecture}.} + \label{fig:message_broker} +\end{figure} +\section{The Digital Thread and Predictive Analytics}\label{ss:detailed_design} +\subsection{Message Broker}\label{sss:message_broker} +The Message Broker (\myCircled{2b}) is a component is slightly more complex, and necessities a separate diagram. +In \Cref{fig:message_broker} we present the composite elements that make up the Message Broker. +In particular, the \emph{schema registry} and the \emph{connector manager} play a crucial part in fulfilling the functional requirements of our system. +The \emph{schema registry} allows the telemetry producer to submit \emph{any} data format for sending (and storing) the telemetry data.The registry is responsible for detecting what kind of format is the data sent in, and automatically adjusting the schema within the \emph{data pipeline}. +The connector manager is responsible for joining multiple distinct services to a single data pipeline. +In the reference architecture, the consumers would constitute the Database (\myCircled{3a}), the Cache (\myCircled{3b}), and the Interactive Dashboard (\myCircled{2a}). +What is remarkable about the connector manager is the ability to swiftly connect more consumers to the system. +This way, the predictive digital twin can facilitate multiple different types of analytics engines or techniques. +Additionally, the setup is \Cref{fig:message_broker} is currently standard industry practice for large software deployments. +\section{Discussion}\label{ss:design_discussion} diff --git a/content/evaluation.tex b/content/evaluation.tex index 3329bba..8f587d1 100644 --- a/content/evaluation.tex +++ b/content/evaluation.tex @@ -1,13 +1,187 @@ -\chapter{Evaluation}\label{s:evaluation} +\chapter{Experimental Evaluation through a Real-World Prototype}\label{s:evaluation} +\section{Overview}\label{ss:evaluation_overview} +\begin{mynote} + The contribution of this chapter is two-fold: + \vspace{-0.2cm} + \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] + \item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}. + \item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}. + \end{enumerate} + Our findings indicate: + \vspace{-0.2cm} + \begin{enumerate}[label=\emph{F\textsubscript{\arabic*}}] + \item Digital twinning can be used for failure detection to the benefit of datacenter operators. + \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime. + \item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime. + \item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated tasks. + \end{enumerate} +\end{mynote} -\todo{ - Discuss the design of your experiments, the results you obtained, and how they - help in evaluating the claims you made in the introduction. You may also use the - evaluation results in this section to justify your design choices or assess the - contributions of different aspects of your design towards the overall goals. -} +\section{Experimental Setup}\label{ss:experimental_setup} +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/novel_eval_method.png} + \caption{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.} + \label{fig:novel_eval_method} +\end{figure} -\lipsum[1-16] +In this section we describe the technical setup used to evaluate \mysystem. +However, \Cref{fig:reference_architecture} assumes the system designer is capable of connecting the digital twin directly to the datacenter. +This raises a problem, we cannot just go and test digital twins on large systems, because we do not have large systems at hand. +Moreover, real-world experimentation is costly and unsustainable in the long run~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. +To overcome this problem, we present a novel datacenter digital twin method capable of evaluating a \gls{dcdt} without the physical datacenter. +\Cref{fig:novel_eval_method} details our approach. + +In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. +In our implementation, this is a second \code{OpenDC} +process (see \Cref{fig:implementation}). +The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. +For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}. + +The technical setup used for all experiment adheres to the \code{OpenDC} +documentation (see Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}). +The workload trace used for all experiments comes from BitBrains~\cite{DBLP:conf/ccgrid/ShenBI15}. +In the experiments we model a Dutch SURF datacenter for scientific computing. +The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}. +The scheduling policy for all experiments is the \code{FilterScheduler} which considers the RAM and CPU capacity for choosing hosts to run tasks on. +This scheduling policy is also used by \code{SmartScheduler}, as outlined in \Cref{ss:programming}, albeit with modifications to enable the system knobs to take autonomous action. + +In all experiments we use either \emph{failure traces} or \emph{failure models}. +For a brief explanation on the differences between the two, consult \Cref{sss:failures}. +In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +We chose a diverse range of failure models, based on the mean failure intensity in each trace (indicated in parentheses). +As a result, we chose the traces from: \begin{enumerate*}[label=(\Roman*)] + \item Gmail (53.26\%), + \item WhatsApp (57.97\%), + \item YouTube (62.1\%), + \item Twitter (65\%), + \item Facebook (64\%). +\end{enumerate*} +In \Cref{ss:experiment2} we used a failure trace from Skype. +This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}). +Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures. +For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5. +Importantly, in our figures we do not report the standard deviation of our experiments. +This is due to the fact that \code{OpenDC} is a fully deterministic simulator, and on each simulation run, given the same random seed will produce exactly the same results. +We believe the deviation in the results of the experiments stemming only from the random number generator is not meaningful, therefore none of the figures contain the standard deviation bars. + +\section{Experiment 1: Failure Detection}\label{ss:experiment1} +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/red_yellow_alarms.pdf} + \caption{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.} + \label{fig:red_yellow_alarms} +\end{figure} + +The purpose of this experiment is two fold: \begin{enumerate*} + \item to show our system works correctly + \item to show our system fulfills the functional and non-functional requirements. +\end{enumerate*} +To this end, we replicate an experiment from Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system. +The experimental setup is as defined in \Cref{ss:experimental_setup}. +The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)] + \item firstly, we use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload. + \item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm) + \item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached. + \item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold. +\end{enumerate*} +The results are in \Cref{fig:red_yellow_alarms}. +\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. +What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems. +Importantly, the more failure-intense the trace, the more alarms are raised on behalf of the digital twin. + +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf} + \caption{Comparison between the total number of raised alarms and the ground truth failure distribution during a BitBrains workload in the SURF-SARA cluster. The failure traced used in this experiment models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.} + \label{fig:alarms_vs_failures} +\end{figure} + +Additionally \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. +In the figure we can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. +For this visualization, we combined both the red and yellow alarms into a single metric. + +However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. +The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate. +What is surprising is Taheri \etal report almost negligible false positive rate and of their system. +Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate. +In our experiment, the numbers differ significantly. +\begin{figure}[t] + \centering + \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf} + \caption{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}). + Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. + We believe this is due to the irreconcilable differences between our experimental setups.} + \label{fig:failure_detecton_rate} +\end{figure} +\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%. +Compared to the DyTwin deployment, the difference is staggering. +However, the discrepancy stems from the fact in our setup we differentiate between different types of failures. +This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe. +Unusually, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace. + +\section{Experiment 2: Failure Prediction}\label{ss:experiment2} + +In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics. +Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures. +In this section we try to show \mysystem can additionally work well together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. + +\begin{figure}[ht] + \hspace{-0.8cm} + \begin{minipage}[b]{0.45\textwidth} + \centering + \includegraphics[width=1.2\linewidth]{images/failure_likelihood.pdf} + \end{minipage} + \hspace{1.2cm} + \begin{minipage}[b]{0.45\textwidth} + \centering + \includegraphics[width=1.2\linewidth]{images/conceptual_experiment.pdf} + \end{minipage} + \caption{Left figure shows the potential failure distribution likelihood to approximate the true failure distribution. + Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.} + \label{fig:failure_likelihood} +\end{figure} +\begin{figure}[ht] + \centering + \includegraphics[width=\linewidth]{images/failure_models_table.png} + \caption{The failure models table, by Javadi \etal~\cite{DBLP:journals/jpdc/JavadiKIE13}.} + \label{tab:failure_models_table} +\end{figure} + + +\subsection{Context}\label{sss:context_experiment2} +In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods. +Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions. +For the task at hand, we chose the Skype trace, because it is supported by 4 different failure models, based on past Skype workload data. +These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}. +The Skype trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +The goal was to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes. + + +Initial experiment results were unpromising. +Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly). +To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution. +Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time. +The results are in \Cref{fig:failure_likelihood}. + +In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$. +Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. +Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis. + +Undeterred, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. +Instead, we designed a \emph{conceptual experiment}. +In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy. +Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment. +The results are in \Cref{fig:failure_likelihood} on the right side. +\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly. + +\section{Experiment 3: Additional Experiment}\label{ss:additional_experiment} + +\section{Discussion}\label{ss:discussion_evaluation} diff --git a/content/implementation.tex b/content/implementation.tex index 5d243f6..0c31453 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -1,4 +1,4 @@ -\chapter{Implementation}\label{s:implementation} +\chapter{Implementation of \mysystem}\label{s:implementation} In this chapter we describe the implementation of \gls{my_system}. The main contribution of this chapter towards answering \emph{RQ3} is the prototype of \gls{my_system}. After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. diff --git a/content/intro.tex b/content/intro.tex index 36703f9..85ae411 100644 --- a/content/intro.tex +++ b/content/intro.tex @@ -1,5 +1,6 @@ \chapter{Introduction}\label{s:intro} -Presently, computer and network systems play a crucial part in the digital industry. +% Comment from Alexandru, change presently -> currently +Currently, computer and network systems play a crucial part in the digital industry. The transport, education and government sectors largely depend on digital services, which are hosted in datacenters~\cite{DBLP:journals/corr/IosupKLVG22}. To address the recent rise in demand for computation, due to the advancements in Artificial Intelligence, managers expand datacenters with new components and more heterogeneous architectures (\eg GPUs, NPUs)~\cite{DBLP:conf/date/MilojicicFDR21}. However, in return datacenter complexity increases significantly. @@ -11,13 +12,13 @@ Over 3 million jobs in the Netherlands directly depend on cloud services, which Since many public services continue to move online (\eg online administration and taxation, education), the fraction of Dutch professionals who depend on the cloud for work will exceed 35\% by 2025~\cite{DBLP:journals/corr/IosupKLVG22}. % What is changing? -In the modern \gls{ai} economy datacenters need diverse and scalable server architectures, because inference-based workloads require more heterogeneous server components (GPUs, TPUs, NPUs \etc) to perform well. -Nowadays, datacenter operators try to meet customer expectations by adding more specialized hardware~\cite{DBLP:conf/date/MilojicicFDR21}, at a cost of increased system complexity. +In the modern \gls{ai} economy, datacenters need diverse and scalable server architectures, because inference-based workloads require more heterogeneous server components (GPUs, TPUs, NPUs \etc) to perform well. +Nowadays, datacenter operators try to meet AI-customer expectations by adding more specialized hardware~\cite{DBLP:conf/date/MilojicicFDR21}, at the cost of increased system complexity. In return, operating a modern datacenter warehouse with thousands of diversified servers presents a difficult challenge that requires fast and well-informed decisions from on-site engineers. The computational requirements of \gls{ai} are expected to increase in the future~\cite{DBLP:journals/computer/AthavaleBBMMPS24}. -Datacenter complexity will continue to grow, and it will become more difficult to manage. -Future servers will include even more specialized hardware, which, while improving datacenter performance, will exhibit behaviour that is harder to predict. +Datacenter complexity will continue to grow, and it will become more difficult to manage~\cite{DBLP:conf/icdcs/IosupUVAEHTBT18}. +Future servers and software related services to them will include even more specialized hardware, which, while improving datacenter performance, will exhibit behaviour that is harder to predict. Already the rapid expansion of datacenters has increased the presence of service failures across all cloud services~\cite{DBLP:conf/acsos/TalluriOVTI21}. Preventing failure-caused outages in advance could help datacenter operators reduce operational costs, as over 20\% of all reported outages amount to more than 1 million US\$~\cite{DBLP:report/AnnualOutageAnalysis2025}. %Moreover, datacenter outages can have catastrophic consequences, cite Fabian. @@ -29,7 +30,9 @@ To address this new problem a concept of a datacenter \gls{dt} was proposed~\cit \begin{figure} \centering \includegraphics[width=0.8\linewidth]{images/simple_dt.pdf} - \caption{Elements of the digital twin ecosystem~\cite{DBLP:modsim24/presentation/Iosup2024} include: the insights and decisions coming from the digital twin (\myCircled{A}), the physical infrastructure (\myCircled{B}), the data coming from the physical twin telemetry (\myCircled{C}), and the digital counterpart to the physical twin (\myCircled{D}).} + \caption{Elements of the digital twin ecosystem~\cite{DBLP:modsim24/presentation/Iosup2024} include: the insights and decisions coming from the digital twin (\myCircled{A}), the physical infrastructure (\myCircled{B}), the data coming from the physical twin telemetry (\myCircled{C}), and the digital counterpart to the physical twin (\myCircled{D}). + This thesis focuses on components (\myCircled{A}), (\myCircled{C}), and (\myCircled{D}) in this ecosystem, proposing design improvements to (\myCircled{D}, \myCircled{C}), experimental improvements to (\myCircled{A}) and a new experimental technique to substitute (\myCircled{B}). + } \label{fig:simple_dt} \end{figure} @@ -157,19 +160,21 @@ We define the correct experiment setup(s) and perform the experiments on a speci \end{enumerate} \end{enumerate} -\section{Plagiarism Declaration}\label{s:plagiarism-declaraion} +\section{Academic Integrity Declaration}\label{s:academic_integrity_declaration} +\subsection{Non-Plagiarism Declaration}\label{ss:plagiarism-declaraion} I hereby declare that this thesis is my own independent work and writing. The thesis does not contain any material copied from other sources (person, Internet, or \gls{ai}), and has not been submitted for assessment elsewhere. I acknowledge that the usage of material from other works or paraphrase of such material without proper citations or credit will be treated as plagiarism. I declare that this thesis is free from \gls{ai} generated content and has been written without the help of any \gls{ai} tools. In order to adhere to the strictest restrictions on AI-usage in higher education, this work follows the Berkley School of Law Artificial Intelligence Policy, as stated in \url{https://www.law.berkeley.edu/wp-content/uploads/2026/05/AI-Final-Policy-26.pdf}. -\section{Reference Fraud}\label{ss:plagiarism_references} +\subsection{Preventing Reference Fraud}\label{ss:plagiarism_references} I hereby declare that all the references in this thesis refer to genuine scientific work published in peer-reviewed journals or other sources of reliable and safe online information (\eg Wikipedia articles) and have been used in accordance to the article authors' wishes. Additionally, under the guidance of the supervisor this work adheres to the strictest rules for referencing and to prove the originality of all references, each \BibTeX citation contains a \texttt{note} field with the following comment: \emph{This BibTeX citation comes from:} followed by the URL leading directly to the citation source. In case of citations not formatted in \BibTeX, the same format follows but with adequate reference-style name (\eg APA, Chicago, MLA). -\section{Societal Impact}\label{s:societal-impact} +\section{Societal Impact Including Open Science Artifacts + }\label{s:societal-impact} Any program that is difficult to understand and reason about is sure to accumulate technical debt. However, sometimes large-scale systems can be complex and hard to comprehend inherently. % Cite Frederick P Brooks here. @@ -188,10 +193,10 @@ Additionally, we contribute to responsible software design by adhering to best s (3) is addressed indirectly, as the consequences of the insights provided by a holistic, \gls{oda} powered \gls{dcdt} can help datacenter managers make decisions that are more sustainable in the future. We contribute to (4) by helping predict unexpected failures and lowering operational costs, ensuring datacenters can continue to be usable in the future. We believe this work has a strong societal impact due to addressing the four grand societal challenges described by Iosup \etal and we hope through this work we can advance the scientific research community towards a more sustainable future. - -\section{Open Science}\label{s:open-science} Abiding the FAIR data principles, the entire source code of the prototype and related work has been made available at the \url{https://git.denounce.ai/opendc.git} repository. The reuse and reproduction of experiments is explained in a detailed guide at the root of the repository, along with the necessary dependencies and experimental setup. +% Comment from Alexandru is to merge the above repository into the AtLarge repository, with the proper attribution to you. +% I will do this once the project is over. \section{Thesis Structure}\label{s:thesis-structure} The remainder of the thesis is structured as depicted in Figure \ref{fig:thesis_structure}. diff --git a/content/preamble/abstract.tex b/content/preamble/abstract.tex index 63e5e13..5f364ec 100644 --- a/content/preamble/abstract.tex +++ b/content/preamble/abstract.tex @@ -1,27 +1,15 @@ +\newpage +\begin{center} + \bfseries\Huge Abstract +\end{center} +In the modern AI economy, the strong computational demand causes datacenters to become complex, diverse facilities. +According to the Jevon's Paradox of Computer Systems, future warehouses are at an even higher risk of becoming unmanageable, due to the sheer volume of CPUs, GPUs, NPUs \etc necessary to satisfy the customers' needs. +To address this problem, the scientific community has proposed digital twinning as a datacenter management tool. +Digital Twins, which mirror complex objects and processes, provide a novel way to tackle the rising warehouse complexity. +However, datacenter digital twins are still under development, and lack crucial features, such as predictive analytics, mandatory for system administrators to make well-informed operational decisions. + +In this work, we propose to enable predictive analytics for datacenters using digital twinning. +We survey the datacenter digital twinning field, and organize our findings into a system model. +Additionally, we design \mysystem~-- a novel reference architecture for predictive datacenter digital twins, and evaluate it through prototype-based experiments. +Our results indicate \mysystem is capable of reliably differentiating between mild and severe compute failures, and can successfully incorporate a predictive analytics engine to the benefit of datacenter managers. -% Thesis Abstract ----------------------------------------------------- - - -%\begin{abstractslong} %uncommenting this line, gives a different abstract heading -\begin{abstracts} %this creates the heading for the abstract page - -\noindent \textit{Context}. -\todo{at the end} - -\noindent \textit{Goal}. -\todo{at the end} - -\noindent \textit{Method}. -\todo{at the end} - -\noindent \textit{Results}. -\todo{at the end} - -\noindent \textit{Conclusions}. -\todo{at the end} - -\end{abstracts} -%\end{abstractlongs} - - -% ---------------------------------------------------------------------- diff --git a/content/preamble/acknowledgement.tex b/content/preamble/acknowledgement.tex index 4efb21e..bbb7169 100644 --- a/content/preamble/acknowledgement.tex +++ b/content/preamble/acknowledgement.tex @@ -1,13 +1,17 @@ -% Thesis Acknowledgements ------------------------------------------------ +\newpage +\begin{center} + \bfseries \Huge Acknowledgements +\end{center} +I wish to express my thanks to the many people who have helped me in preparation of this thesis. +Dante Niewehuis, my daily supervisor, with patience and understanding mentored me during our collaboration, which led to the timely and successful completion of this thesis. -%\begin{acknowledgementslong} %uncommenting this line, gives a different acknowledgements heading -%\begin{acknowledgements} %this creates the heading for the acknowlegments +Alexandru Iosup, my 1\textsuperscript{st} supervisor, provided invaluable feedback and suggestions that helped immensely during writing. +Jesse Donkervliet guided the thesis timeline and greatly assisted in preparation of the final defence during the weekly thesis meetings. -%\end{acknowledgements} -%\end{acknowledgmentslong} - -% ------------------------------------------------------------------------ +Daniel Halasz provided many insightful comments which shaped the technical contributions. +Daniele Bonetta, Matthijs Jansen, Tiziano De Matteis, Radu Nicolae, Jure Antunovi\'c, Sacheendra Talluri, and Krijn Doekemeijer of AtLarge Research provided indispensable knowledge on conducting good Computer Systems research over the last 2 years. +Most importantly, I would like to thank Waldemar, Anna and Julia for their love and support while I was working on this thesis. diff --git a/content/preamble/dedication.tex b/content/preamble/dedication.tex index 9bc75a8..ba1667d 100644 --- a/content/preamble/dedication.tex +++ b/content/preamble/dedication.tex @@ -1,9 +1,9 @@ -% Thesis Dedication --------------------------------------------------- - -%\begin{dedication} %this creates the heading for the dedication page - -%To ... - -%\end{dedication} - -% ----------------------------------------------------------------------
\ No newline at end of file +\newpage +\begin{center} + {\bfseries \Huge Dedication} +\end{center} +\begin{center} + \vspace*{\fill} + \emph{\textbf{To my sister Julia}} + \vspace*{\fill} +\end{center} |