diff options
Diffstat (limited to 'content/evaluation.tex')
| -rw-r--r-- | content/evaluation.tex | 84 |
1 files changed, 41 insertions, 43 deletions
diff --git a/content/evaluation.tex b/content/evaluation.tex index b17fd9d..5e173c7 100644 --- a/content/evaluation.tex +++ b/content/evaluation.tex @@ -18,21 +18,19 @@ \end{enumerate} \end{mynote} -\section{Experimental Setup}\label{ss:experimental_setup} - +\section{Novel Evaluation Technique}\label{ss:experimental_setup} \begin{figure}[t] \centering - \includegraphics[width=0.8\linewidth]{images/novel_eval_method.png} + \includegraphics[width=0.75\linewidth]{images/novel_eval_method.png} \caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.} \label{fig:novel_eval_method} \end{figure} - -In this section we describe the technical setup used to evaluate \mysystem. -However, \Cref{fig:reference_architecture} assumes the system designer is capable of connecting the digital twin directly to the datacenter. -This raises a problem, we cannot just go and test digital twins on large systems, because we do not have large systems at hand. -Moreover, real-world experimentation is costly and unsustainable in the long run~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. -To overcome this problem, we present a novel datacenter digital twin method capable of evaluating a \gls{dcdt} without the physical datacenter. +In the this section, we evaluate \mysystem as described in the previous sections. +To do this, we run \code{OpenDC} and all the additional software (see \Cref{s:implementation}. +However, as explained earlier in this work, running large experiments on real-world data centers is expensive and time-consuming. +Additionally, we do not have access to a data center. +To alleviate this problem, we again utilize simulation for the benefit of evaluating \gls{dcdt}. \Cref{fig:novel_eval_method} details our approach. In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}. @@ -41,32 +39,28 @@ process (see \Cref{fig:implementation}). The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation. For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}. -The technical setup used for all experiment adheres to the \code{OpenDC} -documentation (see Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}). -The workload trace used for all experiments comes from BitBrains~\cite{DBLP:conf/ccgrid/ShenBI15}. + +\section{Experiment Parameters}\label{ss:experiment_parameters} +The workload trace used for all experiments is based on the BitBrains trace~\cite{DBLP:conf/ccgrid/ShenBI15}. In the experiments we model a Dutch SURF datacenter for scientific computing. The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}. -The scheduling policy for all experiments is the \code{FilterScheduler} which considers the RAM and CPU capacity for choosing hosts to run tasks on. -This scheduling policy is also used by \code{SmartScheduler}, as outlined in \Cref{ss:programming}, albeit with modifications to enable the system knobs to take autonomous action. - +The \code{OpenDC} that models the physical datacenter uses the \code{SmartScheduler}. +The event driven simulator within the predictive analytics engine of \mysystem uses the default \code{FilterScheduler}. In all experiments we use either \emph{failure traces} or \emph{failure models}. -For a brief explanation on the differences between the two, consult \Cref{sss:failures}. +For a detailed explanation on the differences between the two, consult \Cref{sss:failures}. +In short, failure traces contain the data necessary to reproduce the outages experienced by the real datacenter during the simulation. +A trace contains 3 elements, failure interval, failure duration and failure intensity (see \Cref{tab:failure_traces_overview}). +A failure model is characterized as as a statistical distribution of failures based on scientific research. +It is akin to the output of an algorithm that determines when failures should happen, rather than a real-world trace~\cite{DBLP:journals/jpdc/JavadiKIE13}. In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -We chose a diverse range of failure models, based on the mean failure intensity in each trace (indicated in parentheses). -As a result, we chose the traces from: \begin{enumerate*}[label=(\Roman*)] - \item Gmail (53.26\%), - \item WhatsApp (57.97\%), - \item YouTube (62.1\%), - \item Twitter (65\%), - \item Facebook (64\%). -\end{enumerate*} +We chose a diverse range of failure traces, based on the mean failure intensity in each trace (indicated in parentheses, see \Cref{tab:failure_traces_overview}). +\input{sources/failure_model_statistics.tex} +\input{sources/failure_models.tex} +\input{sources/failure_characteristics.tex} In \Cref{ss:experiment2} we used a failure trace from Skype. This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}). Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures. For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5. -Importantly, in our figures we do not report the standard deviation of our experiments. -This is due to the fact that \code{OpenDC} is a fully deterministic simulator, and on each simulation run, given the same random seed will produce exactly the same results. -We believe the deviation in the results of the experiments stemming only from the random number generator is not meaningful, therefore none of the figures contain the standard deviation bars. \section{Experiment 1: Failure Detection}\label{ss:experiment1} \begin{figure}[t] @@ -75,12 +69,21 @@ We believe the deviation in the results of the experiments stemming only from th \caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.} \label{fig:red_yellow_alarms} \end{figure} - -The purpose of this experiment is two fold: \begin{enumerate*} +For the first experiment we copy the idea of Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24} for different ways a \gls{dcdt} can notify the datacenter. +Imagine a scenario: a datacenter will soon run a workload. +We want to detect and differentiate between failures that are big and unexpected and failures we anticipated would occur. +To achieve this: the \gls{dcdt} runs the workload using the simulator. +We cannot know what kind of failures we can expect, so we use a statistical distribution to approximate what might occur in practice. +In result, we get a overview of what kind of failures the datacenter engineers could expect, based on statistics alone. + +We now use the real-time feedback loop to notify the datacenter operators that what is happening in reality is different from simulation. +If we get within 80\% of the predicted threshold for number of failures we send a yellow alarm. +If we get within 90\% we send a red alarm. + +The purpose of this experiment is two fold: \begin{enumerate*}[label=(\arabic*)] \item to show our system works correctly \item to show our system fulfills the functional and non-functional requirements. \end{enumerate*} -To this end, we replicate an experiment from Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system. The experimental setup is as defined in \Cref{ss:experimental_setup}. The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)] @@ -124,13 +127,13 @@ Compared to the DyTwin deployment, the difference is staggering. However, the discrepancy stems from the fact in our setup we differentiate between different types of failures. This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe. -Unusually, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace. +Surprisingly, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace. \section{Experiment 2: Failure Prediction}\label{ss:experiment2} In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics. Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures. -In this section we try to show \mysystem can additionally work well together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. +In this section we try to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. \begin{figure}[ht] \hspace{-0.8cm} @@ -147,35 +150,30 @@ In this section we try to show \mysystem can additionally work well together wit Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.} \label{fig:failure_likelihood} \end{figure} -\input{sources/failure_models.tex} \subsection{Context}\label{sss:context_experiment2} In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods. Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions. -For the task at hand, we chose the Skype trace, because it is supported by 4 different failure models, based on past Skype workload data. +For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models, based on past Skype workload data. These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}. -The Skype trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. -The goal was to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes. - +The Skype Failure Trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes. Initial experiment results were unpromising. Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly). To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution. Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time. -The results are in \Cref{fig:failure_likelihood}. +The results are shown in \Cref{fig:failure_likelihood}. In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$. Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis. -Undeterred, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. +Despite the aforementioned results, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. Instead, we designed a \emph{conceptual experiment}. In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy. Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment. -The results are in \Cref{fig:failure_likelihood} on the right side. +The results are shown in \Cref{fig:failure_likelihood} on the right side. \Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly. -\section{Experiment 3: Additional Experiment}\label{ss:additional_experiment} - -\section{Discussion}\label{ss:discussion_evaluation} |