diff options
Diffstat (limited to 'content/evaluation.tex')
| -rw-r--r-- | content/evaluation.tex | 140 |
1 files changed, 85 insertions, 55 deletions
diff --git a/content/evaluation.tex b/content/evaluation.tex index a0a6617..87a9306 100644 --- a/content/evaluation.tex +++ b/content/evaluation.tex @@ -2,11 +2,13 @@ \section{Overview}\label{ss:evaluation_overview} \begin{mynote} - The contribution of this chapter is two-fold: + Addressing RQ$_3$ together with the previous chapter, the contribution of this chapter is two-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] \item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}. - \item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}. + + \item We provide a comprehensive set of experiments to evaluate \mysystem~(\Cref{ss:experiment1,ss:experiment2}). + \end{enumerate} Our findings indicate: \vspace{-0.2cm} @@ -85,7 +87,7 @@ By showing that \mysystem works using this setup, we encourage scientists and ac \caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.} \label{fig:red_yellow_alarms} \end{figure} -For the first experiment we copy the idea of Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +For the first experiment we adopt the idea introduced by Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. We show 2 different ways a \gls{dcdt} can notify datacenter engineers of detected failures. \subsection{Context}\label{sss:failure_detection_context} @@ -95,22 +97,24 @@ Failures in a datacenter can arise from a number of problems, \eg software confi Some failures can have negligible consequences (around 38\% of all failures are negligible~\cite{DBLP:report/AnnualOutageAnalysis2025}), whilst other can cause severe downtime and loss of millions of \$USD~\cite{DBLP:report/AnnualOutageAnalysis2025}. By differentiating between insignificant and severe failures, datacenter operators can focus their efforts on addressing the most impactful problems, minimizing missed \gls{sla} and ensuring swift datacenter operation. -In \Cref{ss:experiment1} we try to model such scenario. +In \Cref{ss:experiment1} we model such a scenario. For example, imagine the following sequence of events: \begin{enumerate}[label=\textbf{\arabic*.},align=left] \item The datacenter receives a scheduled workload. - \item To predict what kind of failures might occur during the workload we ask the digital twin to run the workload first. - \item We have no \emph{a priori} knowledge of the workload type, so we can only estimate the distribution of failures. To do this, the \gls{dcdt} will run the workload, assuming failures follow \eg a normal distribution. + \item To predict what kind of failures might occur during the workload, we ask the digital twin to run the workload first. + \item We have no \emph{a priori} knowledge of the workload type, so we can only estimate the distribution of failures. To do this, the \gls{dcdt} will run the workload, assuming failures follow \eg a normal distribution. (Unlike traditional analytical tools, the distribution can be selected arbitrarily, even changed during the run, and the predictive results do not rely on underlying statistical assumptions.) \item After running the simulation, the \gls{dcdt} can analyze the results and compare them with the running workload. - \item If the running workload has more failures than what the \gls{dcdt} predicted, it notifies the datacenter operators that something is wrong. - Through the real-time feedback loop, the \gls{dcdt} notifies the datacenter operators that what is happening in reality is different from simulation. + \item If the running workload has more failures than what the \gls{dcdt} predicted, with the latter representing the known errors inherent in typical datacenter operations, the discrepancy is used to notify the datacenter operators -- + something is wrong. + Through the real-time feedback loop, the \gls{dcdt} not only notifies the datacenter operators, but can also help understand where and how what is happening in reality is different from the simulation. \end{enumerate} -In our experiment, we notify the datacenter operators in two cases: if we get within 80\% of the predicted threshold for number of failures we send a yellow alarm. -If we get within 90\% we send a red alarm. +In our experiment, we notify the datacenter operators in two cases: +(1) we send a yellow alarm if the 80\% threshold is reached, that is, the simulated results get within 80\% of the predicted threshold for the number of failures; +(2) we send a red alarm if the 90\% threshold is reached. -The purpose of this experiment is two fold: \begin{enumerate*}[label=(\arabic*)] - \item to show our system works correctly +The purpose of this experiment is two-fold: \begin{enumerate*}[label=(\arabic*)] + \item to show our system works correctly, and \item to show our system fulfills the functional and non-functional requirements. \end{enumerate*} The technical experimental setup can be described as follows: \begin{enumerate*}[label=(\arabic*)] @@ -132,56 +136,64 @@ The results are in \Cref{fig:red_yellow_alarms,fig:alarms_vs_failures,fig:failur \subsection{Discussion}\label{sss:experiment1_discussion} \Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters. -In the figure we can see that as the mean failure intensity rises, so does the total number of failures detected. -This validates our design, as this behaviour is expected -- the higher the mean failure intensity, the more severe the failures, the more notifications to datacenter operators are sent. +In the figure, we can see that as the mean failure intensity rises, so does the total number of failures detected. +This behaviour is expected -- the higher the mean failure intensity, the more severe the failures, the more notifications to datacenter operators are sent. The more failure-intense the trace, the more alarms are raised on behalf of the digital twin. What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems (\ie the number of yellow alarms is smaller than the number of red alarms). -To offer a second perspective, we provide \Cref{fig:alarms_vs_failures}. + + +To offer a second perspective, we also provide the results depicted in \Cref{fig:alarms_vs_failures}. In this figure, we show the total number of alarms raised and the ground truth (how many failures occurred in reality.) We can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload. For this visualization, we combined both the red and yellow alarms into a single metric. In short, \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}. -However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. -The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate. -What is surprising is Taheri \etal report almost negligible false positive rate and of their system. -Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate. -In our experiment, the numbers differ significantly. -\begin{figure}[t] +\begin{figure}[!t] \centering \includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf} - \caption[Failure detection rate overview.]{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}). - Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. - We believe this is due to the irreconcilable differences between our experimental setups.} + \caption[Failure detection ratio.]{Total failure detection ratio (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}). + } \label{fig:failure_detecton_rate} \end{figure} -\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%. -Compared to the DyTwin deployment, the difference is staggering. -We stipulate the discrepancy stems from the fact in our setup we differentiate between different types of failures. -This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. -As a result, we interpret \Cref{fig:failure_detecton_rate} as showing that on average 12\% of failures in the workload are severe. -Surprisingly, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2\textsuperscript{nd} least failure-intensive trace. + +However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead. +The rate is simply calculated as the anomalies detected correctly over the true anomaly count~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. (Note: The correct term for this should be the \emph{anomaly detection ratio}.) +Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection ratio, defined as the sum of Red and Yellow alarm-counts, over the Total Failure count. +What is surprising is Taheri \etal report almost negligible false positive rate and of their system. +Moreover, after testing with DyTwin in a specific experimental setup, Taheri \etal conclude their proposed method achieves 100\% anomaly detection rate. +In our experiment, the numbers differ significantly; we attribute this to irreconcilable differences between our experimental setups. + + +% Our results are also different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +% We believe this is due to the irreconcilable differences between our experimental setups. + + + +\Cref{fig:failure_detecton_rate} shows the mean failure detection ratio to be around 12\%. +Compared to the DyTwin results reported by Taheri~\etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}, the difference is significant. +We stipulate that the root of the discrepancy is that our setup differentiates between different types of failures, a capability that is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}. +As a new result, we interpret \Cref{fig:failure_detecton_rate} as showing that on average 12\% of failures in the workload are severe. +Surprisingly, the WhatsApp failure detection ratio is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2\textsuperscript{nd} least failure-intensive trace. \section{Experiment 2: Failure Prediction}\label{ss:experiment2} -In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics. +In \Cref{ss:experiment1} we showed \mysystem is capable of incorporating descriptive analytics. Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures. -In this section we try to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. +In this section, we aim to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter. \begin{figure}[t] \centering \includegraphics[width=0.8\linewidth]{images/failure_likelihood.png} - \caption[The failure model likelihood over time.]{How closely a \emph{failure model} approximates the real failure distribution (the Skype failure trace) during runtime.}. + \caption[The failure model likelihood over time.]{How closely a \emph{failure model} approximates the real failure distribution (the Skype failure trace) during runtime. The vertical axis is linear but does \textit{not} start at 0.} \label{fig:failure_likelihood} \end{figure} \subsection{Context}\label{sss:context_experiment2} -In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods. -Our goal was to approximate the real failure distribution of a workload, using past data and relevant statistical distributions. -For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models, based on past Skype workload data. -These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}. -The Skype Failure Trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. +To predict when a host failure might occur, a straightforward approach is to use long-established statistical methods. +Our goal is to approximate the real failure distribution of a workload, using past data and relevant statistical distributions. +For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models~(see \Cref{tab:failure_models_table}, sourced from~\cite{DBLP:journals/jpdc/JavadiKIE13}), based on past Skype workload data. +%These 4 statistical distributions, published in a peer-reviewed journal are in . +The Skype Failure Trace model was sourced from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}. The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines \emph{before} it crashes. Initial experiment results were unpromising. @@ -190,27 +202,42 @@ To investigate why this might be the case, we run an experiment to identify whic Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time. The results are shown in \Cref{fig:failure_likelihood}. -In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$. -Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. -Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. -\Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. +% In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +% Any given failure model, at any time interval is almost as likely to model the actual failures as the other models. +% Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +% \Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. -\begin{figure}[t] - \centering - \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} - \caption[The conceptual experiment.]{The conceptual experiment. \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} - \label{fig:conceptual_experiment} -\end{figure} +% \begin{figure}[!t] +% \centering +% \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} +% \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} +% \label{fig:conceptual_experiment} +% \end{figure} -Despite the aforementioned results, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine. +%Despite the aforementioned results, we set out for a different solution to show +We also show \mysystem is capable of incorporating a predictive analytics engine. Instead, we designed a \emph{conceptual experiment}. In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy. Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment. The results are shown in \Cref{fig:conceptual_experiment}. \subsection{Discussion}\label{sss:experiment2_discussion} -\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly. -In this experiment we used only 3 failure traces from user reports of Twitter, WhatsApp and Gmail. + +In \Cref{fig:failure_likelihood}, we notice an almost random fluctuation of the similarity score $\mathcal{S}$. +For any failure model, at any time interval, it is almost as likely to accurately model the actual failures as the other models. +Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%. +\Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics. + + +\begin{figure}[!t] + \centering + \includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png} + \caption[What-if experiment.]{The What-if experiment: \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.} + \label{fig:conceptual_experiment} +\end{figure} + +\Cref{fig:conceptual_experiment} shows that using a perfect predictive-analytics engine, \mysystem is capable of lowering the total number of failures significantly. +In this experiment, we used only 3 failure traces from user reports of Twitter, WhatsApp, and Gmail. Importantly, we assume in this experiment a perfect precognition of what failures might occur and when. While unrealistic in a practical scenario, this assumption serves to approximate the potential gains from incorporating a predictive analytics engine with \mysystem. The results are only indicative. @@ -218,7 +245,8 @@ In the figure we can see that the mean number of failures in a ``physical datace \section{Experiment 3: Failure Exploration}\label{ss:experiment3} -For this experiment, we adapt the idea of multi-modal simulation from Nicolae \etal~\cite{DBLP:conf/cf/NicolaeNTI26} to estimating failures with \gls{dcdt}. +For this experiment, we adapt the idea of using multi-modal simulation from Nicolae \etal~\cite{DBLP:conf/cf/NicolaeNTI26} to estimate failures with \gls{dcdt}. + \subsection{Context}\label{sss:context_experiment3} In this experiment we use the \gls{dcdt} to explore the number of terminated tasks when different failure injection algorithms are used. The primary objective of this experiment is to identify the worst-case scenario and best-case scenario of running a workload prone to failures. @@ -247,7 +275,9 @@ The results are depicted in \Cref{fig:victim_selector_comparison}. The victim selector that achieves the most tasks terminated is the \code{CpuUtilSelector}, which stops the hosts with the highest \gls{cpu} utilization. It reaches over 700 terminated tasks (709) in total. The \code{Random} selector, which randomly picks the hosts to stop, achieves the second-highest number of tasks terminated, at around 450. -The most optimistic scenario for datacenter operators is presented by the \code{RoundRobin} selector, which picks the hosts to terminate in a round-robin fashion. + +The most optimistic scenario for datacenter operators tested here is the \code{RoundRobin} selector, which picks the hosts to terminate in a round-robin fashion. No more than 130 tasks are terminated using this fault injection algorithm. -To summarize, this experiment shows that \mysystem can be used for the exploration of different failure algorithms for the benefit of datacenter operators. + +To summarize, this experiment shows that \mysystem can be used to explore different failure algorithms for the benefit of datacenter operators. Using \mysystem, on-site engineers can effectively estimate the best-case and worst-case number of terminated tasks during a workload. |