\chapter{Experimental Evaluation through a Real-World Prototype}\label{s:evaluation}

\section{Overview}\label{ss:evaluation_overview}
\begin{mynote}
	The contribution of this chapter is two-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}]
		\item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}.
		\item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}.
	\end{enumerate}
	Our findings indicate:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{F\textsubscript{\arabic*}}]
		\item Digital twinning can be used for failure detection to the benefit of datacenter operators.
		      \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime.
		\item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime.
		\item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated tasks.
	\end{enumerate}
\end{mynote}

\section{Novel Evaluation Technique}\label{ss:experimental_setup}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.75\linewidth]{images/novel_eval_method.png}
	\caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.}
	\label{fig:novel_eval_method}
\end{figure}

In the this section, we evaluate \mysystem as described in the previous sections.
To do this, we run \code{OpenDC} and all the additional software (see \Cref{s:implementation}.
However, as explained earlier in this work, running large experiments on real-world data centers is expensive and time-consuming.
Additionally, we do not have access to a data center.
To alleviate this problem, we again utilize simulation for the benefit of evaluating \gls{dcdt}.
\Cref{fig:novel_eval_method} details our approach.

In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}.
In our implementation, this is a second \code{OpenDC}
process (see \Cref{fig:implementation}).
The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation.
For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}.


\section{Experiment Parameters}\label{ss:experiment_parameters}
The workload trace used for all experiments is based on the BitBrains trace~\cite{DBLP:conf/ccgrid/ShenBI15}.
In the experiments we model a Dutch SURF datacenter for scientific computing.
The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}.
The \code{OpenDC} that models the physical datacenter uses the \code{SmartScheduler}.
The event driven simulator within the predictive analytics engine of \mysystem uses the default \code{FilterScheduler}.
In all experiments we use either \emph{failure traces} or \emph{failure models}.
For a detailed explanation on the differences between the two, consult \Cref{sss:failures}.
In short, failure traces contain the data necessary to reproduce the outages experienced by the real datacenter during the simulation.
A trace contains 3 elements, failure interval, failure duration and failure intensity (see \Cref{tab:failure_traces_overview}).
A failure model is characterized as as a statistical distribution of failures based on scientific research.
It is akin to the output of an algorithm that determines when failures should happen, rather than a real-world trace~\cite{DBLP:journals/jpdc/JavadiKIE13}.
In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
We chose a diverse range of failure traces, based on the mean failure intensity in each trace (indicated in parentheses, see \Cref{tab:failure_traces_overview}).
\input{sources/failure_model_statistics.tex}
\input{sources/failure_models.tex}
\input{sources/failure_characteristics.tex}
In \Cref{ss:experiment2} we used a failure trace from Skype.
This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}).
Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures.
For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5.

\section{Experiment 1: Failure Detection}\label{ss:experiment1}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/red_yellow_alarms.pdf}
	\caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.}
	\label{fig:red_yellow_alarms}
\end{figure}
For the first experiment we copy the idea of Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24} for different ways a \gls{dcdt} can notify the datacenter.
Imagine a scenario: a datacenter will soon run a workload.
We want to detect and differentiate between failures that are big and unexpected and failures we anticipated would occur.
To achieve this: the \gls{dcdt} runs the workload using the simulator.
We cannot know what kind of failures we can expect, so we use a statistical distribution to approximate what might occur in practice.
In result, we get a overview of what kind of failures the datacenter engineers could expect, based on statistics alone.

We now use the real-time feedback loop to notify the datacenter operators that what is happening in reality is different from simulation.
If we get within 80\% of the predicted threshold for number of failures we send a yellow alarm.
If we get within 90\% we send a red alarm.

The purpose of this experiment is two fold: \begin{enumerate*}[label=(\arabic*)]
	\item to show our system works correctly
	\item to show our system fulfills the functional and non-functional requirements.
\end{enumerate*}
Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system.
The experimental setup is as defined in \Cref{ss:experimental_setup}.
The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)]
	\item firstly, we  use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload.
	\item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm)
	\item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached.
	\item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold.
\end{enumerate*}
The results are in \Cref{fig:red_yellow_alarms}.
\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters.
What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems.
Importantly, the more failure-intense the trace, the more alarms are raised on behalf of the digital twin.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf}
	\caption[Total number of failures versus numbe rof alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a BitBrains workload in the SURF-SARA cluster. The failure traced used in this experiment models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.}
	\label{fig:alarms_vs_failures}
\end{figure}

Additionally \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}.
In the figure we can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload.
For this visualization, we combined both the red and yellow alarms into a single metric.

However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead.
The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate.
What is surprising is Taheri \etal report almost negligible false positive rate and of their system.
Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate.
In our experiment, the numbers differ significantly.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf}
	\caption[Failure detection rate overview.]{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}).
		Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
		We believe this is due to the irreconcilable differences between our experimental setups.}
	\label{fig:failure_detecton_rate}
\end{figure}
\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%.
Compared to the DyTwin deployment, the difference is staggering.
However, the discrepancy stems from the fact in our setup we differentiate between different types of failures.
This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe.
Surprisingly, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace.

\section{Experiment 2: Failure Prediction}\label{ss:experiment2}

In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics.
Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures.
In this section we try to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter.

\begin{figure}[ht]
	\hspace{-0.8cm}
	\begin{minipage}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=1.2\linewidth]{images/failure_likelihood.pdf}
	\end{minipage}
	\hspace{1.2cm}
	\begin{minipage}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=1.2\linewidth]{images/conceptual_experiment.pdf}
	\end{minipage}
	\caption[The results of Experiment 2.]{Left figure shows the potential failure distribution likelihood to approximate the true failure distribution.
		Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.}
	\label{fig:failure_likelihood}
\end{figure}

\subsection{Context}\label{sss:context_experiment2}
In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods.
Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions.
For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models, based on past Skype workload data.
These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}.
The Skype Failure Trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes.

Initial experiment results were unpromising.
Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly).
To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution.
Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time.
The results are shown in \Cref{fig:failure_likelihood}.

In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$.
Any given failure model, at any time interval is almost as likely to model the actual failures as the other models.
Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%.
This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis.

Despite the aforementioned results, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine.
Instead, we designed a \emph{conceptual experiment}.
In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy.
Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment.
The results are shown in \Cref{fig:failure_likelihood} on the right side.
\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly.

