\chapter{Experimental Evaluation through a Real-World Prototype}\label{s:evaluation}

\section{Overview}\label{ss:evaluation_overview}
\begin{mynote}
	The contribution of this chapter is two-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}]
		\item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}.
		\item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}.
	\end{enumerate}
	Our findings indicate:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{F\textsubscript{\arabic*}}]
		\item Digital twinning can be used for failure detection to the benefit of datacenter operators.
		      \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime.
		\item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime.
		\item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated tasks.
	\end{enumerate}
\end{mynote}

\section{Experimental Setup}\label{ss:experimental_setup}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/novel_eval_method.png}
	\caption{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.}
	\label{fig:novel_eval_method}
\end{figure}


In this section we describe the technical setup used to evaluate \mysystem.
However, \Cref{fig:reference_architecture} assumes the system designer is capable of connecting the digital twin directly to the datacenter.
This raises a problem, we cannot just go and test digital twins on large systems, because we do not have large systems at hand.
Moreover, real-world experimentation is costly and unsustainable in the long run~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.
To overcome this problem, we present a novel datacenter digital twin method capable of evaluating a \gls{dcdt} without the physical datacenter.
\Cref{fig:novel_eval_method} details our approach.

In this approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}.
In our implementation, this is a second \code{OpenDC}
process (see \Cref{fig:implementation}).
The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation.
For a detailed overview of the data flow within \Cref{fig:novel_eval_method}, see \Cref{fig:flow_diagram}.

The technical setup used for all experiment adheres to the \code{OpenDC}
documentation (see Mastenbroek \etal~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}).
The workload trace used for all experiments comes from BitBrains~\cite{DBLP:conf/ccgrid/ShenBI15}.
In the experiments we model a Dutch SURF datacenter for scientific computing.
The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}.
The scheduling policy for all experiments is the \code{FilterScheduler} which considers the RAM and CPU capacity for choosing hosts to run tasks on.
This scheduling policy is also used by \code{SmartScheduler}, as outlined in \Cref{ss:programming}, albeit with modifications to enable the system knobs to take autonomous action.

In all experiments we use either \emph{failure traces} or \emph{failure models}.
For a brief explanation on the differences between the two, consult \Cref{sss:failures}.
In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
We chose a diverse range of failure models, based on the mean failure intensity in each trace (indicated in parentheses).
As a result, we chose the traces from: \begin{enumerate*}[label=(\Roman*)]
	\item Gmail (53.26\%),
	\item WhatsApp (57.97\%),
	\item YouTube (62.1\%),
	\item Twitter (65\%),
	\item Facebook (64\%).
\end{enumerate*}
In \Cref{ss:experiment2} we used a failure trace from Skype.
This is the only trace that can be paired with a corresponding failure model (\Cref{tab:failure_models_table}).
Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures.
For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5.
Importantly, in our figures we do not report the standard deviation of our experiments.
This is due to the fact that \code{OpenDC} is a fully deterministic simulator, and on each simulation run, given the same random seed will produce exactly the same results.
We believe the deviation in the results of the experiments stemming only from the random number generator is not meaningful, therefore none of the figures contain the standard deviation bars.

\section{Experiment 1: Failure Detection}\label{ss:experiment1}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/red_yellow_alarms.pdf}
	\caption{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.}
	\label{fig:red_yellow_alarms}
\end{figure}

The purpose of this experiment is two fold: \begin{enumerate*}
	\item to show our system works correctly
	\item to show our system fulfills the functional and non-functional requirements.
\end{enumerate*}
To this end, we replicate an experiment from Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
Inspired by the idea of red and yellow alarms, based on the different confidence threshold, we adapt their experiment to our system.
The experimental setup is as defined in \Cref{ss:experimental_setup}.
The experiment can be described as follows: \begin{enumerate*}[label=(\arabic*)]
	\item firstly, we  use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload.
	\item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm)
	\item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached.
	\item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold.
\end{enumerate*}
The results are in \Cref{fig:red_yellow_alarms}.
\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters.
What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems.
Importantly, the more failure-intense the trace, the more alarms are raised on behalf of the digital twin.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf}
	\caption{Comparison between the total number of raised alarms and the ground truth failure distribution during a BitBrains workload in the SURF-SARA cluster. The failure traced used in this experiment models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.}
	\label{fig:alarms_vs_failures}
\end{figure}

Additionally \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}.
In the figure we can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload.
For this visualization, we combined both the red and yellow alarms into a single metric.

However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead.
The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate.
What is surprising is Taheri \etal report almost negligible false positive rate and of their system.
Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate.
In our experiment, the numbers differ significantly.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf}
	\caption{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}).
		Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
		We believe this is due to the irreconcilable differences between our experimental setups.}
	\label{fig:failure_detecton_rate}
\end{figure}
\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%.
Compared to the DyTwin deployment, the difference is staggering.
However, the discrepancy stems from the fact in our setup we differentiate between different types of failures.
This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
As a result, we interpret \Cref{fig:failure_detecton_rate} as showing on average, 12\% of failures in the workload are severe.
Unusually, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2nd least failure-intensive trace.

\section{Experiment 2: Failure Prediction}\label{ss:experiment2}

In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics.
Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures.
In this section we try to show \mysystem can additionally work well together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter.

\begin{figure}[ht]
	\hspace{-0.8cm}
	\begin{minipage}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=1.2\linewidth]{images/failure_likelihood.pdf}
	\end{minipage}
	\hspace{1.2cm}
	\begin{minipage}[b]{0.45\textwidth}
		\centering
		\includegraphics[width=1.2\linewidth]{images/conceptual_experiment.pdf}
	\end{minipage}
	\caption{Left figure shows the potential failure distribution likelihood to approximate the true failure distribution.
		Right figure shows the results of the conceptual experiment to show the \emph{potential} gains of employing a good predictive analytics engine with \mysystem.}
	\label{fig:failure_likelihood}
\end{figure}
\begin{figure}[ht]
	\centering
	\includegraphics[width=\linewidth]{images/failure_models_table.png}
	\caption{The failure models table, by Javadi \etal~\cite{DBLP:journals/jpdc/JavadiKIE13}.}
	\label{tab:failure_models_table}
\end{figure}


\subsection{Context}\label{sss:context_experiment2}
In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods.
Our goal was to approximate the real failure distribution of a workload, using past data, and relevant statistical distributions.
For the task at hand, we chose the Skype trace, because it is supported by 4 different failure models, based on past Skype workload data.
These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}.
The Skype trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
The goal was to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines before it crashes.


Initial experiment results were unpromising.
Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly).
To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution.
Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time.
The results are in \Cref{fig:failure_likelihood}.

In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$.
Any given failure model, at any time interval is almost as likely to model the actual failures as the other models.
Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%.
This shows, the difficulty of good predictive analytics, and the correct design of a predictive analytics engine, which is not within the scope of this thesis.

Undeterred, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine.
Instead, we designed a \emph{conceptual experiment}.
In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy.
Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment.
The results are in \Cref{fig:failure_likelihood} on the right side.
\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly.

\section{Experiment 3: Additional Experiment}\label{ss:additional_experiment}

\section{Discussion}\label{ss:discussion_evaluation}