path: root/content/evaluation.tex

\chapter{Experimental Evaluation through a Real-World Prototype}\label{s:evaluation}

\section{Overview}\label{ss:evaluation_overview}
\begin{mynote}
	The contribution of this chapter is two-fold:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}]
		\item We provide a novel method for evaluating datacenter digital twins in \Cref{ss:experimental_setup}.
		\item We provide a set of exhaustive experiments to evaluate \mysystem in \Cref{ss:experiment1,ss:experiment2}.
	\end{enumerate}
	Our findings indicate:
	\vspace{-0.2cm}
	\begin{enumerate}[label=\emph{F\textsubscript{\arabic*}}]
		\item Digital twinning can be used for failure detection to the benefit of datacenter operators.
		      \emph{Sunfish} is able to effectively differentiate between large failures and insignificant downtime.
		\item \emph{Sunfish} is capable of dynamic adjustments to the scheduling policy of the datacenter, during workload runtime.
		\item If supplied with a state-of-the-art predictive analytics engine, \emph{Sunfish} is capable of lowering the number of terminated during a workload.
		\item \mysystem can estimate both the best-case and worst-case number of total tasks terminated due to failures during a datacenter workload.
	\end{enumerate}
\end{mynote}

\section{Novel Evaluation Technique}\label{ss:experimental_setup}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.74\linewidth]{images/novel_eval_method_improved.png}
	\caption[A novel evaluation method proposal.]{A novel evaluation method which solves the issue of real-world experimentation, which is unsustainable and costly~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}. In this approach, both the physical and virtual datacenter are emulated using discrete-event simulation.}
	\label{fig:novel_eval_method}
\end{figure}

In the this section, we evaluate \mysystem using \code{OpenDC} and all the necessary software (\ie \code{Grafana}, \code{PostgreSQL}, \code{Confluent Kafka}.
A na\"ive experimental approach would consist of connecting \mysystem to a physical datacenter.
However, as explained earlier in this work, running large experiments on real-world data centers is expensive and time-consuming~\cite{DBLP:conf/ccgrid/MastenbroekAJLB21}.
Additionally, we do not have access to a datacenter.
To alleviate this problem, we again utilize simulation for the benefit of evaluating \mysystem, and \gls{dcdt}s in general.

In this non-trivial approach, we replace the real-world datacenter with \emph{another} instance of the event-driven simulator from \Cref{fig:reference_architecture}.
For our implementation, we decided to use a second \code{OpenDC}
process.
A schematic overview of this experimental technique can be seen in \Cref{fig:novel_eval_method}, where the \textcolor{Red}{\ding{110} \textbf{red}} box highlights our initial approach, and the \textcolor{Green}{\ding{110} \textbf{green}} box shows the novel evaluation method.
The advantages of this approach are as follows:
\begin{enumerate*}[label=(\arabic*)]
	\item  The ``physical twin'' simulator is capable of fully replacing the real-world facility, and allows for reproducible experimentation.
	\item The environmental footprint is smaller. As a result, \mysystem is more sustainable.
	\item Datacenter engineers can fine-tune the digital twin using this setup before pairing it with the production environment.
	      In result, engineers can effectively use our setup to test and debug the digital twin before installation.
	\item With this method, \gls{dcdt} research becomes more accessible both to the scientific community and higher education.
\end{enumerate*}

\section{Experiment Parameters}\label{ss:experiment_parameters}
In this section we detail the experimental setup that follows most of the experiments.
The workload trace used for all experiments is based on a one-month SURF workload trace~\cite{DBLP:conf/ccgrid/ShenBI15}.
In the experiments we model a Dutch SURF datacenter for scientific computing.
The cluster, SURF-SARA, contains 277 hosts, each with 128GB of RAM and 16 processing cores running at maximum 2.1GHz~\cite{DBLP:conf/wosp/NicolaeTKLI26}.
This setup ensures a realistic experiment scenario that is representative of a real-world setting.

The \code{OpenDC} that models the physical datacenter uses the \code{SmartScheduler}.
The event driven simulator within the predictive analytics engine of \mysystem uses the \code{FilterScheduler} by default, unless otherwise specified.

For the experiments that model datacenter compute failures, we use \emph{failure traces} or \emph{failure models}.
For a detailed explanation on the differences between the two, consult \Cref{sss:failures}.
In short, failure traces contain the data necessary to reproduce the outages experienced by the real datacenter during the simulation.
A trace contains 3 elements, failure interval, failure duration and failure intensity (see \Cref{tab:failure_traces_overview}).
A failure model is characterized as as a statistical distribution of failures based on scientific research.
It is akin to the output of an algorithm that determines when failures should happen, rather than a real-world trace~\cite{DBLP:journals/jpdc/JavadiKIE13}.
In the experiments we use traces from the archive developed by Talluri \etal~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
We chose a diverse range of failure traces, based on the mean failure intensity in each trace (see \Cref{tab:failure_traces_overview}).
Mean failure intensity is a metric that shows best how severe the failures are.
The higher the failure intensity, the more hosts go down at once, resulting in more serious downtime and missed \gls{sla}s.
According to the mean failure intensity, we chose the 5 traces in \Cref{tab:failure_traces_overview} for our experiments.
\input{sources/failure_model_statistics.tex}
\input{sources/failure_models.tex}
\input{sources/failure_characteristics.tex}
In \Cref{ss:experiment2} we used a failure trace from Skype.
This is the only trace that can be paired with a corresponding failure model (see \Cref{tab:failure_models_table}).
Additionally, in \Cref{ss:experiment1} we find a need to define a threshold based on a statistical distribution of failures.
For this purpose, we use a normal distribution with mean 1.5 and standard deviation 1.5.

All the experiments were run on a commodity laptop, with an AMD Ryzen 7840U CPU containing 16, double-threaded cores and maximum frequency of 5.13 GHz, 32GB of DDR5 RAM.
By showing that \mysystem works using this setup, we encourage scientists and academia (stakeholder \textbf{S3}) to freely experiment with \mysystem.

\section{Experiment 1: Failure Detection}\label{ss:experiment1}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/red_yellow_alarms.pdf}
	\caption[The results of Experiment 1.]{The results of Experiment 1. \textcolor{Orange}{\ding{110} \textbf{\sffamily Red Alarms}} signify 90\% of acceptable failure threshold was reached. \textcolor{Goldenrod}{\ding{110} \textbf{\sffamily Yellow Alarms}} signify 80\% of the threshold was reached.}
	\label{fig:red_yellow_alarms}
\end{figure}
For the first experiment we copy the idea of Taheri \etal~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
We show 2 different ways a \gls{dcdt} can notify datacenter engineers of detected failures.

\subsection{Context}\label{sss:failure_detection_context}
In this section we try to provide the rationale behind this experiment.
Failure detection is a primary use-case for \gls{dcdt}s~\cite{DBLP:journals/computer/AthavaleBBMMPS24,DBLP:usdoe/report/AP26894,DBLP:conf/sc/TaheriBPRHDEWPM24,DBLP:journal/IJAE/Teugel2011}.
Failures in a datacenter can arise from a number of problems, \eg software configuration issues, power outage, network congestion~\cite{DBLP:report/AnnualOutageAnalysis2025}, and must be immediately detected to ensure their negative impact is minimized.
Some failures can have negligible consequences (around 38\% of all failures are negligible~\cite{DBLP:report/AnnualOutageAnalysis2025}), whilst other can cause severe downtime and loss of millions of \$USD~\cite{DBLP:report/AnnualOutageAnalysis2025}.
By differentiating between insignificant and severe failures, datacenter operators can focus their efforts on addressing the most impactful problems, minimizing missed \gls{sla} and ensuring swift datacenter operation.

In \Cref{ss:experiment1} we try to model such scenario.
For example, imagine the following sequence of events:
\begin{enumerate}[label=\textbf{\arabic*.},align=left]
	\item The datacenter receives a scheduled workload.
	\item To predict what kind of failures might occur during the workload we ask the digital twin to run the workload first.
	\item  We have no \emph{a priori} knowledge of the workload type, so we can only estimate the distribution of failures. To do this, the \gls{dcdt} will run the workload, assuming failures follow \eg a normal distribution.
	\item After running the simulation, the \gls{dcdt} can analyze the results and compare them with the running workload.
	\item If the running workload has more failures than what the \gls{dcdt} predicted, it notifies the datacenter operators that something is wrong.
	      Through the real-time feedback loop, the \gls{dcdt} notifies the datacenter operators that what is happening in reality is different from simulation.
\end{enumerate}

In our experiment, we notify the datacenter operators in two cases: if we get within 80\% of the predicted threshold for number of failures we send a yellow alarm.
If we get within 90\% we send a red alarm.

The purpose of this experiment is two fold: \begin{enumerate*}[label=(\arabic*)]
	\item to show our system works correctly
	\item to show our system fulfills the functional and non-functional requirements.
\end{enumerate*}
The technical experimental setup can be described as follows: \begin{enumerate*}[label=(\arabic*)]
	\item firstly, we  use \code{OpenDC} and a failure model with the normal distribution $\mathcal{N}(\mu = 1.5,\sigma=1.5)$ to model the failures we might expect from a given workload.
	\item then, using the predictions, we establish a threshold acceptable to datacenter operators (\ie how many failures can we tolerate before we raise any alarm)
	\item the red alarm is raised when 90\% of the threshold is reached, and the yellow alarm is raised when 80\% of the threshold is reached.
	\item lastly, the \code{OpenDC} acting as the real datacenter runs the workload, and \mysystem closely monitors the datacenter to see if the number of failures exceeds the accepted threshold.
\end{enumerate*}
The results are in \Cref{fig:red_yellow_alarms,fig:alarms_vs_failures,fig:failure_detecton_rate}.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/alarms_vs_failures.pdf}
	\caption[Total number of failures versus numbe rof alarms raised.]{Comparison between the total number of raised alarms and the ground truth failure distribution during a SURF month workload in the SURF-SARA cluster. The failure trace used in this plot models Gmail outage reports~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.}
	\label{fig:alarms_vs_failures}
\end{figure}


\subsection{Discussion}\label{sss:experiment1_discussion}

\Cref{fig:red_yellow_alarms} indicates \mysystem is capable of accurately detecting failures in datacenters.
In the figure we can see that as the mean failure intensity rises, so does the total number of failures detected.
This validates our design, as this behaviour is expected -- the higher the mean failure intensity, the more severe the failures, the more notifications to datacenter operators are sent.
The more failure-intense the trace, the more alarms are raised on behalf of the digital twin.
What is more, using the different threshold values, \mysystem can differentiate between serious failures and insignificant, single host problems (\ie the number of yellow alarms is smaller than the number of red alarms).
To offer a second perspective, we provide \Cref{fig:alarms_vs_failures}.
In this figure, we show the total number of alarms raised and the ground truth (how many failures occurred in reality.)
We can see a clear correlation between the total number of alarms raised, and the actual number of failures have occurred at each time during the workload.
For this visualization, we combined both the red and yellow alarms into a single metric.
In short, \Cref{fig:alarms_vs_failures} backs our claims, and verifies the results obtained in \Cref{fig:red_yellow_alarms}.

However, Taheri \etal present their results differently, using the \emph{anomaly detection rate} instead.
The rate is simply calculated as the anomalies detected correctly over the true amount of anomalies~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
Therefore, in \Cref{fig:failure_detecton_rate} we also plot the failure detection rate.
What is surprising is Taheri \etal report almost negligible false positive rate and of their system.
Moreover, they conclude through DyTwin's experimental setup, Taheri \etal achieve 100\% anomaly detection rate.
In our experiment, the numbers differ significantly.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/failure_detecton_rate.pdf}
	\caption[Failure detection rate overview.]{In this figure we show the total failure detection rate (\textcolor{Thistle}{\ding{110} \textbf{\sffamily Red + Yellow Alarms / Total Failures}}).
		Our results are much different from DyTwin's performance~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
		We believe this is due to the irreconcilable differences between our experimental setups.}
	\label{fig:failure_detecton_rate}
\end{figure}
\Cref{fig:failure_detecton_rate} shows the mean failure detection rate to be around 12\%.
Compared to the DyTwin deployment, the difference is staggering.
We stipulate the discrepancy stems from the fact in our setup we differentiate between different types of failures.
This capability is not present in the DyTwin digital twin~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.
As a result, we interpret \Cref{fig:failure_detecton_rate} as showing that on average 12\% of failures in the workload are severe.
Surprisingly, the WhatsApp failure detection rate is the lowest, contrary to the mean failure intensity, which places WhatsApp trace as the 2\textsuperscript{nd} least failure-intensive trace.

\section{Experiment 2: Failure Prediction}\label{ss:experiment2}

In \Cref{ss:experiment1} we show \mysystem is capable of incorporating descriptive analytics.
Through experiment-based evaluation, we concluded \mysystem can detect and differentiate between severe and one-off host failures.
In this section we try to show \mysystem can additionally work together with a predictive analytics engine, enabling actionable insights into the future behaviour of the datacenter.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/failure_likelihood.pdf}
	\caption[The failure model likelihood over time.]{How closely a \emph{failure model} approximates the real failure distribution (the Skype failure trace) during runtime.}.
	\label{fig:failure_likelihood}
\end{figure}

\subsection{Context}\label{sss:context_experiment2}
In order to predict when a host failure might occur, the most straightforward approach is to use long-established statistical methods.
Our goal was to approximate the real failure distribution of a workload, using past data and relevant statistical distributions.
For the task at hand, we chose the Skype Failure Trace, because it is supported by 4 different failure models, based on past Skype workload data.
These 4 statistical distribution, published in a peer-reviewed journal are in \Cref{tab:failure_models_table}~\cite{DBLP:journals/jpdc/JavadiKIE13}.
The Skype Failure Trace model was taken from the Cloud Uptime Archive~\cite{DBLP:journals/tpds/TalluriNCKCBI26}.
The goal is to use the failure distribution to predict when a host will fail, and then in advance re-schedule all the tasks from the hosts onto different machines \emph{before} it crashes.

Initial experiment results were unpromising.
Using the insights from the failure models we were not able to do better than the baseline (switching hosts on and off randomly).
To investigate why this might be the case, we run an experiment to identify which failure distribution at any given moment is most likely to resemble the actual, ground truth failure distribution.
Using a similarity score $\mathcal{S}$, which is a weighted average of the exported metrics, we tried to determine the most similar distribution at any given time.
The results are shown in \Cref{fig:failure_likelihood}.

In \Cref{fig:failure_likelihood} we can notice an almost random fluctuation of the similarity score $\mathcal{S}$.
Any given failure model, at any time interval is almost as likely to model the actual failures as the other models.
Moreover, the similarity score $\mathcal{S}$ of each failure model is never higher than 32\%.
\Cref{fig:failure_likelihood} exacerbates the difficulty of predictive analytics.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{images/conceptual_experiment.png}
	\caption[The conceptual experiment.]{The conceptual experiment. \textcolor{Apricot}{\ding{110} \textbf{Sunfish}} performance shows the benefits of digital twinning over  a normal datacenter without digital twinning insights \textcolor{Gray}{\ding{110} \textbf{OpenDC}}. Lowering the mean number of failures is of significant importance to datacenter operators.}
	\label{fig:conceptual_experiment}
\end{figure}

Despite the aforementioned results, we set out for a different solution to show \mysystem is capable of incorporating a predictive analytics engine.
Instead, we designed a \emph{conceptual experiment}.
In this setup, we \emph{assume} the predictive analytics engine is capable of fully predicting when each failure is going to happen with 100\% accuracy.
Equipped with this assumption, which only serves to show \mysystem meets the functional and non-functional requirements, we conducted the second experiment.
The results are shown in \Cref{fig:conceptual_experiment}.

\subsection{Discussion}\label{sss:experiment2_discussion}
\Cref{fig:failure_likelihood} shows that using a perfect predictive analytics engine, \mysystem is capable of lowering the total number of failures significantly.
In this experiment we used only 3 failure traces from user reports of Twitter, WhatsApp and Gmail.
Importantly, we assume in this experiment a perfect precognition of what failures might occur and when.
While unrealistic in a practical scenario, this assumption serves to approximate the potential gains from incorporating a predictive analytics engine with \mysystem.
The results are only indicative.
In the figure we can see that the mean number of failures in a ``physical datacenter'' (\ie \code{OpenDC}) is much higher without the insights of \mysystem.

\section{Experiment 3: Failure Exploration}\label{ss:experiment3}

For this experiment, we adapt the idea of multi-modal simulation from Nicolae \etal~\cite{DBLP:conf/cf/NicolaeNTI26} to estimating failures with \gls{dcdt}.
\subsection{Context}\label{sss:context_experiment3}
In this experiment we use the \gls{dcdt} to explore the number of terminated tasks when different failure injection algorithms are used.
The primary objective of this experiment is to identify the worst-case scenario and best-case scenario of running a workload prone to failures.

Nicolae \etal explore how multi-modal simulation can provide additional insight into datacenter operation and can potentially halve the error of singular models~\cite{DBLP:conf/cf/NicolaeNTI26}.
For our work, we adopt a similar approach to failure prediction.
Imagine the following sequence of events:
\begin{enumerate}[label=\textbf{\arabic*.}, align=left]
	\item The datacenter receives a scheduled workload, and the operators want to know the worst-case scenario and best-case scenario of how severe failures they can expect.
	\item Datacenter engineers ask the digital twin to run the workload first with different failure injection algorithms.
	\item After running the simulation, the \gls{dcdt} can provide the maximum and minimum number of expected failures.
	\item By continuously monitoring the datacenter during the runtime of the real workload, the operators can use the estimate to make more informative decisions.
\end{enumerate}
\begin{figure}
	\centering
	\includegraphics[width=0.85\linewidth]{images/victim_selector_comparison.pdf}
	\caption[Comparison of different victim selector algorithms.]{Comparison of different algorithms used for injecting failures into the simulated digital twin to explore the worst-case and best-case scenarios.}
	\label{fig:victim_selector_comparison}
\end{figure}
For this experiment we used a SURF month trace running on the SURF-SARA cluster, as explained in \Cref{ss:experiment_parameters}.
Additionally, we use the Gmail failure trace to inject failures, and use 3 different algorithms for selecting the hosts to fail: \code{CpuUtilSelector}, \code{RoundRobinSelector}, \code{RandomVictimSelector}.
The results are depicted in \Cref{fig:victim_selector_comparison}.

\subsection{Discussion}\label{sss:discussion_experiment3}
\Cref{fig:victim_selector_comparison} shows using different victim selection algorithms can effectively present the worst-case and best-case number of tasks terminated before running the real workload.
The victim selector that achieves the most tasks terminated is the \code{CpuUtilSelector}, which stops the hosts with the highest \gls{cpu} utilization.
It reaches over 700 terminated tasks (709) in total.
The \code{Random} selector, which randomly picks the hosts to stop, achieves the second-highest number of tasks terminated, at around 450.
The most optimistic scenario for datacenter operators is presented by the \code{RoundRobin} selector, which picks the hosts to terminate in a round-robin fashion.
No more than 130 tasks are terminated using this fault injection algorithm.
To summarize, this experiment shows that \mysystem can be used for the exploration of different failure algorithms for the benefit of datacenter operators.
Using \mysystem, on-site engineers can effectively estimate the best-case and worst-case number of terminated tasks during a workload.