diff options
Diffstat (limited to 'content/implementation.tex')
| -rw-r--r-- | content/implementation.tex | 26 |
1 files changed, 14 insertions, 12 deletions
diff --git a/content/implementation.tex b/content/implementation.tex index 16cf7ca..79c5191 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -3,17 +3,19 @@ \section{Overview}\label{ss:overview_implementation} \begin{mynote} - The contribution of this chapter is two-fold: + Addressing RQ$_3$ together with the next chapter, the contribution of this chapter is two-fold: \vspace{-0.2cm} \begin{enumerate}[label=\emph{C\textsubscript{\arabic*}}] \item We implement the real-world prototype of \mysystem (see \Cref{ss:detailed_implementation_overview}) realizing key features of the design. - \item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming}) + + \item We engineer \mysystem to enable predictive analytics engines (see \Cref{ss:programming}). + \end{enumerate} \end{mynote} \section{Detailed Implementation of \mysystem}\label{ss:detailed_implementation_overview} -In this section we describe the detailed implementation of \gls{my_system}. -After reading one should understand the technical decisions, choice of tools and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. +In this section, we describe the detailed implementation of \gls{my_system}. +After completing it, the reader should understand the technical decisions, choice of tools, and modifications to existing software necessary for evaluation of \gls{my_system} in \Cref{s:evaluation}. Any complex system is more than the sum of its parts~\cite{Wikipedia:article/Systems_Thinking}. To understand \gls{my_system} it is crucial to provide a holistic view on the prototype. @@ -30,7 +32,7 @@ Lastly, \Cref{ss:programming} carefully explains the design decisions behind the At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications. The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}. -In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. +To facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. To enable the flow of data into the \gls{dt}, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}. To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}. To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}. @@ -81,9 +83,9 @@ For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a t \label{fig:flow_diagram} \end{figure} -\section{Data Flow}\label{ss:data_flow} +\section{Main Data Flow}\label{ss:data_flow} -In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram. +In this section, we describe the data flow within \Cref{fig:implementation} using a separate diagram. Efficient data flow is of utmost importance to \gls{dt}s. In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}. In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data. @@ -133,7 +135,7 @@ Scheduling in \code{OpenDC} also works using this method. \Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation. A task (\myCircled{1}) is represented by its submission time, duration and computational requirements. -In order to be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). +To be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time. \code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}. @@ -148,7 +150,7 @@ The \code{SimHost} object is created via the \code{HostProvisioningStep} class ( \section{Extensions to \code{OpenDC}}\label{ss:extensions} \code{OpenDC} is a state-of-the-art datacenter simulator. -In order to turn it into a \gls{dt}, we have made several design decisions and extensions. +To turn it into a \gls{dt}, we have made several design decisions and extensions. \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{SmartScheduler}}\\ @@ -166,7 +168,7 @@ In order to turn it into a \gls{dt}, we have made several design decisions and e The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter. We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice. \item \textbf{\code{CpuUtilVictimSelector}}\\ - In order to ensure \code{OpenDC} can model different failure injection algorithms, we added new methods for selecting the hosts to be stopped. + To ensure \code{OpenDC} can model different failure injection algorithms, we added new methods for selecting the hosts to be stopped. The \code{CpuUtilVictimSelector} selects hosts based on their current \gls{cpu} utilization (\ie the hosts with the highest utilization are stopped first). \item \textbf{\code{RoundRobinVictimSelector}}\\ The \code{RoundRobinVictimSelector} is another algorithm to used to inject failures into hosts. @@ -193,12 +195,12 @@ For future work, we envision a system that implements the reference architecture \begin{enumerate}[label=\textbf{\arabic*.}] \item \textbf{\code{AnalyticsEngine}}\\ - The \code{AnalyticsEngine} module is necessary in order to encapsulate the logic of data preprocessing and analysis from monitoring. + The \code{AnalyticsEngine} module is necessary to encapsulate the logic of data preprocessing and analysis from monitoring. This component can contain capabilities for different statistical metrics, subject to \gls{dt}s focus. In \gls{my_system} \code{AnalyticsEngine} continuously checks whether the incoming datacenter sensor readings exceed different thresholds. For example, the \code{AnalyticsEngine} is capable of calculating a similarity score $S$ between potential failure distributions and the true failure distribution. \item \textbf{\code{HTTPServer}}\\ - The \code{HTTPServer} is crucial for interrupting the operation of the datacenter in order to adjust its operation or offer insights. + The \code{HTTPServer} is crucial for interrupting the operation of the datacenter to adjust its operation or offer insights. It maintains a python \code{Queue} structure. The \code{Queue} \emph{producer} is the \code{AnalyticsEngine} (\grayCircled{4}). The \emph{consumer} is the \code{HTTPClient} within \code{OpenDC} (\ie the real datacenter, (\myCircled{2}), (\myCircled{1})). |