diff options
Diffstat (limited to 'content/implementation.tex')
| -rw-r--r-- | content/implementation.tex | 33 |
1 files changed, 32 insertions, 1 deletions
diff --git a/content/implementation.tex b/content/implementation.tex index b289aa8..571afd3 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -120,10 +120,31 @@ Due to the massive volume of data incoming from the physical datacenter, the \co \code{Kafka} comes with excellent capability to efficiently compare data packets against a condition and filter our packets that are of no use to the \code{Analytics Engine} (see \Cref{lst:redis-sink}). On the contrary, the \code{PostgreSQL} sink (\myCircled{3}) contains all metrics collected by the datacenter sensors (see \Cref{lst:postgresql-sink}). This setup achieves excellent abstraction level, because only the most important metrics are forwarded to the \code{Analytics Engine}, with the majority of packets being filtered out. - \begin{figure}[t] \input{sources/listing_sinks.tex} \end{figure} + +\section{The \code{OpenDC} Scheduling Paradigm}\label{ss:opendc_scheduling} +In this section we introduce the scheduling paradigm of \code{OpenDC}. +\code{OpenDC}, a robust datacenter simulator uses discrete-event simulation. +``Discrete-event simulation models the operation of a system as a (discrete) sequence of events in time''~\cite{Wikipedia:article/DiscreteEventSimulation}. +Colloquially, it is akin to calling an \code{Update()} method on a set of objects to model changes in the simulator. +Scheduling in \code{OpenDC} also works using this method. +\Cref{fig:scheduling_in_opendc} represents how a task is assigned to a host in the simulation. + +A task (\myCircled{1}) is represented by its submission time, duration and computational requirements. +In order to be assigned a server to run on, it is deserialized into a \code{ServiceTask} (\myCircled{2}). +The \code{FilterScheduler} (\myCircled{4}) class takes care of scheduling the service task once the simulation reaches its submission time. +\code{OpenDC} maps tasks to available hosts via an \emph{allocation policy}~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. +In our work, this is always (also in \code{SmartScheduler}) the \code{FilterPolicy}. +In the \code{FilterPolicy}, a series of \code{HostFilter}s (\myCircled{3}) and \code{Host Weighter}s (\myCircled{5}) are used to find the matching host for the task. +In this project, the different filters and weighters take into account the default metrics (\ie \gls{cpu} capacity, \gls{ram} capacity, number of \gls{cpu} cores). +The host itself is represented as a \code{HostView} class (\myCircled{6}). +Importantly, the \code{HostView} class does not serve to simulate the behaviour of the host, but to provide a interface for the \emph{current state} of the host. +For actual computation, the \code{SimHost} (\myCircled{8}) class is used. +The \code{SimHost} object is created via the \code{HostProvisioningStep} class (\myCircled{7})~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. + + \section{Extensions to \code{OpenDC}}\label{ss:extensions} \code{OpenDC} is a state-of-the-art datacenter simulator. @@ -134,6 +155,9 @@ In order to turn it into a \gls{dt}, we have made several design decisions and e The new \code{SmartScheduler} is a scheduling mechanism capable of incorporating the insights from the \gls{dt} into its scheduling decisions. It relies on the functionality of the \code{HTTPClient} to poll the \gls{dt} at each scheduling step for potential insights. For example, if \gls{dt} sends to the datacenter a list of hosts likely to fail in the future, the \code{SmartScheduler} acts as \emph{system knobs} to enforce the \gls{dt} insights (\ie it can be mapped to \myCircled{2c} from \Cref{fig:implementation}). + It \emph{replaces} the \code{FilterScheduler} (\myCircled{4}) in \Cref{fig:scheduling_in_opendc}. + Importantly, it is functionally almost exactly the same as the \code{FilterScheduler}, with the exception that it can change the scheduling outcome based on the information from the \gls{dt}. + It is a \code{FilterScheduler} with an attached network socket. \item \textbf{\code{KafkaMonitor}}\\ The datacenter acts as the \emph{producer} of metrics, ingested by the \code{Kafka} topic (see \Cref{fig:flow_diagram}). We equip \code{OpenDC} with a new \code{ComputeMonitor} capable of exporting data directly into a \code{Kafka} topic. @@ -142,6 +166,12 @@ In order to turn it into a \gls{dt}, we have made several design decisions and e The \code{HTTPClient} offers the necessary functionality to communicate between the \gls{dt} and the datacenter. We decided to use the \gls{http} protocol for short, one-off communications between the \gls{dt} and the datacenter, as is common industry practice. \end{enumerate} +\begin{figure}[t] + \centering + \includegraphics[width=\linewidth]{images/scheduling_opendc.png} + \caption[OpenDC scheduling paradigm.]{The scheduling paradigm in \code{OpenDC}. Adapted from Musc{\u a} \etal~\cite{VUAmsterdam:thesis/AnaMariaMusca2025}. The highlighted \textcolor{Orchid}{\ding{110} \textbf{Filter Scheduler}} is the component that is substituted by the \code{SmartScheduler}.} + \label{fig:scheduling_in_opendc} +\end{figure} \section{Python Modules}\label{ss:programming} @@ -167,3 +197,4 @@ For future work, we envision a system that implements the reference architecture It contains a \code{while True} Python loop which contains the function call to fetch the latest changes to the \code{Redis} stream. Upon update, the \code{MonitoringService} informs the \code{AnalyticsEngine} that new data is awaiting \code{AnalyticsEngine} (\grayCircled{6}). \end{enumerate} + |