diff options
| -rw-r--r-- | content/implementation.tex | 28 | ||||
| -rw-r--r-- | style/style.tex | 2 |
2 files changed, 28 insertions, 2 deletions
diff --git a/content/implementation.tex b/content/implementation.tex index a7df18e..9bb87d2 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -65,15 +65,39 @@ For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a t \end{figure} \section{Data Flow}\label{ss:data_flow} -\input{sources/listing_sinks.tex} +In this section we describe the data flow within \Cref{fig:implementation} using a separate diagram. Efficient data flow is of utmost importance to digital twinning. In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}. - +In the diagram whenever we refer to \emph{control}, we mean small, one-in-a-while data packets that contain either instructions, insights or small amount of data. +Whenever we refer to \emph{data}, we mean the hundreds of thousands of metrics exported with minimal latency from the operating datacenter (\code{OpenDC}). +We describe the flow of data through a timeline. + +First, the datacenter informs the digital twin of an upcoming workload (\grayCircled{2}). +This packet contains the datacenter topology and the upcoming workload tasks (workload trace, collected from \eg BitBrains). +The digital twin stores this data locally, and passes it forward to the \code{Analytics Engine} (\grayCircled{3}). +The analytics engine queries the \code{OpenDC} +simulator to run a simulation of what might happen in the datacenter under such workload (\grayCircled{7}). +\code{OpenDC} returns the potential results to the \code{Analytics Engine} directly (\myCircled{6}). +This data for the purposes of the prototype is stored in the \code{.parquet} files. +In real-world scenario, this data flow (\myCircled{6}) would be connected to a separate \code{Kafka} topic. + +In the meantime, the datacenter executes the workload. +The datacenter continuously sends the metrics into the \code{Kafka} topic (\myCircled{1}) (\ie the datacenter is the \emph{producer}). +As this happens in real-time, \code{Redis} ingests the data from the \code{Kafka} topic (\myCircled{2}) alongside \code{PostgreSQL} (\myCircled{3}) (\ie \code{Redis} and \code{PostgreSQL} are the \emph{consumers}). +At the same time, \code{Grafana} polls the \code{PostgreSQL} database for incoming metrics (\myCircled{4}). +Each time \code{PostgreSQL} ingests the data from the \code{Kafka} topic (\myCircled{3}), at the same time \code{Grafana} updates its real-time dashboard (\grayCircled{8}). +This provides real-time feedback to datacenter operators (\grayCircled{8}). + +Simultaneously, the \code{Monitoring Service} checks the in-band data within \code{Redis}, in real-time (\grayCircled{5}). +Should anything unusual occur, the \code{Monitoring Service} notifies the \code{Analytics Engine} to perform necessary analysis (\myCircled{6}). +Then, the \code{Analytics Engine}, ingests the data from the \code{Redis} stream (\myCircled{5}) and analyzes it for further insights. +All insights generated in this way, are sent to the \code{HTTP Server} (\myCircled{4}) to communicate to the system knobs within the datacenter and to the datacenter operators (\grayCircled{1}). \begin{figure}[t] \input{sources/listing_schema.tex} \end{figure} +\input{sources/listing_sinks.tex} \section{Programming Effort}\label{ss:programming} diff --git a/style/style.tex b/style/style.tex index 915e4a4..6bd59ea 100644 --- a/style/style.tex +++ b/style/style.tex @@ -27,6 +27,8 @@ \newcommand\myCircled[1]{\scalebox{0.75}{\Circled[fill color=black,inner color=white]{{\sffamily#1}}}} +\newcommand\grayCircled[1]{\scalebox{0.75}{\Circled[fill color=gray,inner color=white]{{\sffamily#1}}}} + \newcommand{\code}[1]{\texttt{#1}} \newcommand{\BibTeX}{\textsc{B\kern-0.1emi\kern-0.017emb}\kern-0.15em\TeX\xspace} |