diff options
Diffstat (limited to 'content/implementation.tex')
| -rw-r--r-- | content/implementation.tex | 63 |
1 files changed, 36 insertions, 27 deletions
diff --git a/content/implementation.tex b/content/implementation.tex index e2fd23a..371569d 100644 --- a/content/implementation.tex +++ b/content/implementation.tex @@ -10,43 +10,52 @@ presents the rationale for using the specific software packages, \Cref{ss:data_f \section{Overview}\label{ss:implementation_overview} -At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications. -In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana}~\cite{Wikipedia:article/Grafana}. -To enable the flow of data into the digital twin, we use \code{Kafka}~\cite{Wikipedia:article/Confluent}. -To store the in-band data we use a \code{Redis}~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}~\cite{Wikipedia:article/Postgresql}. -To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}~\cite{GitHub:software/OpenDC}. -The \code{Analytics Engine}, \code{Monitoring Service}, and \code{HTTP Server} are described in detail in \Cref{ss:programming}. +\begin{figure}[t] + \centering + \includegraphics[width=0.85\linewidth]{images/implementation.pdf} + \caption{The prototype and its components based on the architecture. + The time-series data flows first to the \texttt{Grafana} dashboard, \texttt{PostgreSQL} database and \texttt{Redis} cache as advised in ~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24}.} + \label{fig:implementation} +\end{figure} -\code{Grafana} is a state-of-the-art industry tool to visualize dashboards. +At the onset of the project, we decided \gls{my_system} will use only state-of-the-art software, deployed in the industry or evaluated in peer-reviewed scientific publications. +The mapping of software packages used onto the reference architecture can be seen in \Cref{fig:implementation}. +In order to facilitate visualizations and interactive dashboards, we decided to use \code{Grafana} (\myCircled{2a})~\cite{Wikipedia:article/Grafana}. +To enable the flow of data into the digital twin, we use \code{Kafka} (\myCircled{2b})~\cite{Wikipedia:article/Confluent}. +To store the in-band data we use a \code{Redis} (\myCircled{3b})~\cite{Wikipedia:article/Redis} cache, and for out-of-band data we use a \code{PostgreSQL}(\myCircled{3a})~\cite{Wikipedia:article/Postgresql}. +To enable predictive analytics, we chose a discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}. +The \code{Analytics Engine} (\myCircled{4b}), \code{Monitoring Service} (\myCircled{4c}), and \code{HTTP Server} (\myCircled{3c}) are described in detail in \Cref{ss:programming}. + +\code{Grafana} (\myCircled{2a})is a state-of-the-art industry tool to visualize dashboards. We posit it is crucial to include a user-friendly \gls{ui}. A number of previous publications on digital twinning find dashboards important~\cite{DBLP:conf/sc/TaheriBPRHDEWPM24, DBLP:conf/wosp/SumanCNTMI24, DBLP:conf/wosp/NicolaeTKLI26}. We chose \code{Grafana} instead of other software packages because of its seamless integration with \code{PostgreSQL}. \code{Grafana} provides good separation of concerns and compartmentalization as it does not store the displayed metrics itself. -Instead, it queries the \code{PostgreSQL} database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}. +Instead, it queries the \code{PostgreSQL} (\myCircled{3a}) database in real-time~\cite{Wikipedia:article/Grafana}, unlike \eg \code{InfluxDB}. Good alternatives to \code{Grafana} are \code{Kibana}~\cite{Wikipedia:article/Kibana}, \code{Prometheus}~\cite{Wikipedia:article/Prometheus}, and \code{Graphite}~\cite{Wikipedia:article/Graphite}. -\code{Kafka}, particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time. -We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} sink, \code{Redis} sink, \code{OpenDC} source). +\code{Kafka} (\myCircled{2b}), particularly \code{Kafka} developed by Confluent~\cite{Wikipedia:article/Confluent} is a battle-tested message broker that provides versatile capabilities to transfer huge volumes of data with little latency, in real-time. +We decided to use \code{Confluent Kafka} instead of \code{Kafka} developed by the Apache Foundation, because of it's masterful connector system allowing to easily add sources and sinks (\eg \code{PostgreSQL} (\myCircled{3a}) sink, \code{Redis} (\myCircled{3b}) sink, \code{OpenDC} source (\myCircled{4a}) ). Additionally, as opposed to \code{Apache Kafka}, \code{Confluent Kafka} comes equipped with a \code{Schema Registry}. The \code{Schema Registry} is a important component that allows the storage of database and cache schemas for easy retrieval. -With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} tables and in \code{Redis} streams contains the exact same schema. +With \code{Schema Registry}, we ensure that the data stored in \code{PostgreSQL} (\myCircled{3a}) tables and in \code{Redis} (\myCircled{3b}) streams contains the exact same schema. Moreover, \code{Schema Registry} is compatible with versatile data interchange formats, such as \code{ProtoBuf}~\cite{Wikipedia:article/ProtoBuf}. -\code{Redis}, is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}. -In particular, \code{Redis} is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data. -\code{Redis} is the industry leader in key value caching. -The only alternative to \code{Redis} is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka}. +\code{Redis} (\myCircled{3b}), is a key value store that provides efficient store and retrieval operations~\cite{Wikipedia:article/Redis}. +In particular, \code{Redis} (\myCircled{3b}) is capable of storing \emph{streams} -- append only logs which allow for fast and quick query of large volumes of data. +\code{Redis} (\myCircled{3b}) is the industry leader in key value caching. +The only alternative to \code{Redis} (\myCircled{3b}) is \code{memcached}~\cite{Wikipedia:article/Memcached}, which does not provide the capability to integrate with \code{Kafka} (\myCircled{2b}). -\code{PostgreSQL} is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter. -The \code{PostgreSQL} server provides a simple and straightforward interface to query the data via \code{psql}. -Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} does not provide any \gls{ui}. -Additionally, there exist many integrations between \code{PostgreSQL} and other software, including \code{Kafka}. -The many alternatives to \code{PostgreSQL} are listed in~\cite{Wikipedia:article/Postgresql}. +\code{PostgreSQL} (\myCircled{3a})is a database management system, necessary to store large volumes of out-of-band data coming from the physical datacenter. +The \code{PostgreSQL} (\myCircled{3a}) server provides a simple and straightforward interface to query the data via \code{psql}. +Importantly, to adhere to the single responsibility principle, \code{PostgreSQL} (\myCircled{3a}) does not provide any \gls{ui}. +Additionally, there exist many integrations between \code{PostgreSQL} (\myCircled{3a}) and other software, including \code{Kafka} (\myCircled{2b}). +The many alternatives to \code{PostgreSQL} (\myCircled{3a}) are listed in~\cite{Wikipedia:article/Postgresql}. An alternative used in previous work is \code{InfluxDB}~\cite{DBLP:conf/wosp/SumanCNTMI24}. -Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}~\cite{GitHub:software/OpenDC}. -\code{OpenDC} is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning). -For a specific overview of advantages of \code{OpenDC} and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}. +Lastly, to enable predictive analytics we use a state-of-the-art discrete-event simulator, \code{OpenDC}(\myCircled{4a})~\cite{GitHub:software/OpenDC}. +\code{OpenDC} (\myCircled{4a}) is a leading software package capable of modeling complex datacenter phenomena and workloads (\eg failures, workflows, machine learning). +For a specific overview of advantages of \code{OpenDC} (\myCircled{4a}) and a thorough comparison with other alternatives, see \Cref{tab:datacenter_simulator_comparison}. \begin{figure}[t] \centering @@ -58,13 +67,13 @@ For a specific overview of advantages of \code{OpenDC} and a thorough comparison \section{Data Flow}\label{ss:data_flow} \input{sources/listing_sinks.tex} +Efficient data flow is of utmost importance to digital twinning. +In \Cref{fig:flow_diagram} we present the moving of data within \gls{my_system}. + + -\ipsum[1-4] \begin{figure}[t] \input{sources/listing_schema.tex} \end{figure} - - - \section{Programming Effort}\label{ss:programming} |