From b593268b50784964c672f8b5aaa857e9b9243634 Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 20 Dec 2019 11:26:33 +0100 Subject: Adapt documentation outline and README --- docs/architecture.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 docs/architecture.md (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..d26a8749 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,4 @@ +# 2. Architecture Overview + +--- +[< Previous](setup.md) | [Next >](models.md) -- cgit v1.2.3 From 773eaf770313fa245250e8e1661ee50444223c98 Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 20 Dec 2019 11:37:25 +0100 Subject: Create guidelines for documentation sections --- docs/architecture.md | 4 ++++ 1 file changed, 4 insertions(+) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index d26a8749..a854bf6d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,4 +1,8 @@ # 2. Architecture Overview +**TODO: What are the main components (odcsim-core, odcsim-engine, opendc-core, opendc-format, opendc-testkit, opendc-workflows, I assume)? Short description of each in a bullet list (or even a schema drawing, if you have the time). Why this division?** + +**TODO: One section per component, explaining what their responsibility is and what other modules they talk to. In what kind of scenarios do you as OpenDC developer need to touch/change them?** + --- [< Previous](setup.md) | [Next >](models.md) -- cgit v1.2.3 From f33cfe79358c31cc786fdb1594cb1c40ab8e0a7b Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 14 Feb 2020 11:37:32 +0100 Subject: Get first two parts of docs up to date and remove odcsim doc --- docs/architecture.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index a854bf6d..60ac3e0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,8 +1,17 @@ -# 2. Architecture Overview +# 3. Architecture Overview + +The OpenDC simulator has two main components: `odcsim`, a general framework for discrete event-based simulation, and `opendc` a collection of components using `odcsim` to simulate datacenters. We discuss both in turn! + +`TODO: Add a schematic general overview here.` + +## 3.1 `odcsim` + + +## 3.2 `opendc` **TODO: What are the main components (odcsim-core, odcsim-engine, opendc-core, opendc-format, opendc-testkit, opendc-workflows, I assume)? Short description of each in a bullet list (or even a schema drawing, if you have the time). Why this division?** **TODO: One section per component, explaining what their responsibility is and what other modules they talk to. In what kind of scenarios do you as OpenDC developer need to touch/change them?** --- -[< Previous](setup.md) | [Next >](models.md) +[< Previous](setup.md) | [Next >](run.md) -- cgit v1.2.3 From 4ad068f741321973a70f10755074505a3f813ebd Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 14 Feb 2020 12:28:20 +0100 Subject: Add documentation for the odcsim component --- docs/architecture.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index 60ac3e0d..834de69c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,13 +1,23 @@ # 3. Architecture Overview - -The OpenDC simulator has two main components: `odcsim`, a general framework for discrete event-based simulation, and `opendc` a collection of components using `odcsim` to simulate datacenters. We discuss both in turn! +The OpenDC simulator has two main components: `odcsim`, a general framework for discrete event-based simulation, and `opendc` a collection of components using `odcsim` to simulate datacenters. While `opendc` is the focus of this project and thus these docs, we also shortly explain what `odcsim` looks like, to give you a better understanding of the other concepts. `TODO: Add a schematic general overview here.` ## 3.1 `odcsim` +The `odcsim` framework has an API module (`odcsim-api`) and an module implementing this API (`odcsim-engine-omega`). + +### 3.1.1 `odcsim-api` +The API defines the behavior of any implementation adhering to the `odcsim` framework. It centers around a `SimulationEngine`, which can be created through a `SimulationEngineProvider`. + +A simulation has a root process with certain `Behavior` (a coroutine). Processes have a `ProcessContext` which allow them to spawn other processes and open communication `Channel`s with other processes. Each of these `Channel`s has a `SendPort` and a `ReceivePort`. + +### 3.1.2 `odcsim-engine-omega` +The implementation is an executable interpretation of this API. The main class is `OmegaSimulationEngine` and takes care of transmitting timestamped events between processes. It ensures that message delivery order is the same as sent order. +You might note that the simulation framework and the simulator itself makes extensive use of Kotlin's coroutine feature. This is a paradigm for asynchronous programming. If you are not familiar with coroutines, we suggest having a look at the [Kotlin documentation on this](https://kotlinlang.org/docs/reference/coroutines-overview.html), especially their [quick start guide](https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html) and their [hands-on guide](https://play.kotlinlang.org/hands-on/Introduction%20to%20Coroutines%20and%20Channels/01_Introduction). ## 3.2 `opendc` +The `opendc` package consists of **TODO: What are the main components (odcsim-core, odcsim-engine, opendc-core, opendc-format, opendc-testkit, opendc-workflows, I assume)? Short description of each in a bullet list (or even a schema drawing, if you have the time). Why this division?** -- cgit v1.2.3 From 19997aaef104e63f6cf7dc3f831339e5fa5b933d Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 14 Feb 2020 19:13:54 +0100 Subject: Add texts around non-compute modules --- docs/architecture.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index 834de69c..dcb24c88 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,7 +1,7 @@ # 3. Architecture Overview The OpenDC simulator has two main components: `odcsim`, a general framework for discrete event-based simulation, and `opendc` a collection of components using `odcsim` to simulate datacenters. While `opendc` is the focus of this project and thus these docs, we also shortly explain what `odcsim` looks like, to give you a better understanding of the other concepts. -`TODO: Add a schematic general overview here.` +**TODO: Add a schematic general overview here.** ## 3.1 `odcsim` The `odcsim` framework has an API module (`odcsim-api`) and an module implementing this API (`odcsim-engine-omega`). @@ -17,11 +17,21 @@ The implementation is an executable interpretation of this API. The main class i You might note that the simulation framework and the simulator itself makes extensive use of Kotlin's coroutine feature. This is a paradigm for asynchronous programming. If you are not familiar with coroutines, we suggest having a look at the [Kotlin documentation on this](https://kotlinlang.org/docs/reference/coroutines-overview.html), especially their [quick start guide](https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html) and their [hands-on guide](https://play.kotlinlang.org/hands-on/Introduction%20to%20Coroutines%20and%20Channels/01_Introduction). ## 3.2 `opendc` -The `opendc` package consists of +The `opendc` package consists of a number of submodules, the most important being `opendc-compute`. Below, we will explain each in turn. -**TODO: What are the main components (odcsim-core, odcsim-engine, opendc-core, opendc-format, opendc-testkit, opendc-workflows, I assume)? Short description of each in a bullet list (or even a schema drawing, if you have the time). Why this division?** +### 3.2.1 `opendc-compute` +**TODO** -**TODO: One section per component, explaining what their responsibility is and what other modules they talk to. In what kind of scenarios do you as OpenDC developer need to touch/change them?** +### 3.2.2 `opendc-workflows` +This module contains all workflow-related models and logic of the simulator. The models for workflows can be found in the `workload` package: A `Job` and a `Task`. The logic concerning the scheduling of a workflow is contained in the `service` package. It follows the Reference Architecture for Datacenter Schedulers by [Andreadis et al.](https://dl.acm.org/doi/10.5555/3291656.3291706). For a good introduction into datacenter schedulers and to fully grasp the modeling approach taken, we highly recommend reading this publication (or its more extensive [technical report](https://arxiv.org/pdf/1808.04224.pdf)). + +The `service` package merits its own explanation. A scheduler is defined by the `StageWorkflowScheduler` class, according to the architecture. The main component, however, is the `StageWorkflowSchedulerLogic`, responsible for pulling together the different scheduling stage implementations from the `stage` package. The scheduler is managed by the `WorkflowService`, which also orchestrates the lifecycle of a workflow. + +### 3.2.3 `opendc-format` +Running scientific experiments does not require running the full OpenDC stack. We also support directly reading out environment and workload trace files. Example implementations of these can be found in the `opendc-format` module. To parse a different format, you can take one of the existing parsers and adapt it to your needs. + +### 3.2.4 `opendc-experiments-sc18` +This is a module created for the experiments of our [SC18 publication](https://dl.acm.org/doi/10.5555/3291656.3291706). We aim to separate these kinds of custom experiment setups from the rest of the codebase by placing them in separate modules. --- [< Previous](setup.md) | [Next >](run.md) -- cgit v1.2.3 From 97afd8f9c77a5800bed11ddab1f51e7dbfc3128f Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Fri, 14 Feb 2020 19:19:18 +0100 Subject: Add core module to list --- docs/architecture.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index dcb24c88..95d72126 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -17,20 +17,23 @@ The implementation is an executable interpretation of this API. The main class i You might note that the simulation framework and the simulator itself makes extensive use of Kotlin's coroutine feature. This is a paradigm for asynchronous programming. If you are not familiar with coroutines, we suggest having a look at the [Kotlin documentation on this](https://kotlinlang.org/docs/reference/coroutines-overview.html), especially their [quick start guide](https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html) and their [hands-on guide](https://play.kotlinlang.org/hands-on/Introduction%20to%20Coroutines%20and%20Channels/01_Introduction). ## 3.2 `opendc` -The `opendc` package consists of a number of submodules, the most important being `opendc-compute`. Below, we will explain each in turn. +The `opendc` package consists of a number of submodules, the most feature-rich being `opendc-compute`. Below, we will explain each in turn. -### 3.2.1 `opendc-compute` +### 3.2.1 `opendc-core` **TODO** -### 3.2.2 `opendc-workflows` +### 3.2.2 `opendc-compute` +**TODO** + +### 3.2.3 `opendc-workflows` This module contains all workflow-related models and logic of the simulator. The models for workflows can be found in the `workload` package: A `Job` and a `Task`. The logic concerning the scheduling of a workflow is contained in the `service` package. It follows the Reference Architecture for Datacenter Schedulers by [Andreadis et al.](https://dl.acm.org/doi/10.5555/3291656.3291706). For a good introduction into datacenter schedulers and to fully grasp the modeling approach taken, we highly recommend reading this publication (or its more extensive [technical report](https://arxiv.org/pdf/1808.04224.pdf)). The `service` package merits its own explanation. A scheduler is defined by the `StageWorkflowScheduler` class, according to the architecture. The main component, however, is the `StageWorkflowSchedulerLogic`, responsible for pulling together the different scheduling stage implementations from the `stage` package. The scheduler is managed by the `WorkflowService`, which also orchestrates the lifecycle of a workflow. -### 3.2.3 `opendc-format` +### 3.2.4 `opendc-format` Running scientific experiments does not require running the full OpenDC stack. We also support directly reading out environment and workload trace files. Example implementations of these can be found in the `opendc-format` module. To parse a different format, you can take one of the existing parsers and adapt it to your needs. -### 3.2.4 `opendc-experiments-sc18` +### 3.2.5 `opendc-experiments-sc18` This is a module created for the experiments of our [SC18 publication](https://dl.acm.org/doi/10.5555/3291656.3291706). We aim to separate these kinds of custom experiment setups from the rest of the codebase by placing them in separate modules. --- -- cgit v1.2.3 From 24d42cabd6c4a40313c0de0a53d2cec6aa5b0a5e Mon Sep 17 00:00:00 2001 From: Fabian Mastenbroek Date: Fri, 14 Feb 2020 22:54:10 +0100 Subject: docs: Add initial description of opendc-core and opendc-compute modules --- docs/architecture.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index 95d72126..4f271360 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -20,10 +20,13 @@ You might note that the simulation framework and the simulator itself makes exte The `opendc` package consists of a number of submodules, the most feature-rich being `opendc-compute`. Below, we will explain each in turn. ### 3.2.1 `opendc-core` -**TODO** +This module defines a base model for datacenter simulation, establishing core concepts and terminology of datacenters. +The other `opendc` modules build on this model and extend it in various directions (e.g. virtual machines or workflows). ### 3.2.2 `opendc-compute` -**TODO** +This module models management and provisioning of compute instances such as virtual machines and bare-metal servers. We +represent such compute instances as a `Server`. The hardware configuration of the server is represented as a `Flavor`. +Servers run bootable disk images called `Image`s, which characterizes the runtime behavior of a server. ### 3.2.3 `opendc-workflows` This module contains all workflow-related models and logic of the simulator. The models for workflows can be found in the `workload` package: A `Job` and a `Task`. The logic concerning the scheduling of a workflow is contained in the `service` package. It follows the Reference Architecture for Datacenter Schedulers by [Andreadis et al.](https://dl.acm.org/doi/10.5555/3291656.3291706). For a good introduction into datacenter schedulers and to fully grasp the modeling approach taken, we highly recommend reading this publication (or its more extensive [technical report](https://arxiv.org/pdf/1808.04224.pdf)). -- cgit v1.2.3 From 976ea9e7a6071aa75e32a61bf2b150a42f16224b Mon Sep 17 00:00:00 2001 From: Fabian Mastenbroek Date: Tue, 18 Feb 2020 21:11:22 +0100 Subject: docs: Add extended description of compute module --- docs/architecture.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index 4f271360..33fb90e5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -20,13 +20,22 @@ You might note that the simulation framework and the simulator itself makes exte The `opendc` package consists of a number of submodules, the most feature-rich being `opendc-compute`. Below, we will explain each in turn. ### 3.2.1 `opendc-core` -This module defines a base model for datacenter simulation, establishing core concepts and terminology of datacenters. -The other `opendc` modules build on this model and extend it in various directions (e.g. virtual machines or workflows). +This module defines a base model for datacenter simulation, establishing core concepts and terminology of datacenters +that we share across the various modules. Other `opendc` modules build on this model and extend it in various directions (e.g. virtual machines or workflows). ### 3.2.2 `opendc-compute` -This module models management and provisioning of compute instances such as virtual machines and bare-metal servers. We -represent such compute instances as a `Server`. The hardware configuration of the server is represented as a `Flavor`. -Servers run bootable disk images called `Image`s, which characterizes the runtime behavior of a server. +This module is concerned with modeling cloud computing services (such as [Amazon EC2](https://aws.amazon.com/ec2/) and [Google Compute Engine](https://cloud.google.com/compute)) and provides the following features: + +1. **Model for simulated workloads** + We represent workloads as bootable disk images (called `Image`) which characterize the runtime behavior + of a running server in terms of the cpu time they require over time. +2. **Bare-metal management & provisioning** + We provide models for simulating management of bare-metal machines (`Node`) and deploying workloads on it (using `BareMetalDriver`). + Furthermore, we also include functionality for simulating and experimenting with (custom) provisioning policies on a pool of bare-metal machies (using `ProvisioningService`) +3. **Virtual machine management, scheduling and provisioning** + We provide functionality for simulating deployment of multiple `Image`s on a single machine using a hypervisor, which + is concerned with scheduling/distributing the load of the running guest machines on the host machine. This may be used to experiment with overcommitting of virtual resources. + Moreover, we also model provisioning policies for virtual machine provisioning with a pool of host machines. ### 3.2.3 `opendc-workflows` This module contains all workflow-related models and logic of the simulator. The models for workflows can be found in the `workload` package: A `Job` and a `Task`. The logic concerning the scheduling of a workflow is contained in the `service` package. It follows the Reference Architecture for Datacenter Schedulers by [Andreadis et al.](https://dl.acm.org/doi/10.5555/3291656.3291706). For a good introduction into datacenter schedulers and to fully grasp the modeling approach taken, we highly recommend reading this publication (or its more extensive [technical report](https://arxiv.org/pdf/1808.04224.pdf)). -- cgit v1.2.3 From f334239b072855561ee522d70d78d8b61e1b6474 Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Tue, 18 Feb 2020 21:14:56 +0100 Subject: Update architecture.md Adds missing fullstop --- docs/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'docs/architecture.md') diff --git a/docs/architecture.md b/docs/architecture.md index 33fb90e5..940ec09b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -31,7 +31,7 @@ This module is concerned with modeling cloud computing services (such as [Amazon of a running server in terms of the cpu time they require over time. 2. **Bare-metal management & provisioning** We provide models for simulating management of bare-metal machines (`Node`) and deploying workloads on it (using `BareMetalDriver`). - Furthermore, we also include functionality for simulating and experimenting with (custom) provisioning policies on a pool of bare-metal machies (using `ProvisioningService`) + Furthermore, we also include functionality for simulating and experimenting with (custom) provisioning policies on a pool of bare-metal machies (using `ProvisioningService`). 3. **Virtual machine management, scheduling and provisioning** We provide functionality for simulating deployment of multiple `Image`s on a single machine using a hypervisor, which is concerned with scheduling/distributing the load of the running guest machines on the host machine. This may be used to experiment with overcommitting of virtual resources. -- cgit v1.2.3