diff options
| -rw-r--r-- | README.md | 14 | ||||
| -rw-r--r-- | docs/architecture.md | 52 | ||||
| -rw-r--r-- | docs/introduction.md | 15 | ||||
| -rw-r--r-- | docs/run.md | 5 | ||||
| -rw-r--r-- | docs/setup.md | 47 | ||||
| -rw-r--r-- | odcsim/README.md | 15 | ||||
| -rw-r--r-- | odcsim/docs/build.md | 1 | ||||
| -rw-r--r-- | odcsim/docs/concepts.md | 0 | ||||
| -rw-r--r-- | odcsim/docs/models.md | 0 | ||||
| -rw-r--r-- | odcsim/docs/run.md | 0 |
10 files changed, 135 insertions, 14 deletions
@@ -24,10 +24,22 @@ This repository hosts the source code and development of the simulation componen - **[opendc](/opendc)** A collection of models for simulating clouds, datacenters and their components using the **odcsim** framework. +## Documentation +Check out the [Getting Started](#getting-started) section for a quick +overview. +The documentation is located in the [docs/](docs) directory and is divided as follows: +1. [Introduction](docs/introduction.md) +1. [Toolchain Setup](docs/setup.md) +2. [Architecture Overview](docs/architecture.md) +3. [Running an Experiment](docs/run.md) + +## Getting Started +TODO + ## Contributing ### Contributing Guide Read our [contributing guide](CONTRIBUTING.md) to learn about our -development process, how to propose bug fixes and improvements, and how +development process, how to propose bug fixes and improvements, and how to build and test your changes to the project. ### License diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..940ec09b --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,52 @@ +# 3. Architecture Overview +The OpenDC simulator has two main components: `odcsim`, a general framework for discrete event-based simulation, and `opendc` a collection of components using `odcsim` to simulate datacenters. While `opendc` is the focus of this project and thus these docs, we also shortly explain what `odcsim` looks like, to give you a better understanding of the other concepts. + +**TODO: Add a schematic general overview here.** + +## 3.1 `odcsim` +The `odcsim` framework has an API module (`odcsim-api`) and an module implementing this API (`odcsim-engine-omega`). + +### 3.1.1 `odcsim-api` +The API defines the behavior of any implementation adhering to the `odcsim` framework. It centers around a `SimulationEngine`, which can be created through a `SimulationEngineProvider`. + +A simulation has a root process with certain `Behavior` (a coroutine). Processes have a `ProcessContext` which allow them to spawn other processes and open communication `Channel`s with other processes. Each of these `Channel`s has a `SendPort` and a `ReceivePort`. + +### 3.1.2 `odcsim-engine-omega` +The implementation is an executable interpretation of this API. The main class is `OmegaSimulationEngine` and takes care of transmitting timestamped events between processes. It ensures that message delivery order is the same as sent order. + +You might note that the simulation framework and the simulator itself makes extensive use of Kotlin's coroutine feature. This is a paradigm for asynchronous programming. If you are not familiar with coroutines, we suggest having a look at the [Kotlin documentation on this](https://kotlinlang.org/docs/reference/coroutines-overview.html), especially their [quick start guide](https://kotlinlang.org/docs/tutorials/coroutines/coroutines-basic-jvm.html) and their [hands-on guide](https://play.kotlinlang.org/hands-on/Introduction%20to%20Coroutines%20and%20Channels/01_Introduction). + +## 3.2 `opendc` +The `opendc` package consists of a number of submodules, the most feature-rich being `opendc-compute`. Below, we will explain each in turn. + +### 3.2.1 `opendc-core` +This module defines a base model for datacenter simulation, establishing core concepts and terminology of datacenters +that we share across the various modules. Other `opendc` modules build on this model and extend it in various directions (e.g. virtual machines or workflows). + +### 3.2.2 `opendc-compute` +This module is concerned with modeling cloud computing services (such as [Amazon EC2](https://aws.amazon.com/ec2/) and [Google Compute Engine](https://cloud.google.com/compute)) and provides the following features: + +1. **Model for simulated workloads** + We represent workloads as bootable disk images (called `Image`) which characterize the runtime behavior + of a running server in terms of the cpu time they require over time. +2. **Bare-metal management & provisioning** + We provide models for simulating management of bare-metal machines (`Node`) and deploying workloads on it (using `BareMetalDriver`). + Furthermore, we also include functionality for simulating and experimenting with (custom) provisioning policies on a pool of bare-metal machies (using `ProvisioningService`). +3. **Virtual machine management, scheduling and provisioning** + We provide functionality for simulating deployment of multiple `Image`s on a single machine using a hypervisor, which + is concerned with scheduling/distributing the load of the running guest machines on the host machine. This may be used to experiment with overcommitting of virtual resources. + Moreover, we also model provisioning policies for virtual machine provisioning with a pool of host machines. + +### 3.2.3 `opendc-workflows` +This module contains all workflow-related models and logic of the simulator. The models for workflows can be found in the `workload` package: A `Job` and a `Task`. The logic concerning the scheduling of a workflow is contained in the `service` package. It follows the Reference Architecture for Datacenter Schedulers by [Andreadis et al.](https://dl.acm.org/doi/10.5555/3291656.3291706). For a good introduction into datacenter schedulers and to fully grasp the modeling approach taken, we highly recommend reading this publication (or its more extensive [technical report](https://arxiv.org/pdf/1808.04224.pdf)). + +The `service` package merits its own explanation. A scheduler is defined by the `StageWorkflowScheduler` class, according to the architecture. The main component, however, is the `StageWorkflowSchedulerLogic`, responsible for pulling together the different scheduling stage implementations from the `stage` package. The scheduler is managed by the `WorkflowService`, which also orchestrates the lifecycle of a workflow. + +### 3.2.4 `opendc-format` +Running scientific experiments does not require running the full OpenDC stack. We also support directly reading out environment and workload trace files. Example implementations of these can be found in the `opendc-format` module. To parse a different format, you can take one of the existing parsers and adapt it to your needs. + +### 3.2.5 `opendc-experiments-sc18` +This is a module created for the experiments of our [SC18 publication](https://dl.acm.org/doi/10.5555/3291656.3291706). We aim to separate these kinds of custom experiment setups from the rest of the codebase by placing them in separate modules. + +--- +[< Previous](setup.md) | [Next >](run.md) diff --git a/docs/introduction.md b/docs/introduction.md new file mode 100644 index 00000000..711a829b --- /dev/null +++ b/docs/introduction.md @@ -0,0 +1,15 @@ +# 1. Introduction +This repository hosts the source code and development of the simulation component of the [OpenDC](https://opendc.org) project. OpenDC is a simulation platform for cloud infrastructure, with the aim of making datacenter concepts and technology accessible to everyone. To learn more about OpenDC, have a look through our [vision paper](https://ieeexplore.ieee.org/document/8121623)! + +The simulator is one of the components of the OpenDC stack. It is responsible for modelling and simulation of datacenters and their components. This entails receiving environments and experiment configurations from up the stack, simulating these configurations, and reporting back results. + +The simulator is composed of two main components: +- **[odcsim](/odcsim)** + A framework for discrete event simulation. +- **[opendc](/opendc)** + A collection of models for simulating clouds, datacenters and their components, using the **odcsim** framework. + +In the remainder of this documentation, we walk you through a typical toolchain setup for amending OpenDC in your own way, explain the architecture of the system, and show you how to run your own experiments with the simulator. + +--- +[Next >](setup.md) diff --git a/docs/run.md b/docs/run.md new file mode 100644 index 00000000..76be70ba --- /dev/null +++ b/docs/run.md @@ -0,0 +1,5 @@ +# 4. Running an Experiment +**TODO: What did we do for SC18?** + +--- +[< Previous](architecture.md) diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 00000000..801dfe58 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,47 @@ +# 2. Toolchain Setup +The OpenDC simulator is built using the [Kotlin](https://kotlinlang.org/) language. This is a JVM-based language that should appear familiar to programmers knowledgeable in Java or Scala. For a short interactive introduction to Kotlin, the [Learn Kotlin By Example](https://play.kotlinlang.org/byExample/overview) docs are a great place to start. + +For the build and dependency toolchain, we use [Gradle](https://gradle.org/). You will likely not need to change the Gradle build configurations of components, but you will use Gradle to execute builds and tests on the codebase. + +Follow the steps below to get it all set up! + +## 2.1 Installing Java +Kotlin requires a Java installation of version 8 or higher. Make sure to install the [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html), not only the JRE (the JDK also includes a JRE). + +## 2.2 Building and Developing +With Java installed, we're ready to set up the development environment on your machine. You can either use a visual IDE or work from a command line shell. We outline both approaches below, feel free to choose which you are most comfortable with. If in doubt which one to choose, we suggest going with the first one. + +## 2.2.1 Setup With IntelliJ IDEA +We suggest using [IntelliJ IDEA](https://www.jetbrains.com/idea/) as development environment. Once you have installed any version of this IDE on your machine, choose "Get from Version Control" in the new project dialogue. Enter `https://atlarge.ewi.tudelft.nl/gitlab/opendc/opendc-simulator` as URL and submit your credentials when asked. Open the project once it's ready fetching the codebase, and let it set up with the defaults (IntelliJ will recognize that this is a Gradle codebase). + +You will now be prompted in a dialogue to enable auto-import for Gradle, which we suggest you do. Wait for any progress bars in the lower bar to disappear and then look for the Gradle context menu on the right hand side. In it, go to `opendc-simulator > Tasks > verification > test`. This will build the codebase and run checks to verify that tests pass. If you get a `BUILD SUCCESSFUL` message, you're ready to go to the [next section](architecture.md)! + +## 2.2.2 Setup With Command Line +First, clone the repository with the following command: + +```shell script +git clone https://atlarge.ewi.tudelft.nl/gitlab/opendc/opendc-simulator +``` + +And enter the directory: + +```shell script +cd opendc-simulator +``` + +If on Windows, run the batch file included in the root, as follows: + +```commandline +gradlew.bat test +``` + +If on Linux/macOS, run the shell script included in the root, as follows: + +```shell script +./gradlew test +``` + +If the build is successful, you are ready to go to the next section! + +--- +[< Previous](introduction.md) | [Next >](architecture.md) diff --git a/odcsim/README.md b/odcsim/README.md index e5dcc319..b930315a 100644 --- a/odcsim/README.md +++ b/odcsim/README.md @@ -11,21 +11,12 @@ by the [OpenDC](https://opendc.org) project. Simulations are defined in terms of a hierarchical grouping of actors and the interactions between these actors -([Actor model](https://en.wikipedia.org/wiki/Actor_model)), using -an API similar to [Akka](https://doc.akka.io/docs/akka/current/index.html). - -## Documentation -Check out the [Getting Started](#getting-started) section for a quick -overview. -The documentation is located in the [docs/](docs/) directory and is divided as follows: -* [Main Concepts](docs/concepts.md) -* [Building a Model](docs/build.md) -* [Running a Model](docs/run.md) -* [API Reference](https://atlarge-research.github.io/opendc-simulator) -* [Contributing Guide](../CONTRIBUTING.md) +([Actor model](https://en.wikipedia.org/wiki/Actor_model)). ## Getting Started +The instructions below are meant for when you would like to use `odcsim` separately from `opendc`. If you simply want to use the simulator, you do not need to follow them, please refer to the [main README](../README.md) for instructions. + ### Installation Please add the required packages as dependency in your project. Releases are available in the [Maven Central Repository](https://search.maven.org/). diff --git a/odcsim/docs/build.md b/odcsim/docs/build.md deleted file mode 100644 index f442e70b..00000000 --- a/odcsim/docs/build.md +++ /dev/null @@ -1 +0,0 @@ -# Building a Simulation Model diff --git a/odcsim/docs/concepts.md b/odcsim/docs/concepts.md deleted file mode 100644 index e69de29b..00000000 --- a/odcsim/docs/concepts.md +++ /dev/null diff --git a/odcsim/docs/models.md b/odcsim/docs/models.md deleted file mode 100644 index e69de29b..00000000 --- a/odcsim/docs/models.md +++ /dev/null diff --git a/odcsim/docs/run.md b/odcsim/docs/run.md deleted file mode 100644 index e69de29b..00000000 --- a/odcsim/docs/run.md +++ /dev/null |