From 8f75336cb802b770d1ac53efb155632dd2dd94d2 Mon Sep 17 00:00:00 2001 From: Fabian Mastenbroek Date: Thu, 8 Apr 2021 20:59:35 +0200 Subject: docs: Revamp project documentation This change updates the project documentation by moving most of the documentation to the docs directory. --- docs/architecture.md | 25 +++++++++ docs/deploy.md | 65 ++++++++++++++++++++++++ docs/images/logo.png | Bin 0 -> 2825 bytes docs/images/opendc-component-diagram.png | Bin 0 -> 39965 bytes docs/images/opendc-frontend-construction.png | Bin 0 -> 76461 bytes docs/images/opendc-frontend-simulation-zoom.png | Bin 0 -> 100583 bytes docs/images/opendc-frontend-simulation.png | Bin 0 -> 96351 bytes 7 files changed, 90 insertions(+) create mode 100644 docs/architecture.md create mode 100644 docs/deploy.md create mode 100644 docs/images/logo.png create mode 100644 docs/images/opendc-component-diagram.png create mode 100644 docs/images/opendc-frontend-construction.png create mode 100644 docs/images/opendc-frontend-simulation-zoom.png create mode 100644 docs/images/opendc-frontend-simulation.png (limited to 'docs') diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..032e8f4b --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,25 @@ +# Architecture + +OpenDC consists of four components: a Kotlin simulator, a MongoDB database, a Python +Flask [API](/opendc-web/opendc-web-api), and a React.js [frontend](/opendc-web/opendc-web-ui). + +

+ OpenDC Component Diagram +

+ +On the frontend, users can construct a topology by specifying a datacenter's rooms, racks and machines, and create +scenarios to see how a workload trace runs on that topology. The frontend communicates with the web server over +SocketIO, through a custom REST request/response layer. For example, the frontend might make a `GET` request +to `/api/v1/users/{userId}`, but this request is completed via SocketIO, not plain HTTP requests. Note that the API +itself can also be accessed by HTTP. + +The (Swagger/OpenAPI compliant) API spec specifies what requests the frontend can make to the web server. To view this +specification, go to the [Swagger Editor](https://editor.swagger.io/) and paste in +our [opendc-api-spec.yml](../opendc-api-spec.yml). + +The web server receives API requests and processes them in the database. When the frontend requests to run a new +scenario, the web server adds it to the `scenarios` collection in the database and sets its `state` as `QUEUED`. + +The simulator monitors the database for `QUEUED` scenarios, and simulates them as they are submitted. The results of the +simulations are processed and aggregated in memory. Afterwards, the aggregated summary is written to the database, which +the frontend can then again retrieve via the web server. diff --git a/docs/deploy.md b/docs/deploy.md new file mode 100644 index 00000000..b9d65d51 --- /dev/null +++ b/docs/deploy.md @@ -0,0 +1,65 @@ +# Deploying OpenDC + +### Preamble + +The official way to run OpenDC is using Docker. Other options include building and running locally, and building and +running to deploy on a server. + +For all of these options, you have to create a Google API Console project and client ID, which the OpenDC frontend and +web server will use to authenticate users and requests. +Follow [these steps](https://developers.google.com/identity/sign-in/web/sign-in) to make such a project. In the ' +Authorized JavaScript origins' and 'Authorized redirect URI' fields, be sure to add `http://localhost:8080` (frontend) +, `http://localhost:8081` (api) and `https://localhost:3000` (frontend dev). Download the JSON of the OAuth 2.0 client +ID you created from the Credentials tab, and specifically note the `client_id`, which you'll need to build OpenDC. + +### Installing Docker + +OpenDC uses [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) to orchestrate the +deployment of the software stack. Please refer to [Docker Desktop](https://www.docker.com/products/docker-desktop) for +instructions on how install Docker on your machine. + +### Running OpenDC + +To build and run the full OpenDC stack locally on Linux or Mac, you first need to clone the project: + +```bash +git clone https://github.com/atlarge-research/opendc.git + +# Enter the directory +cd opendc/ +``` + +In the directory you just entered, you need to set up a set of environment variables. To do this, create a file +called `.env` in the `opendc` folder. In this file, replace `your-google-oauth-client-id` with your `client_id` from the +OAuth client ID you created. For a standard setup, you can leave the other settings as-is. + +```.env +MONGO_INITDB_ROOT_USERNAME=root +MONGO_INITDB_ROOT_PASSWORD=rootpassword +MONGO_INITDB_DATABASE=admin +OPENDC_DB=opendc +OPENDC_DB_USERNAME=opendc +OPENDC_DB_PASSWORD=opendcpassword +OPENDC_FLASK_SECRET="This is a secret flask key, please change" +OPENDC_OAUTH_CLIENT_ID=your-google-oauth-client-id +OPENDC_API_BASE_URL=http://localhost:8081 +``` + +We provide a list of default traces for you to experiment with. If you want to add others, place them in the `traces` +directory and add entries to the database (see also [the database folder](database/mongo-init-opendc-db.sh)) + +If you plan to publicly deploy, please also tweak the other settings. In that case, also check the `docker-compose.yml` +and `docker-compose.pod.yml` for further instructions. + +Now, start the server: + +```bash +# Build the Docker image +docker-compose build + +# Start the containers +docker-compose up +``` + +Wait a few seconds and open `http://localhost:8080` in your browser to use OpenDC. We recommend Google Chrome for the +best development experience. diff --git a/docs/images/logo.png b/docs/images/logo.png new file mode 100644 index 00000000..d743038b Binary files /dev/null and b/docs/images/logo.png differ diff --git a/docs/images/opendc-component-diagram.png b/docs/images/opendc-component-diagram.png new file mode 100644 index 00000000..312ca72a Binary files /dev/null and b/docs/images/opendc-component-diagram.png differ diff --git a/docs/images/opendc-frontend-construction.png b/docs/images/opendc-frontend-construction.png new file mode 100644 index 00000000..223e8d48 Binary files /dev/null and b/docs/images/opendc-frontend-construction.png differ diff --git a/docs/images/opendc-frontend-simulation-zoom.png b/docs/images/opendc-frontend-simulation-zoom.png new file mode 100644 index 00000000..d7744926 Binary files /dev/null and b/docs/images/opendc-frontend-simulation-zoom.png differ diff --git a/docs/images/opendc-frontend-simulation.png b/docs/images/opendc-frontend-simulation.png new file mode 100644 index 00000000..bbf4cbd6 Binary files /dev/null and b/docs/images/opendc-frontend-simulation.png differ -- cgit v1.2.3 From 68eca742856c82ea2625001b377d995ba76a5f62 Mon Sep 17 00:00:00 2001 From: Fabian Mastenbroek Date: Mon, 3 May 2021 20:21:41 +0200 Subject: docs: Add document for toolchain setup --- docs/toolchain.md | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 docs/toolchain.md (limited to 'docs') diff --git a/docs/toolchain.md b/docs/toolchain.md new file mode 100644 index 00000000..4b029ebc --- /dev/null +++ b/docs/toolchain.md @@ -0,0 +1,70 @@ +# Toolchain Setup + +The OpenDC simulator is built using the [Kotlin](https://kotlinlang.org/) language. This is a JVM-based language that +should appear familiar to programmers knowledgeable in Java or Scala. For a short interactive introduction to Kotlin, +the [Learn Kotlin By Example](https://play.kotlinlang.org/byExample/overview) docs are a great place to start. + +For the build and dependency toolchain, we use [Gradle](https://gradle.org/). You will likely not need to change the +Gradle build configurations of components, but you will use Gradle to execute builds and tests on the codebase. + +Follow the steps below to get it all set up! + +## Contents + +1. [Installing Java](#1-installing-java) +1. [Building and Developing](#2-building-and-developing) + 1. [Setup with IntelliJ IDEA](#21-setup-with-intellij-idea) + 1. [Setup with Command Line](#22-setup-with-command-line) + +## 1. Installing Java + +Kotlin requires a Java installation of version 8 or higher. Make sure to install +the [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html), not only the JRE (the JDK also includes +a JRE). + +## 2. Building and Developing + +With Java installed, we're ready to set up the development environment on your machine. You can either use a visual IDE +or work from a command line shell. We outline both approaches below, feel free to choose which you are most comfortable +with. If in doubt which one to choose, we suggest going with the first one. + +## 2.1 Setup with IntelliJ IDEA + +We suggest using [IntelliJ IDEA](https://www.jetbrains.com/idea/) as development environment. Once you have installed +any version of this IDE on your machine, choose "Get from Version Control" in the new project dialogue. +Enter `https://github.com/atlarge-research/opendc` as URL and submit your credentials when asked. +Open the project once it's ready fetching the codebase, and let it set up with the defaults (IntelliJ will recognize +that this is a Gradle codebase). + +You will now be prompted in a dialogue to enable auto-import for Gradle, which we suggest you do. Wait for any progress +bars in the lower bar to disappear and then look for the Gradle context menu on the right-hand side. In it, go +to `opendc > Tasks > verification > test`. This will build the codebase and run checks to verify that tests +pass. If you get a `BUILD SUCCESSFUL` message, you're ready to go to the [next section](architecture.md)! + +## 2.2 Setup with Command Line + +First, clone the repository with the following command: + +```shell script +git clone https://github.com/atlarge-research/opendc +``` + +And enter the directory: + +```shell script +cd opendc +``` + +If on Windows, run the batch file included in the root, as follows: + +```commandline +gradlew.bat test +``` + +If on Linux/macOS, run the shell script included in the root, as follows: + +```shell script +./gradlew test +``` + +If the build is successful, you are ready to go to the next section! -- cgit v1.2.3 From f081b9cab58a31a55c4f9dc6ccedd8cb92fc9242 Mon Sep 17 00:00:00 2001 From: Fabian Mastenbroek Date: Mon, 3 May 2021 20:31:34 +0200 Subject: docs: Add document on OpenDC publications --- docs/research.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 docs/research.md (limited to 'docs') diff --git a/docs/research.md b/docs/research.md new file mode 100644 index 00000000..1eeefecf --- /dev/null +++ b/docs/research.md @@ -0,0 +1,35 @@ +## Research with OpenDC + +This document contains publications related to OpenDC or publications that have used OpenDC in their work. + +## Publications about OpenDC + +1. [OpenDC 2.0: Convenient Modeling and Simulation of Emerging Technologies in Cloud Datacenters](https://atlarge-research.com/pdfs/ccgrid21-opendc-paper.pdf) + **CCGrid 2021** + Fabian Mastenbroek, Georgios Andreadis, Soufiane Jounaid, Wenchen Lai, Jacob Burley, Jaro Bosch, Erwin van Eyk, + Laurens Versluis, Vincent van Beek, Alexandru Iosup +1. [The OpenDC vision: Towards collaborative datacenter simulation and exploration for everybody](https://atlarge-research.com/pdfs/opendc-vision17ispdc_cr.pdf) + **ISPDC 2017 (invited paper)** + Alexandru Iosup, Georgios Andreadis, Vincent van Beek, Matthijs Bijman, Erwin van Eyk, Mihai Neacsu, Leon Overweel, + Sacheendra Talluri, Laurens Versluis, Maaike Visser. + +## Publications using OpenDC + +1. [A Reference Architecture for Datacenter Scheduling](https://arxiv.org/pdf/1808.04224) + **International Conference for High Performance Computing, Networking, Storage and Analysis 2018 (SC18)** + Georgios Andreadis, Laurens Versluis, Fabian S. Mastenbroek, Alexandru Iosup + +## Students using OpenDC + +1. [Capelin: Fast Data-Driven Capacity Planning for Cloud Datacenters](https://repository.tudelft.nl/islandora/object/uuid:d6d50861-86a3-4dd3-a13f-42d84db7af66?collection=education) + **Master Thesis, 2020** + Georgios Andreadis +1. [OpenDC Serverless: Design, Implementation and Evaluation of a FaaS Platform Simulator](https://zenodo.org/record/4046675) + **Bachelor Thesis, 2020** + Soufiane Jounaid +1. LEGO, but with Servers: Creating the Building Blocks to Design and Simulate Datacenters + **Bachelor Thesis, 2020** + Jacob Burley +1. [A Systematic Design Space Exploration of Datacenter Schedulers](https://repository.tudelft.nl/islandora/object/uuid%3A20478016-cc7d-4c87-aa12-25b46f511277?collection=education) + **Bachelor Thesis, 2019** + Fabian S. Mastenbroek -- cgit v1.2.3