From 15730a89974e10fb19e689c0379a7e761fba0c14 Mon Sep 17 00:00:00 2001 From: Dante Niewenhuis Date: Tue, 5 Nov 2024 18:41:50 +0100 Subject: Hiding all tutorials that are currently not working to reduce confusion (#263) * Hidden all tutorials that are currently not working to reduce confusion * removed advanced guides from docusaurus * removed advanced guides from whats-next * removed advanced guides from support page --- site/community/0-support.mdx | 1 - site/docs/advanced-guides/_category_.json | 7 - site/docs/advanced-guides/architecture.md | 26 -- site/docs/advanced-guides/deploy.md | 84 ------- .../docs/advanced-guides/img/component-diagram.png | Bin 39965 -> 0 bytes site/docs/advanced-guides/toolchain.md | 74 ------ site/docs/documentation/Output.md | 5 +- site/docs/getting-started/3-whats-next.md | 1 - site/docs/tutorials/cloud-capacity-planning.mdx | 277 --------------------- site/docusaurus.config.js | 4 - site/old_files/advanced-guides/_category_.json | 7 + site/old_files/advanced-guides/architecture.md | 26 ++ site/old_files/advanced-guides/deploy.md | 84 +++++++ .../advanced-guides/img/component-diagram.png | Bin 0 -> 39965 bytes site/old_files/advanced-guides/toolchain.md | 74 ++++++ site/old_files/cloud-capacity-planning.mdx | 277 +++++++++++++++++++++ site/old_files/old_tutorials/0-installation.md | 31 +++ site/old_files/old_tutorials/1-design.mdx | 154 ++++++++++++ site/old_files/old_tutorials/2-experiment.mdx | 74 ++++++ site/old_files/old_tutorials/3-whats-next.md | 12 + site/old_files/old_tutorials/_category_.json | 8 + site/old_tutorials/0-installation.md | 31 --- site/old_tutorials/1-design.mdx | 154 ------------ site/old_tutorials/2-experiment.mdx | 74 ------ site/old_tutorials/3-whats-next.md | 12 - site/old_tutorials/_category_.json | 8 - 26 files changed, 750 insertions(+), 755 deletions(-) delete mode 100644 site/docs/advanced-guides/_category_.json delete mode 100644 site/docs/advanced-guides/architecture.md delete mode 100644 site/docs/advanced-guides/deploy.md delete mode 100644 site/docs/advanced-guides/img/component-diagram.png delete mode 100644 site/docs/advanced-guides/toolchain.md delete mode 100644 site/docs/tutorials/cloud-capacity-planning.mdx create mode 100644 site/old_files/advanced-guides/_category_.json create mode 100644 site/old_files/advanced-guides/architecture.md create mode 100644 site/old_files/advanced-guides/deploy.md create mode 100644 site/old_files/advanced-guides/img/component-diagram.png create mode 100644 site/old_files/advanced-guides/toolchain.md create mode 100644 site/old_files/cloud-capacity-planning.mdx create mode 100644 site/old_files/old_tutorials/0-installation.md create mode 100644 site/old_files/old_tutorials/1-design.mdx create mode 100644 site/old_files/old_tutorials/2-experiment.mdx create mode 100644 site/old_files/old_tutorials/3-whats-next.md create mode 100644 site/old_files/old_tutorials/_category_.json delete mode 100644 site/old_tutorials/0-installation.md delete mode 100644 site/old_tutorials/1-design.mdx delete mode 100644 site/old_tutorials/2-experiment.mdx delete mode 100644 site/old_tutorials/3-whats-next.md delete mode 100644 site/old_tutorials/_category_.json (limited to 'site') diff --git a/site/community/0-support.mdx b/site/community/0-support.mdx index a0a158bc..fb712459 100644 --- a/site/community/0-support.mdx +++ b/site/community/0-support.mdx @@ -10,7 +10,6 @@ Need help? You can find out ways to talk to maintainers and community members be
diff --git a/site/docs/advanced-guides/_category_.json b/site/docs/advanced-guides/_category_.json deleted file mode 100644 index a74f4f42..00000000 --- a/site/docs/advanced-guides/_category_.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "label": "Advanced Guides", - "position": 4, - "link": { - "type": "generated-index" - } -} diff --git a/site/docs/advanced-guides/architecture.md b/site/docs/advanced-guides/architecture.md deleted file mode 100644 index 2a65a6c6..00000000 --- a/site/docs/advanced-guides/architecture.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -sidebar_position: 2 ---- - -# Architecture - -OpenDC consists of four components: a Kotlin simulator, a SQL database, a Quarkus-based -[API](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-api), and a -React.js [frontend](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-api). - -![OpenDC Component Diagram](img/component-diagram.png) - -On the frontend, users can construct a topology by specifying a datacenter's rooms, racks and machines, and create -scenarios to see how a workload trace runs on that topology. The frontend communicates with the web server via a REST -API over HTTP. - -The (Swagger/OpenAPI compliant) API spec specifies what requests the frontend can make to the web server. To view this -specification, go to the [Swagger Editor](https://editor.swagger.io/) and paste in -our [API spec](https://api.opendc.org/q/openapi). - -The web server receives API requests and processes them in the database. When the frontend requests to run a new -scenario, the web server adds it to the `scenarios` collection in the database and sets its `state` as `PENDING`. - -The simulator monitors the database for `PENDING` scenarios, and simulates them as they are submitted. The results of -the simulations are processed and aggregated in memory. Afterwards, the aggregated summary is written to the database, -which the frontend can then again retrieve via the web server. diff --git a/site/docs/advanced-guides/deploy.md b/site/docs/advanced-guides/deploy.md deleted file mode 100644 index 2ee69c07..00000000 --- a/site/docs/advanced-guides/deploy.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -sidebar_position: 3 ---- - -# Deploying OpenDC -This document explains how you can deploy a multi-tenant instance of OpenDC using Docker. - -## Contents - -1. [Setting up Auth0](#setting-up-auth0) -1. [Installing Docker](#installing-docker) -1. [Running OpenDC from source](#running-opendc-from-source) - -## Setting up Auth0 - -OpenDC uses [Auth0](https://auth0.com) as Identity Provider so that OpenDC does not have to manage user data itself, -which greatly simplifies our frontend and backend implementation. We have chosen to use Auth0 as it is a well-known -Identity Provider with good software support and a free tier for users to experiment with. - -To deploy OpenDC yourself, you need to have an [Auth0 tenant](https://auth0.com/docs/get-started/learn-the-basics) and -create: - -1. **An API** - You need to define the OpenDC API server in Auth0. Please refer to the [following guide](https://auth0.com/docs/quickstart/backend/python/01-authorization#create-an-api) - on how to define an API in Auth0. - - Remember the identifier you created the API with, as we need it in the next steps (as `OPENDC_AUTH0_AUDIENCE`). -2. **A Single Page Application (SPA)** - You need to define the OpenDC frontend application in Auth0. Please see the [following guide](https://auth0.com/docs/quickstart/spa/react#configure-auth0) - on how you can define an SPA in Auth0. Make sure you have added the necessary URLs to the _Allowed Callback URLs_: - for a local deployment, you should add at least `http://localhost:3000, http://localhost:8080`. - - Once your application has been created, you should have a _Domain_ and _Client ID_ which we need to pass to the - frontend application (as `OPENDC_AUTH0_DOMAIN` and `OPENDC_AUTH0_CLIENT_ID` respectively). - - -## Installing Docker - -OpenDC uses [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) to orchestrate the -deployment of the software stack. Please refer to [Docker Desktop](https://www.docker.com/products/docker-desktop) for -instructions on how install Docker on your machine. - -## Running OpenDC from source - -To build and run the full OpenDC stack locally on Linux or Mac, you first need to clone the project: - -```bash -git clone https://github.com/atlarge-research/opendc.git - -# Enter the directory -cd opendc/ -``` - -In the directory you just entered, you need to set up a set of environment variables. To do this, create a file -called `.env` in the `opendc` folder. In this file, replace `your-auth0-*` with the Auth0 details you got from the first -step. For a standard setup, you can leave the other settings as-is. - -```.env -OPENDC_DB_USERNAME=opendc -OPENDC_DB_PASSWORD=opendcpassword -OPENDC_AUTH0_DOMAIN=your-auth0-domain -OPENDC_AUTH0_CLIENT_ID=your-auth0-client-id -OPENDC_AUTH0_AUDIENCE=your-auth0-api-identifier -OPENDC_API_BASE_URL=http://web -``` - -We provide a set of default traces for you to experiment with. If you want to add others, place them in the `traces` -directory and add entries to the database (see also [the SQL init script](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-server/src/main/resources/db/migration/V1.0.0__core.sql)) - -If you plan to deploy publicly, please also tweak the other settings. In that case, also check the `docker-compose.yml` -and `docker-compose.prod.yml` for further instructions. - -Now, start the server: - -```bash -# Build the Docker image -docker-compose build - -# Start the containers -docker-compose up -``` - -Wait a few seconds and open `http://localhost:8080` in your browser to use OpenDC. We recommend Google Chrome for the -best user experience. diff --git a/site/docs/advanced-guides/img/component-diagram.png b/site/docs/advanced-guides/img/component-diagram.png deleted file mode 100644 index 312ca72a..00000000 Binary files a/site/docs/advanced-guides/img/component-diagram.png and /dev/null differ diff --git a/site/docs/advanced-guides/toolchain.md b/site/docs/advanced-guides/toolchain.md deleted file mode 100644 index 1673d974..00000000 --- a/site/docs/advanced-guides/toolchain.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -sidebar_position: 1 ---- - -# Toolchain Setup - -The OpenDC simulator is built using the [Kotlin](https://kotlinlang.org/) language. This is a JVM-based language that -should appear familiar to programmers knowledgeable in Java or Scala. For a short interactive introduction to Kotlin, -the [Learn Kotlin By Example](https://play.kotlinlang.org/byExample/overview) docs are a great place to start. - -For the build and dependency toolchain, we use [Gradle](https://gradle.org/). You will likely not need to change the -Gradle build configurations of components, but you will use Gradle to execute builds and tests on the codebase. - -Follow the steps below to get it all set up! - -## Contents - -1. [Installing Java](#1-installing-java) -2. [Building and Developing](#2-building-and-developing) -3. [Setup with IntelliJ IDEA](#21-setup-with-intellij-idea) -4. [Setup with Command Line](#22-setup-with-command-line) - -## 1. Installing Java - -OpenDC requires a Java installation of version 17 or higher. Make sure to install -the [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html), not only the JRE (the JDK also includes -a JRE). - -## 2. Building and Developing - -With Java installed, we're ready to set up the development environment on your machine. You can either use a visual IDE -or work from a command line shell. We outline both approaches below, feel free to choose which you are most comfortable -with. If in doubt which one to choose, we suggest going with the first one. - -## 2.1 Setup with IntelliJ IDEA - -We suggest using [IntelliJ IDEA](https://www.jetbrains.com/idea/) as development environment. Once you have installed -any version of this IDE on your machine, choose "Get from Version Control" in the new project dialogue. -Enter `https://github.com/atlarge-research/opendc` as URL and submit your credentials when asked. -Open the project once it's ready fetching the codebase, and let it set up with the defaults (IntelliJ will recognize -that this is a Gradle codebase). - -You will now be prompted in a dialogue to enable auto-import for Gradle, which we suggest you do. Wait for any progress -bars in the lower bar to disappear and then look for the Gradle simHyperVisorContext menu on the right-hand side. In it, go -to `opendc > Tasks > verification > test`. This will build the codebase and run checks to verify that tests -pass. If you get a `BUILD SUCCESSFUL` message, you're ready to go to the [next section](architecture)! - -## 2.2 Setup with Command Line - -First, clone the repository with the following command: - -```shell script -git clone https://github.com/atlarge-research/opendc -``` - -And enter the directory: - -```shell script -cd opendc -``` - -If on Windows, run the batch file included in the root, as follows: - -```commandline -gradlew.bat test -``` - -If on Linux/macOS, run the shell script included in the root, as follows: - -```shell script -./gradlew test -``` - -If the build is successful, you are ready to go to the next section! diff --git a/site/docs/documentation/Output.md b/site/docs/documentation/Output.md index 3f9eb3d5..339ac615 100644 --- a/site/docs/documentation/Output.md +++ b/site/docs/documentation/Output.md @@ -1,6 +1,7 @@ -Running OpenDC results in three output files. The first file ([Server](#server)) contains metrics related to the jobs being executed. -The second file ([Host](#host)) contains all metrics related to the hosts on which jobs can be executed. Finally, the third file ([Service](#service)) +Running OpenDC results in three output files. The first file ([Task](#task)) contains metrics related to the jobs being executed. +The second file ([Host](#host)) contains all metrics related to the hosts on which jobs can be executed. The third file ([Power](#power)) +contains all metrics related to the power sources that power the hosts. Finally, the third file ([Service](#service)) contains metrics describing the overall performance. An experiment in OpenDC has ### Task diff --git a/site/docs/getting-started/3-whats-next.md b/site/docs/getting-started/3-whats-next.md index 7c021119..e613eba7 100644 --- a/site/docs/getting-started/3-whats-next.md +++ b/site/docs/getting-started/3-whats-next.md @@ -7,6 +7,5 @@ description: How to supercharge your designs and experiments with OpenDC. Congratulations! You have just learned how to design and experiment with a (virtual) datacenter in OpenDC. What's next? - Follow one of the [tutorials](/docs/category/tutorials) using OpenDC. -- Check the [advanced guides](/docs/category/advanced-guides) for more complex material. - Read about [existing work using OpenDC](/community/research). - Get involved in the [OpenDC Community](/community/support). diff --git a/site/docs/tutorials/cloud-capacity-planning.mdx b/site/docs/tutorials/cloud-capacity-planning.mdx deleted file mode 100644 index df9cb566..00000000 --- a/site/docs/tutorials/cloud-capacity-planning.mdx +++ /dev/null @@ -1,277 +0,0 @@ ---- -sidebar_position: 1 -title: Cloud Capacity Planning -hide_title: true -sidebar_label: Cloud Capacity Planning -description: Cloud Capacity Planning ---- - -# Cloud Capacity Planning Tutorial - -Using OpenDC to plan and design cloud datacenters. - -:::info Learning goal - -By doing this assignment, you will learn more about the basic concepts of datacenters and cloud computing systems, as -well as how such systems are planned and designed. -::: - -## Preamble - -Datacenter infrastructure is important in today’s digital society. Stakeholders across industry, government, and -academia employ a vast and diverse array of cloud services hosted by datacenter infrastructure, and expect services to -be reliable, high speed, and low cost. In turn, datacenter operators must maintain efficient operation at unprecedented -scale. - -To keep up with growing demand and increasing complexity, architects of datacenters must address complex challenges in -distributed systems, software engineering and performance engineering. One of these challenges is efficient utilization -of resources in datacenters, which is only 6-12% industry-wide despite the fact that it is inconvenient for datacenter -operators to keep much of their infrastructure idle, due to resulting high energy consumption and thus unnecessary -costs. - -It is often quite difficult to implement optimizations or other changes in datacenters. Datacenter operators tend to be -conservative in adopting such changes in fear of failure or misbehaving systems. Furthermore, testing changes at the -scale of modern datacenter infrastructure in a real-world setting is prohibitively expensive and hard to reproduce, -notwithstanding environmental concerns. - -A more viable alternative is the use of datacenter simulators such as OpenDC or CloudSim. These tools model datacenter -infrastructure at a good accuracy and allow us to test changes in a controllable and repeatable environment. - -In this tutorial, we will use the OpenDC datacenter simulator to experiment with datacenters and demonstrate the -process of designing and optimizing datacenters using simulation. - -## What is OpenDC - -OpenDC is an open source platform for datacenter simulation developed by AtLarge Research. The purpose of OpenDC is -twofold: we aim to both enable cloud computing education and support research into datacenters. -An example of the former is this tutorial, and examples of the latter include the numerous BSc and MSc research projects -that are using OpenDC to run experiments and perform research. - -:::caution - -OpenDC is still an experimental tool. Your data may get lost, overwritten, or otherwise become unavailable. Sorry for -the inconvenience. - -::: - -If you are not familiar with the OpenDC web interface, please follow the [Getting Started](/docs/category/getting-started) -guide to get an understanding of the main concepts of OpenDC and how to design a datacenter. - -:::note Action - -Set up a project on the OpenDC website. - -::: - - -## Assignment - -Acme Inc. is a small datacenter operator in the Netherlands. They are currently in the process of acquiring a new client -and closing a deal where Acme will migrate all of the client’s business-critical workloads from internal machines to -Acme’s datacenters. With this deal, the client aims to outsource the maintenance of their digital infrastructure, but in -turn expects reliable and efficient operation from Acme. - -To demonstrate that Acme is capable of this task, it has started a pilot project with the client where Acme will migrate -already a small subset of the client’s workloads. You are an engineer at Acme. and have been tasked with the design and -procurement of the datacenter infrastructure required for this pilot project. - -To guide your design, the client has provided a workload trace of their business-critical workloads, which consist of -the historical runtime behavior of 50 virtual machines over time. These virtual machines differ in resource -requirements (e.g. number of vCPUs or memory) and in resource consumption over time. We can use OpenDC to simulate this -workload trace and validate your datacenter design. - -The assignment is divided into four parts: -1. Analyzing the requirements to estimate what resources are needed. -2. Building your design in OpenDC -3. Validating your design in OpenDC -4. Optimizing your design in OpenDC - -Make notes of your thoughts on the following assignments & questions and discuss with your partner(s). - -## Analyze the Requirements - -The first step of the assignment is to analyze the requirements of the client in order to come up with a reasonable -estimation of the datacenter infrastructure needed. This estimation will become our initial design which we will build -and validate in OpenDC. - -Since the client has provided a workload trace representative of the workload that will eventually be running in the -datacenter, we can use it to guide our design. In [Figure 1](#resource-distribution), the requested memory and vCPUs are -depicted for the virtual machines in the workload trace. - -:::note Action - -Determine the total amount of vCPUs and memory required in the trace. - -::: - -
- Resource requirements for the workload -
- Requested number of vCPUs and memory (in GB) by the - virtual machines in the workload. The left figure shows the number of virtual machines that have requested 1, 2, 4 or 8 - vCPUs. The right figure shows the amount of memory requested compared to the number of vCPUs in the virtual machine. -
-
- -Based on this information, we could choose to purchase a new machine for every virtual machine in the workload trace. -Such a design will most certainly be able to handle the workload. At the same time, it is much more expensive and -probably unnecessary. - -In [Figure 2](#cpu-usage), the CPU Usage (in MHz) of the virtual machines in the workload is depicted over time. Observe that the -median CPU usage of the virtual machines over the whole trace is approximately 100 MHz. This means that a 2-core -processor with a base clock 3500 MHz would have utilization of only 1.4% (`100 MHz / (3500 MHz x 2)`) for such a median -workload. - -
- CPU usage over time for the workload -
CPU Usage of the virtual machines in the workload over time.
-
- -Instead, we could try to fit multiple virtual machines onto a single machine. For instance, the 2-core processor -mentioned before is able to handle 70 virtual machines, each running at 100 MHz (`(3500 MHz x 2) / 100 MHz`), ignoring -virtualization overhead and memory requirements. - -:::note Action - -Make a rough estimate of the number of physical cores required to host the vCPUs in the workload trace. - -::: - -Now that we have an indication of the number of physical cores we need to have, we can start to compose the servers in -our datacenter. See **Table 1 and 2** for the equipment list you can choose from. Don’t forget to put enough memory in your -servers, or otherwise you risk that not all virtual machines will fit on the servers in your datacenter. - -| Processor | Intel® Xeon® E-2224G | Intel® Xeon® E-2244G | Intel® Xeon® E-2246G | -|----------------------------------|----------------------|----------------------|----------------------| -| Base clock (in MHz) | 3500 | 3800 | 3600 | -| Core count | 4 | 8 | 12 | -| Average power consumption (in W) | 71 | 71 | 80 | - -**Table 1:** Processor options for your datacenter - - -| Memory module | Crucial MTA9ASF2G72PZ-3G2E | Crucial MTA18ASF4G72PDZ-3G2E1 | -|----------------------------------|----------------------------|-------------------------------| -| Size (in GB) | 16 | 32 | -| Speed (in MHZ) | 3200 | 3200 | -**Table 2:** Memory options for your datacenter - - -:::note Action - -Create a plan detailing the servers you want to have in your datacenter and what resources (e.g. processor or memory) -they should contain. For instance, such a plan could look like: - -1. 8x Server (2x Intel® Xeon® E-2244G, 4x Crucial MTA18ASF4G72PDZ-3G2E1) - -::: - -:::tip Hint - -Budget more capacity than your initial estimates to prevent your datacenter from running at a very high -utilization. Think about how your datacenter would handle a machine failure, will you still have enough capacity left? - -::: - -## Build the datacenter - -Based on the plan we devised in the previous section, we will now construct a (virtual) datacenter in OpenDC. If you -have not yet used the OpenDC web interface to design a datacenter, please read [Getting Started](/docs/category/getting-started) -guide to get an understanding of the main concepts of OpenDC and how to design a datacenter. - -:::note Action - -Implement your plan in the OpenDC web interface. - -::: - -## Validate your design - -We are now at a stage where we can validate whether the datacenter we have just designed and built in OpenDC is suitable -for the workload of the client. We will use OpenDC to simulate the workload in our datacenter and keep track of several -metrics to ensure efficient and reliable operation. - -One of our concerns is that our datacenter does not have enough computing power to deal with the client’s -business-critical workload, leading to degraded performance and consequently an unhappy client. - -A metric that gives us an insight in performance degradation is the Overcommitted CPU Cycles, which represents the -number of CPU cycles that a virtual machine wanted to run, but could not due to the host machine not having enough -computing capacity at that moment. To keep track of this metric during simulation, we create a new portfolio by clicking -the ‘+’ next to “Portfolio” in the left sidebar and select the metrics of interest. - -:::note Action - -Add a new portfolio and select at least the following targets: -1. Overcommitted CPU Cycles -2. Granted CPU Cycles -3. Requested CPU Cycles -4. Maximum Number VMs Finished - -::: - -We will now try to simulate the client’s workload trace (called _Bitbrains (Sample)_ in OpenDC). By clicking on ‘New -Scenario’ below your created portfolio, we can create a base scenario which will represent our baseline datacenter -design which we will compare against future improvements. - -:::note Action - -Add a base scenario to your new portfolio and select as trace _Bitbrains (Sample)_. - -::: - -By creating a new scenario, you will schedule a simulation of your datacenter design that will run on one of the OpenDC -simulation servers. Press the Play button next to your portfolio to see the results of the simulations. If you have -chosen the _Bitbrains (Sample)_ trace, the results should usually appear within one minute or less depending on the queue -size. In case they do not appear within a reasonable timeframe, please contact the instructors. - -You can now see how your design has performed. Check whether all virtual machines have finished and whether the -_Overcommitted CPU Cycles_ metric is not too high. Try to aim for anything below 1 bn cycles. In the next section, we’ll -try to further optimize our design. For now, think of an explanation for the performance of your design. - -## Optimize your design - -Finally, let’s try to optimize your design so that it meets the requirements of the client and is beneficial for your -employer as well. In particular, your company is interested in the follow goals: - -1. Reducing _Overcommitted CPU Cycles_ to a minimum for reliability. -2. Reducing _Total Power Consumption_ to a minimum to save energy costs. - -:::note Action - -Add a new portfolio and select at least the following targets: -1. Overcommitted CPU Cycles -2. Granted CPU Cycles -3. Requested CPU Cycles -4. Total Power Consumption - -Then, add a base scenario to your new portfolio and select as trace _Bitbrains (Sample)_. - -::: - -Try to think of ways in which you can reduce both _Overcommitted CPU Cycles_ and _Total Power Consumption_. Create a new -topology based on your initial topology and apply the changes you have come up with. In this way, you can easily compare -the performance of different topologies in different scenarios. Note that the choice of scheduler might also influence -your results. - -:::tip Hint - -The choice of scheduler (and thus the placement of VMs) might also influence your results. - -::: - - -:::note Action - -1. Create a new topology based on your existing topology. -2. Add a new scenario to your created portfolio and select your newly created topology. -3. Compare the results against the base scenario. - -::: - -Repeat this approach until you are satisfied with your design. - -## Epilogue - -In this tutorial, you should have learned briefly about what datacenters are, and the process of designing and -optimizing a datacenter yourself. If you have any feedback (positive or negative) about your experience using OpenDC -during this tutorial, please let us know! diff --git a/site/docusaurus.config.js b/site/docusaurus.config.js index 659c6b4e..8fbef549 100644 --- a/site/docusaurus.config.js +++ b/site/docusaurus.config.js @@ -103,10 +103,6 @@ const config = { { label: "Tutorials", to: "/docs/category/tutorials" - }, - { - label: "Advanced Guides", - to: "/docs/category/advanced-guides" } ] }, diff --git a/site/old_files/advanced-guides/_category_.json b/site/old_files/advanced-guides/_category_.json new file mode 100644 index 00000000..a74f4f42 --- /dev/null +++ b/site/old_files/advanced-guides/_category_.json @@ -0,0 +1,7 @@ +{ + "label": "Advanced Guides", + "position": 4, + "link": { + "type": "generated-index" + } +} diff --git a/site/old_files/advanced-guides/architecture.md b/site/old_files/advanced-guides/architecture.md new file mode 100644 index 00000000..2a65a6c6 --- /dev/null +++ b/site/old_files/advanced-guides/architecture.md @@ -0,0 +1,26 @@ +--- +sidebar_position: 2 +--- + +# Architecture + +OpenDC consists of four components: a Kotlin simulator, a SQL database, a Quarkus-based +[API](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-api), and a +React.js [frontend](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-api). + +![OpenDC Component Diagram](img/component-diagram.png) + +On the frontend, users can construct a topology by specifying a datacenter's rooms, racks and machines, and create +scenarios to see how a workload trace runs on that topology. The frontend communicates with the web server via a REST +API over HTTP. + +The (Swagger/OpenAPI compliant) API spec specifies what requests the frontend can make to the web server. To view this +specification, go to the [Swagger Editor](https://editor.swagger.io/) and paste in +our [API spec](https://api.opendc.org/q/openapi). + +The web server receives API requests and processes them in the database. When the frontend requests to run a new +scenario, the web server adds it to the `scenarios` collection in the database and sets its `state` as `PENDING`. + +The simulator monitors the database for `PENDING` scenarios, and simulates them as they are submitted. The results of +the simulations are processed and aggregated in memory. Afterwards, the aggregated summary is written to the database, +which the frontend can then again retrieve via the web server. diff --git a/site/old_files/advanced-guides/deploy.md b/site/old_files/advanced-guides/deploy.md new file mode 100644 index 00000000..2ee69c07 --- /dev/null +++ b/site/old_files/advanced-guides/deploy.md @@ -0,0 +1,84 @@ +--- +sidebar_position: 3 +--- + +# Deploying OpenDC +This document explains how you can deploy a multi-tenant instance of OpenDC using Docker. + +## Contents + +1. [Setting up Auth0](#setting-up-auth0) +1. [Installing Docker](#installing-docker) +1. [Running OpenDC from source](#running-opendc-from-source) + +## Setting up Auth0 + +OpenDC uses [Auth0](https://auth0.com) as Identity Provider so that OpenDC does not have to manage user data itself, +which greatly simplifies our frontend and backend implementation. We have chosen to use Auth0 as it is a well-known +Identity Provider with good software support and a free tier for users to experiment with. + +To deploy OpenDC yourself, you need to have an [Auth0 tenant](https://auth0.com/docs/get-started/learn-the-basics) and +create: + +1. **An API** + You need to define the OpenDC API server in Auth0. Please refer to the [following guide](https://auth0.com/docs/quickstart/backend/python/01-authorization#create-an-api) + on how to define an API in Auth0. + + Remember the identifier you created the API with, as we need it in the next steps (as `OPENDC_AUTH0_AUDIENCE`). +2. **A Single Page Application (SPA)** + You need to define the OpenDC frontend application in Auth0. Please see the [following guide](https://auth0.com/docs/quickstart/spa/react#configure-auth0) + on how you can define an SPA in Auth0. Make sure you have added the necessary URLs to the _Allowed Callback URLs_: + for a local deployment, you should add at least `http://localhost:3000, http://localhost:8080`. + + Once your application has been created, you should have a _Domain_ and _Client ID_ which we need to pass to the + frontend application (as `OPENDC_AUTH0_DOMAIN` and `OPENDC_AUTH0_CLIENT_ID` respectively). + + +## Installing Docker + +OpenDC uses [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) to orchestrate the +deployment of the software stack. Please refer to [Docker Desktop](https://www.docker.com/products/docker-desktop) for +instructions on how install Docker on your machine. + +## Running OpenDC from source + +To build and run the full OpenDC stack locally on Linux or Mac, you first need to clone the project: + +```bash +git clone https://github.com/atlarge-research/opendc.git + +# Enter the directory +cd opendc/ +``` + +In the directory you just entered, you need to set up a set of environment variables. To do this, create a file +called `.env` in the `opendc` folder. In this file, replace `your-auth0-*` with the Auth0 details you got from the first +step. For a standard setup, you can leave the other settings as-is. + +```.env +OPENDC_DB_USERNAME=opendc +OPENDC_DB_PASSWORD=opendcpassword +OPENDC_AUTH0_DOMAIN=your-auth0-domain +OPENDC_AUTH0_CLIENT_ID=your-auth0-client-id +OPENDC_AUTH0_AUDIENCE=your-auth0-api-identifier +OPENDC_API_BASE_URL=http://web +``` + +We provide a set of default traces for you to experiment with. If you want to add others, place them in the `traces` +directory and add entries to the database (see also [the SQL init script](https://github.com/atlarge-research/opendc/tree/master/opendc-web/opendc-web-server/src/main/resources/db/migration/V1.0.0__core.sql)) + +If you plan to deploy publicly, please also tweak the other settings. In that case, also check the `docker-compose.yml` +and `docker-compose.prod.yml` for further instructions. + +Now, start the server: + +```bash +# Build the Docker image +docker-compose build + +# Start the containers +docker-compose up +``` + +Wait a few seconds and open `http://localhost:8080` in your browser to use OpenDC. We recommend Google Chrome for the +best user experience. diff --git a/site/old_files/advanced-guides/img/component-diagram.png b/site/old_files/advanced-guides/img/component-diagram.png new file mode 100644 index 00000000..312ca72a Binary files /dev/null and b/site/old_files/advanced-guides/img/component-diagram.png differ diff --git a/site/old_files/advanced-guides/toolchain.md b/site/old_files/advanced-guides/toolchain.md new file mode 100644 index 00000000..1673d974 --- /dev/null +++ b/site/old_files/advanced-guides/toolchain.md @@ -0,0 +1,74 @@ +--- +sidebar_position: 1 +--- + +# Toolchain Setup + +The OpenDC simulator is built using the [Kotlin](https://kotlinlang.org/) language. This is a JVM-based language that +should appear familiar to programmers knowledgeable in Java or Scala. For a short interactive introduction to Kotlin, +the [Learn Kotlin By Example](https://play.kotlinlang.org/byExample/overview) docs are a great place to start. + +For the build and dependency toolchain, we use [Gradle](https://gradle.org/). You will likely not need to change the +Gradle build configurations of components, but you will use Gradle to execute builds and tests on the codebase. + +Follow the steps below to get it all set up! + +## Contents + +1. [Installing Java](#1-installing-java) +2. [Building and Developing](#2-building-and-developing) +3. [Setup with IntelliJ IDEA](#21-setup-with-intellij-idea) +4. [Setup with Command Line](#22-setup-with-command-line) + +## 1. Installing Java + +OpenDC requires a Java installation of version 17 or higher. Make sure to install +the [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html), not only the JRE (the JDK also includes +a JRE). + +## 2. Building and Developing + +With Java installed, we're ready to set up the development environment on your machine. You can either use a visual IDE +or work from a command line shell. We outline both approaches below, feel free to choose which you are most comfortable +with. If in doubt which one to choose, we suggest going with the first one. + +## 2.1 Setup with IntelliJ IDEA + +We suggest using [IntelliJ IDEA](https://www.jetbrains.com/idea/) as development environment. Once you have installed +any version of this IDE on your machine, choose "Get from Version Control" in the new project dialogue. +Enter `https://github.com/atlarge-research/opendc` as URL and submit your credentials when asked. +Open the project once it's ready fetching the codebase, and let it set up with the defaults (IntelliJ will recognize +that this is a Gradle codebase). + +You will now be prompted in a dialogue to enable auto-import for Gradle, which we suggest you do. Wait for any progress +bars in the lower bar to disappear and then look for the Gradle simHyperVisorContext menu on the right-hand side. In it, go +to `opendc > Tasks > verification > test`. This will build the codebase and run checks to verify that tests +pass. If you get a `BUILD SUCCESSFUL` message, you're ready to go to the [next section](architecture)! + +## 2.2 Setup with Command Line + +First, clone the repository with the following command: + +```shell script +git clone https://github.com/atlarge-research/opendc +``` + +And enter the directory: + +```shell script +cd opendc +``` + +If on Windows, run the batch file included in the root, as follows: + +```commandline +gradlew.bat test +``` + +If on Linux/macOS, run the shell script included in the root, as follows: + +```shell script +./gradlew test +``` + +If the build is successful, you are ready to go to the next section! diff --git a/site/old_files/cloud-capacity-planning.mdx b/site/old_files/cloud-capacity-planning.mdx new file mode 100644 index 00000000..df9cb566 --- /dev/null +++ b/site/old_files/cloud-capacity-planning.mdx @@ -0,0 +1,277 @@ +--- +sidebar_position: 1 +title: Cloud Capacity Planning +hide_title: true +sidebar_label: Cloud Capacity Planning +description: Cloud Capacity Planning +--- + +# Cloud Capacity Planning Tutorial + +Using OpenDC to plan and design cloud datacenters. + +:::info Learning goal + +By doing this assignment, you will learn more about the basic concepts of datacenters and cloud computing systems, as +well as how such systems are planned and designed. +::: + +## Preamble + +Datacenter infrastructure is important in today’s digital society. Stakeholders across industry, government, and +academia employ a vast and diverse array of cloud services hosted by datacenter infrastructure, and expect services to +be reliable, high speed, and low cost. In turn, datacenter operators must maintain efficient operation at unprecedented +scale. + +To keep up with growing demand and increasing complexity, architects of datacenters must address complex challenges in +distributed systems, software engineering and performance engineering. One of these challenges is efficient utilization +of resources in datacenters, which is only 6-12% industry-wide despite the fact that it is inconvenient for datacenter +operators to keep much of their infrastructure idle, due to resulting high energy consumption and thus unnecessary +costs. + +It is often quite difficult to implement optimizations or other changes in datacenters. Datacenter operators tend to be +conservative in adopting such changes in fear of failure or misbehaving systems. Furthermore, testing changes at the +scale of modern datacenter infrastructure in a real-world setting is prohibitively expensive and hard to reproduce, +notwithstanding environmental concerns. + +A more viable alternative is the use of datacenter simulators such as OpenDC or CloudSim. These tools model datacenter +infrastructure at a good accuracy and allow us to test changes in a controllable and repeatable environment. + +In this tutorial, we will use the OpenDC datacenter simulator to experiment with datacenters and demonstrate the +process of designing and optimizing datacenters using simulation. + +## What is OpenDC + +OpenDC is an open source platform for datacenter simulation developed by AtLarge Research. The purpose of OpenDC is +twofold: we aim to both enable cloud computing education and support research into datacenters. +An example of the former is this tutorial, and examples of the latter include the numerous BSc and MSc research projects +that are using OpenDC to run experiments and perform research. + +:::caution + +OpenDC is still an experimental tool. Your data may get lost, overwritten, or otherwise become unavailable. Sorry for +the inconvenience. + +::: + +If you are not familiar with the OpenDC web interface, please follow the [Getting Started](/docs/category/getting-started) +guide to get an understanding of the main concepts of OpenDC and how to design a datacenter. + +:::note Action + +Set up a project on the OpenDC website. + +::: + + +## Assignment + +Acme Inc. is a small datacenter operator in the Netherlands. They are currently in the process of acquiring a new client +and closing a deal where Acme will migrate all of the client’s business-critical workloads from internal machines to +Acme’s datacenters. With this deal, the client aims to outsource the maintenance of their digital infrastructure, but in +turn expects reliable and efficient operation from Acme. + +To demonstrate that Acme is capable of this task, it has started a pilot project with the client where Acme will migrate +already a small subset of the client’s workloads. You are an engineer at Acme. and have been tasked with the design and +procurement of the datacenter infrastructure required for this pilot project. + +To guide your design, the client has provided a workload trace of their business-critical workloads, which consist of +the historical runtime behavior of 50 virtual machines over time. These virtual machines differ in resource +requirements (e.g. number of vCPUs or memory) and in resource consumption over time. We can use OpenDC to simulate this +workload trace and validate your datacenter design. + +The assignment is divided into four parts: +1. Analyzing the requirements to estimate what resources are needed. +2. Building your design in OpenDC +3. Validating your design in OpenDC +4. Optimizing your design in OpenDC + +Make notes of your thoughts on the following assignments & questions and discuss with your partner(s). + +## Analyze the Requirements + +The first step of the assignment is to analyze the requirements of the client in order to come up with a reasonable +estimation of the datacenter infrastructure needed. This estimation will become our initial design which we will build +and validate in OpenDC. + +Since the client has provided a workload trace representative of the workload that will eventually be running in the +datacenter, we can use it to guide our design. In [Figure 1](#resource-distribution), the requested memory and vCPUs are +depicted for the virtual machines in the workload trace. + +:::note Action + +Determine the total amount of vCPUs and memory required in the trace. + +::: + +
+ Resource requirements for the workload +
+ Requested number of vCPUs and memory (in GB) by the + virtual machines in the workload. The left figure shows the number of virtual machines that have requested 1, 2, 4 or 8 + vCPUs. The right figure shows the amount of memory requested compared to the number of vCPUs in the virtual machine. +
+
+ +Based on this information, we could choose to purchase a new machine for every virtual machine in the workload trace. +Such a design will most certainly be able to handle the workload. At the same time, it is much more expensive and +probably unnecessary. + +In [Figure 2](#cpu-usage), the CPU Usage (in MHz) of the virtual machines in the workload is depicted over time. Observe that the +median CPU usage of the virtual machines over the whole trace is approximately 100 MHz. This means that a 2-core +processor with a base clock 3500 MHz would have utilization of only 1.4% (`100 MHz / (3500 MHz x 2)`) for such a median +workload. + +
+ CPU usage over time for the workload +
CPU Usage of the virtual machines in the workload over time.
+
+ +Instead, we could try to fit multiple virtual machines onto a single machine. For instance, the 2-core processor +mentioned before is able to handle 70 virtual machines, each running at 100 MHz (`(3500 MHz x 2) / 100 MHz`), ignoring +virtualization overhead and memory requirements. + +:::note Action + +Make a rough estimate of the number of physical cores required to host the vCPUs in the workload trace. + +::: + +Now that we have an indication of the number of physical cores we need to have, we can start to compose the servers in +our datacenter. See **Table 1 and 2** for the equipment list you can choose from. Don’t forget to put enough memory in your +servers, or otherwise you risk that not all virtual machines will fit on the servers in your datacenter. + +| Processor | Intel® Xeon® E-2224G | Intel® Xeon® E-2244G | Intel® Xeon® E-2246G | +|----------------------------------|----------------------|----------------------|----------------------| +| Base clock (in MHz) | 3500 | 3800 | 3600 | +| Core count | 4 | 8 | 12 | +| Average power consumption (in W) | 71 | 71 | 80 | + +**Table 1:** Processor options for your datacenter + + +| Memory module | Crucial MTA9ASF2G72PZ-3G2E | Crucial MTA18ASF4G72PDZ-3G2E1 | +|----------------------------------|----------------------------|-------------------------------| +| Size (in GB) | 16 | 32 | +| Speed (in MHZ) | 3200 | 3200 | +**Table 2:** Memory options for your datacenter + + +:::note Action + +Create a plan detailing the servers you want to have in your datacenter and what resources (e.g. processor or memory) +they should contain. For instance, such a plan could look like: + +1. 8x Server (2x Intel® Xeon® E-2244G, 4x Crucial MTA18ASF4G72PDZ-3G2E1) + +::: + +:::tip Hint + +Budget more capacity than your initial estimates to prevent your datacenter from running at a very high +utilization. Think about how your datacenter would handle a machine failure, will you still have enough capacity left? + +::: + +## Build the datacenter + +Based on the plan we devised in the previous section, we will now construct a (virtual) datacenter in OpenDC. If you +have not yet used the OpenDC web interface to design a datacenter, please read [Getting Started](/docs/category/getting-started) +guide to get an understanding of the main concepts of OpenDC and how to design a datacenter. + +:::note Action + +Implement your plan in the OpenDC web interface. + +::: + +## Validate your design + +We are now at a stage where we can validate whether the datacenter we have just designed and built in OpenDC is suitable +for the workload of the client. We will use OpenDC to simulate the workload in our datacenter and keep track of several +metrics to ensure efficient and reliable operation. + +One of our concerns is that our datacenter does not have enough computing power to deal with the client’s +business-critical workload, leading to degraded performance and consequently an unhappy client. + +A metric that gives us an insight in performance degradation is the Overcommitted CPU Cycles, which represents the +number of CPU cycles that a virtual machine wanted to run, but could not due to the host machine not having enough +computing capacity at that moment. To keep track of this metric during simulation, we create a new portfolio by clicking +the ‘+’ next to “Portfolio” in the left sidebar and select the metrics of interest. + +:::note Action + +Add a new portfolio and select at least the following targets: +1. Overcommitted CPU Cycles +2. Granted CPU Cycles +3. Requested CPU Cycles +4. Maximum Number VMs Finished + +::: + +We will now try to simulate the client’s workload trace (called _Bitbrains (Sample)_ in OpenDC). By clicking on ‘New +Scenario’ below your created portfolio, we can create a base scenario which will represent our baseline datacenter +design which we will compare against future improvements. + +:::note Action + +Add a base scenario to your new portfolio and select as trace _Bitbrains (Sample)_. + +::: + +By creating a new scenario, you will schedule a simulation of your datacenter design that will run on one of the OpenDC +simulation servers. Press the Play button next to your portfolio to see the results of the simulations. If you have +chosen the _Bitbrains (Sample)_ trace, the results should usually appear within one minute or less depending on the queue +size. In case they do not appear within a reasonable timeframe, please contact the instructors. + +You can now see how your design has performed. Check whether all virtual machines have finished and whether the +_Overcommitted CPU Cycles_ metric is not too high. Try to aim for anything below 1 bn cycles. In the next section, we’ll +try to further optimize our design. For now, think of an explanation for the performance of your design. + +## Optimize your design + +Finally, let’s try to optimize your design so that it meets the requirements of the client and is beneficial for your +employer as well. In particular, your company is interested in the follow goals: + +1. Reducing _Overcommitted CPU Cycles_ to a minimum for reliability. +2. Reducing _Total Power Consumption_ to a minimum to save energy costs. + +:::note Action + +Add a new portfolio and select at least the following targets: +1. Overcommitted CPU Cycles +2. Granted CPU Cycles +3. Requested CPU Cycles +4. Total Power Consumption + +Then, add a base scenario to your new portfolio and select as trace _Bitbrains (Sample)_. + +::: + +Try to think of ways in which you can reduce both _Overcommitted CPU Cycles_ and _Total Power Consumption_. Create a new +topology based on your initial topology and apply the changes you have come up with. In this way, you can easily compare +the performance of different topologies in different scenarios. Note that the choice of scheduler might also influence +your results. + +:::tip Hint + +The choice of scheduler (and thus the placement of VMs) might also influence your results. + +::: + + +:::note Action + +1. Create a new topology based on your existing topology. +2. Add a new scenario to your created portfolio and select your newly created topology. +3. Compare the results against the base scenario. + +::: + +Repeat this approach until you are satisfied with your design. + +## Epilogue + +In this tutorial, you should have learned briefly about what datacenters are, and the process of designing and +optimizing a datacenter yourself. If you have any feedback (positive or negative) about your experience using OpenDC +during this tutorial, please let us know! diff --git a/site/old_files/old_tutorials/0-installation.md b/site/old_files/old_tutorials/0-installation.md new file mode 100644 index 00000000..281e811f --- /dev/null +++ b/site/old_files/old_tutorials/0-installation.md @@ -0,0 +1,31 @@ +--- +description: How to install OpenDC locally, and start experimenting in no time. +--- + +# Installation + +This page describes how to set up and configure a local single-user OpenDC installation so that you can quickly get your +experiments running. You can also use the [hosted version of OpenDC](https://app.opendc.org) to get started even +quicker (The web server is however missing some more complex features). + + +## Prerequisites + +1. **Supported Platforms** + OpenDC is actively tested on Windows, macOS and GNU/Linux. +2. **Required Software** + A Java installation of version 19 or higher is required for OpenDC. You may download the + [Java distribution from Oracle](https://www.oracle.com/java/technologies/downloads/) or use the distribution provided + by your package manager. + +## Download + +To get an OpenDC distribution, download a recent version from our [Releases](https://github.com/atlarge-research/opendc/releases) page on GitHub. +For basic usage, the OpenDCExperimentRunner is all that is needed. + +## Setup + +Unpack the downloaded OpenDC distribution. Opening OpenDCExperimentRunner results in two folders, `bin` and `lib`. +`lib` contains all `.jar` files needed to run OpenDC. `bin` two executable versions of the OpenDCExperimentRunner. +In the following pages, we discuss how to run an experiment using the executables. + diff --git a/site/old_files/old_tutorials/1-design.mdx b/site/old_files/old_tutorials/1-design.mdx new file mode 100644 index 00000000..e8ab2c58 --- /dev/null +++ b/site/old_files/old_tutorials/1-design.mdx @@ -0,0 +1,154 @@ +--- +description: How to design a virtual datacenter in OpenDC from scratch. +--- + +# Design a Datacenter + +Now that you have installed OpenDC (or are using the hosted version), we will start designing a (virtual) datacenter +in OpenDC. + +## Before we start + +There are a couple of steps we need to perform before we can start designing a datacenter in OpenDC. First, we need to +enter the OpenDC web application. This done as follows: + +
+
+
+
+
+

Hosted Deployment

+ + To enter the hosted version of OpenDC, you need a user account. User management is provided + by Auth0, which allows you to login with social accounts or via + email. + +
+ +
+
+
+
+
+

Local Deployment

+ + The local distribution of OpenDC runs in single-user mode by default, which does not require + authentication. This allows you to quickly start designing and experimenting with new + datacenters. + +
+ +
+
+
+
+ +### Create a Project + +Next, we need to create a new project. Projects allow you to organize your designs and experiments together. +Click on ‘+ New Project’ in the right corner to open the project creation dialog. +Give your project a name and save it. You can now open it by clicking on it in the project table. If all went well, +you’re redirected to your new project, and are presented with an empty project overview. + +### Create a Topology + +In OpenDC, the datacenter design is also called a **topology**. This topology represents the physical layout of a +datacenter and specifies everything from the architectural layout of the datacenter’s rooms to which CPUs are in a +particular machine. + +To create a design (topology), click on ‘+ New Topology’ in the top right corner of the topology table. +Once you have created the topology, it will appear the topology table. By clicking on the topology, you will be +redirected to a (still empty) overview of the topology. From here, we'll start designing a datacenter. + +### Terminology + +Here’s an overview of some of the language you’ll find when designing a datacenter in OpenDC: + +- **Topology**: the physical layout of your datacenter +- **Room**: a room in the datacenter +- **Tile**: one of the tiles that forms a room +- **Rack**: a rack of servers that stands on top of a tile +- **Machine**: a machine that takes up a single slot in a server rack, containing several components such as CPUs, GPUs, + network interfaces and storage drives. + +## Build the datacenter + +Open the project and topology that you have created and click on the 'Floor Plan' tab (see [Figure 1](#floor-plan)). +We’re now in datacenter construction mode. Notice the grid on the canvas? That’s where you’ll place tiles, in order to +build rooms. Let’s take a moment to familiarize ourselves with the interface. + +If you dismiss the sidebar on your left, you have controls for zooming in and out. Next to the zooming buttons, you also +have a ‘Screenshot’ button, in case you want to record the state of the canvas and export it to an image file. On the +right side of the screen, you have the simHyperVisorContext menu. This menu changes depending on your zoom level. + +As there are currently no rooms, we are in ‘Building’ mode, and our only option is to ‘Construct a new room’. Click on +that button to build a first datacenter room - once you’ve clicked on it, every tile of the canvas that you click on +becomes a tile of that room. There is one restriction though: Each tile that you add must be adjacent to any previous +tiles that you have added. You can see for yourself which tile positions are clickable through the highlight color that +is shown on hovering over them. + +
+ Analysis of results reported by OpenDC +
The floor plan of a (virtual) datacenter in OpenDC.
+
+ +### Create a Room + +:::note Action + +Create at least a single room, with help of the above instructions. + +::: + +Once you’ve placed the tiles, you can give the room a name, if you want to. To do this, click on the room you want to +edit. You’ll notice the application going into ‘Room’ mode, allowing you to manipulate the topology of the datacenter at +a more fine-grained level. In the simHyperVisorContext menu, change the room name, and click on the ‘Save’ button. You can exit +‘Room’ mode by clicking on any of the darkened areas outside of the selected room. This will bring you back to +‘Building’ mode. + +### Place Server Racks + +:::note Action + +Add at least a single rack in the room. + +::: + +Empty rooms are of no use to the stakeholders of a datacenter. They want machines! Let’s place some racks in the room +to fulfill this demand. Click on the room and add some racks. To stop adding racks, click on the blue element in the +sidebar, again. + +### Fill the Racks with Servers + +:::note Action + +Add a couple of servers to the rack. + +::: + +To add actual servers to the empty racks, we’ll need to go one level deeper in the topological hierarchy of the +datacenter. Clicking on a rack lets you do just that. Once you’ve clicked on it, you’ll notice the simHyperVisorContext menu now +displaying slots. In each slot fits exactly one server unit. To add such a server unit, click on the ‘Add machine’ +button of that slot. +Just like in ‘Room’ mode, you can exit ‘Rack’ mode by clicking on any of the darkened tiles around the currently +selected rack. + +### Add Resources to the Servers + +We’re almost done creating our datacenter! The only problem we have is that the machines / servers we just added lack +any real resources (such as CPUs, GPUs, memory cards, and disk storage). + +:::note Action + +Populate the machines with CPU and memory resources. + +::: + +To do this, click on any machine you want to edit. Notice the simHyperVisorContext menu changing, with tabs to add different kinds of +units to your machine. Have a look around as to what can be added. + +Once you are satisfied with the datacenter design, we will experiment with the design in the next chapter. diff --git a/site/old_files/old_tutorials/2-experiment.mdx b/site/old_files/old_tutorials/2-experiment.mdx new file mode 100644 index 00000000..14970ea6 --- /dev/null +++ b/site/old_files/old_tutorials/2-experiment.mdx @@ -0,0 +1,74 @@ +--- +description: How to experiment with your datacenter designs. +--- + +# Create an Experiment + +After designing a datacenter in OpenDC, the next step is to experiment with the design using OpenDC's built-in +simulator, in order to analyze its performance and compare it against other designs. + +In OpenDC, we use the concept of portfolios of scenarios to experiment with datacenter designs. In the next sections, we +will explain how you can use these powerful concepts for the experimental analysis of your designs. + +## Create a Portfolio +OpenDC organizes multiple scenarios (experiments) into a **portfolio**. Each portfolio is composed of a base scenario, +a set of candidate scenarios given by the user and a set of targets (e.g., metrics) used to compare scenarios. + +To create a new portfolio, open a project in the OpenDC web interface and click on ‘+ New Portfolio’ in the top right +corner of the portfolio table. This opens a modal with the following options: + +1. **Name**: the name of your portfolio. +2. **Metrics**: the metrics that you want to use to compare the scenarios in the portfolio. +3. **Repeats per Scenario**: the number of times each scenario should be simulated (to account for variance). + +## Create Scenarios + +A **scenario** represents a point in the datacenter design space that should be explored. It consists of a combination +of workload, topology, and a set of operational phenomena. Phenomena can include correlated failures, performance +variability, security breaches, etc., allowing the scenarios to more accurately capture the real-world operations. + +The baseline for comparison in a portfolio is the **base scenario**. It represents the status quo of the infrastructure +or, when planning infrastructure from scratch, it consists of very simple base workloads and topologies. +The other scenarios in a portfolio, called the **candidate scenarios**, represent changes to the configuration +that might be of interest to the datacenter designer. Dividing scenarios into these two categories ensures that any +comparative insights provided by OpenDC are meaningful within the simHyperVisorContext of the current architecture. + +To create a new scenario, open a portfolio in the OpenDC web interface and click on ‘+ New Scenario’ in the top right +corner of the scenario table. This opens a modal with the following options (as shown in [Figure 1](#explore)): + +1. **Name**: the name of your scenario. The first scenario of a portfolio is always called the _Base scenario_. +2. **Workload**: the applications (e.g., virtual machines, workflows, functions) that consume resources in your + datacenter. + 1. **Workload Trace**: A dataset that characterizes the historical runtime behavior of virtual machines in the + workload over time. + 2. **Load Sampling Fraction**: The percentage of the workload that should be simulated (e.g., 10% of virtual + machines in the workload trace). +3. **Environment**: + 1. **Topology**: one of the topologies of the project to use for the scenario. + 2. **Scheduler**: the algorithm that decides on which hosts the virtual machines should be placed. +4. **Operational Phenomena**: + 1. **Failures**: a flag to enable stochastic host failures during simulation. + 2. **Performance interference**: a flag to enable performance interference between virtual machines (only available + for a subset of traces). + +Once you have created the scenario, it will be enqueued for simulation. Usually the results of the simulation should be +available within one minute after creation. However, if there are lots of queued simulation jobs, it might take a bit +longer. + +
+ Creating a new scenario in OpenDC +
Creating a new scenario in OpenDC. The user can select the topology, workload, and operational phenomena.
+
+ +## Analyze Results + +After creating scenarios, the scenario table will show whether the simulation is still pending, completed successfully, +or failed for some reason. If the scenario was simulated successfully, its results will become visible on the ‘Results’ +tab as shown in [Figure 2](#analysis). + +
+ Analysis of results reported by OpenDC +
Plots and visual summaries generated by OpenDC comparing different scenarios.
+
+ +This tab will show the selected metrics for the portfolio and allow you to compare their values for different scenarios. diff --git a/site/old_files/old_tutorials/3-whats-next.md b/site/old_files/old_tutorials/3-whats-next.md new file mode 100644 index 00000000..7c021119 --- /dev/null +++ b/site/old_files/old_tutorials/3-whats-next.md @@ -0,0 +1,12 @@ +--- +description: How to supercharge your designs and experiments with OpenDC. +--- + +# What's next? + +Congratulations! You have just learned how to design and experiment with a (virtual) datacenter in OpenDC. What's next? + +- Follow one of the [tutorials](/docs/category/tutorials) using OpenDC. +- Check the [advanced guides](/docs/category/advanced-guides) for more complex material. +- Read about [existing work using OpenDC](/community/research). +- Get involved in the [OpenDC Community](/community/support). diff --git a/site/old_files/old_tutorials/_category_.json b/site/old_files/old_tutorials/_category_.json new file mode 100644 index 00000000..169f7a27 --- /dev/null +++ b/site/old_files/old_tutorials/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Getting Started", + "position": 2, + "link": { + "type": "generated-index", + "description": "10 minutes to learn the most important concepts of OpenDC." + } +} diff --git a/site/old_tutorials/0-installation.md b/site/old_tutorials/0-installation.md deleted file mode 100644 index 281e811f..00000000 --- a/site/old_tutorials/0-installation.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -description: How to install OpenDC locally, and start experimenting in no time. ---- - -# Installation - -This page describes how to set up and configure a local single-user OpenDC installation so that you can quickly get your -experiments running. You can also use the [hosted version of OpenDC](https://app.opendc.org) to get started even -quicker (The web server is however missing some more complex features). - - -## Prerequisites - -1. **Supported Platforms** - OpenDC is actively tested on Windows, macOS and GNU/Linux. -2. **Required Software** - A Java installation of version 19 or higher is required for OpenDC. You may download the - [Java distribution from Oracle](https://www.oracle.com/java/technologies/downloads/) or use the distribution provided - by your package manager. - -## Download - -To get an OpenDC distribution, download a recent version from our [Releases](https://github.com/atlarge-research/opendc/releases) page on GitHub. -For basic usage, the OpenDCExperimentRunner is all that is needed. - -## Setup - -Unpack the downloaded OpenDC distribution. Opening OpenDCExperimentRunner results in two folders, `bin` and `lib`. -`lib` contains all `.jar` files needed to run OpenDC. `bin` two executable versions of the OpenDCExperimentRunner. -In the following pages, we discuss how to run an experiment using the executables. - diff --git a/site/old_tutorials/1-design.mdx b/site/old_tutorials/1-design.mdx deleted file mode 100644 index e8ab2c58..00000000 --- a/site/old_tutorials/1-design.mdx +++ /dev/null @@ -1,154 +0,0 @@ ---- -description: How to design a virtual datacenter in OpenDC from scratch. ---- - -# Design a Datacenter - -Now that you have installed OpenDC (or are using the hosted version), we will start designing a (virtual) datacenter -in OpenDC. - -## Before we start - -There are a couple of steps we need to perform before we can start designing a datacenter in OpenDC. First, we need to -enter the OpenDC web application. This done as follows: - -
-
-
-
-
-

Hosted Deployment

- - To enter the hosted version of OpenDC, you need a user account. User management is provided - by Auth0, which allows you to login with social accounts or via - email. - -
- -
-
-
-
-
-

Local Deployment

- - The local distribution of OpenDC runs in single-user mode by default, which does not require - authentication. This allows you to quickly start designing and experimenting with new - datacenters. - -
- -
-
-
-
- -### Create a Project - -Next, we need to create a new project. Projects allow you to organize your designs and experiments together. -Click on ‘+ New Project’ in the right corner to open the project creation dialog. -Give your project a name and save it. You can now open it by clicking on it in the project table. If all went well, -you’re redirected to your new project, and are presented with an empty project overview. - -### Create a Topology - -In OpenDC, the datacenter design is also called a **topology**. This topology represents the physical layout of a -datacenter and specifies everything from the architectural layout of the datacenter’s rooms to which CPUs are in a -particular machine. - -To create a design (topology), click on ‘+ New Topology’ in the top right corner of the topology table. -Once you have created the topology, it will appear the topology table. By clicking on the topology, you will be -redirected to a (still empty) overview of the topology. From here, we'll start designing a datacenter. - -### Terminology - -Here’s an overview of some of the language you’ll find when designing a datacenter in OpenDC: - -- **Topology**: the physical layout of your datacenter -- **Room**: a room in the datacenter -- **Tile**: one of the tiles that forms a room -- **Rack**: a rack of servers that stands on top of a tile -- **Machine**: a machine that takes up a single slot in a server rack, containing several components such as CPUs, GPUs, - network interfaces and storage drives. - -## Build the datacenter - -Open the project and topology that you have created and click on the 'Floor Plan' tab (see [Figure 1](#floor-plan)). -We’re now in datacenter construction mode. Notice the grid on the canvas? That’s where you’ll place tiles, in order to -build rooms. Let’s take a moment to familiarize ourselves with the interface. - -If you dismiss the sidebar on your left, you have controls for zooming in and out. Next to the zooming buttons, you also -have a ‘Screenshot’ button, in case you want to record the state of the canvas and export it to an image file. On the -right side of the screen, you have the simHyperVisorContext menu. This menu changes depending on your zoom level. - -As there are currently no rooms, we are in ‘Building’ mode, and our only option is to ‘Construct a new room’. Click on -that button to build a first datacenter room - once you’ve clicked on it, every tile of the canvas that you click on -becomes a tile of that room. There is one restriction though: Each tile that you add must be adjacent to any previous -tiles that you have added. You can see for yourself which tile positions are clickable through the highlight color that -is shown on hovering over them. - -
- Analysis of results reported by OpenDC -
The floor plan of a (virtual) datacenter in OpenDC.
-
- -### Create a Room - -:::note Action - -Create at least a single room, with help of the above instructions. - -::: - -Once you’ve placed the tiles, you can give the room a name, if you want to. To do this, click on the room you want to -edit. You’ll notice the application going into ‘Room’ mode, allowing you to manipulate the topology of the datacenter at -a more fine-grained level. In the simHyperVisorContext menu, change the room name, and click on the ‘Save’ button. You can exit -‘Room’ mode by clicking on any of the darkened areas outside of the selected room. This will bring you back to -‘Building’ mode. - -### Place Server Racks - -:::note Action - -Add at least a single rack in the room. - -::: - -Empty rooms are of no use to the stakeholders of a datacenter. They want machines! Let’s place some racks in the room -to fulfill this demand. Click on the room and add some racks. To stop adding racks, click on the blue element in the -sidebar, again. - -### Fill the Racks with Servers - -:::note Action - -Add a couple of servers to the rack. - -::: - -To add actual servers to the empty racks, we’ll need to go one level deeper in the topological hierarchy of the -datacenter. Clicking on a rack lets you do just that. Once you’ve clicked on it, you’ll notice the simHyperVisorContext menu now -displaying slots. In each slot fits exactly one server unit. To add such a server unit, click on the ‘Add machine’ -button of that slot. -Just like in ‘Room’ mode, you can exit ‘Rack’ mode by clicking on any of the darkened tiles around the currently -selected rack. - -### Add Resources to the Servers - -We’re almost done creating our datacenter! The only problem we have is that the machines / servers we just added lack -any real resources (such as CPUs, GPUs, memory cards, and disk storage). - -:::note Action - -Populate the machines with CPU and memory resources. - -::: - -To do this, click on any machine you want to edit. Notice the simHyperVisorContext menu changing, with tabs to add different kinds of -units to your machine. Have a look around as to what can be added. - -Once you are satisfied with the datacenter design, we will experiment with the design in the next chapter. diff --git a/site/old_tutorials/2-experiment.mdx b/site/old_tutorials/2-experiment.mdx deleted file mode 100644 index 14970ea6..00000000 --- a/site/old_tutorials/2-experiment.mdx +++ /dev/null @@ -1,74 +0,0 @@ ---- -description: How to experiment with your datacenter designs. ---- - -# Create an Experiment - -After designing a datacenter in OpenDC, the next step is to experiment with the design using OpenDC's built-in -simulator, in order to analyze its performance and compare it against other designs. - -In OpenDC, we use the concept of portfolios of scenarios to experiment with datacenter designs. In the next sections, we -will explain how you can use these powerful concepts for the experimental analysis of your designs. - -## Create a Portfolio -OpenDC organizes multiple scenarios (experiments) into a **portfolio**. Each portfolio is composed of a base scenario, -a set of candidate scenarios given by the user and a set of targets (e.g., metrics) used to compare scenarios. - -To create a new portfolio, open a project in the OpenDC web interface and click on ‘+ New Portfolio’ in the top right -corner of the portfolio table. This opens a modal with the following options: - -1. **Name**: the name of your portfolio. -2. **Metrics**: the metrics that you want to use to compare the scenarios in the portfolio. -3. **Repeats per Scenario**: the number of times each scenario should be simulated (to account for variance). - -## Create Scenarios - -A **scenario** represents a point in the datacenter design space that should be explored. It consists of a combination -of workload, topology, and a set of operational phenomena. Phenomena can include correlated failures, performance -variability, security breaches, etc., allowing the scenarios to more accurately capture the real-world operations. - -The baseline for comparison in a portfolio is the **base scenario**. It represents the status quo of the infrastructure -or, when planning infrastructure from scratch, it consists of very simple base workloads and topologies. -The other scenarios in a portfolio, called the **candidate scenarios**, represent changes to the configuration -that might be of interest to the datacenter designer. Dividing scenarios into these two categories ensures that any -comparative insights provided by OpenDC are meaningful within the simHyperVisorContext of the current architecture. - -To create a new scenario, open a portfolio in the OpenDC web interface and click on ‘+ New Scenario’ in the top right -corner of the scenario table. This opens a modal with the following options (as shown in [Figure 1](#explore)): - -1. **Name**: the name of your scenario. The first scenario of a portfolio is always called the _Base scenario_. -2. **Workload**: the applications (e.g., virtual machines, workflows, functions) that consume resources in your - datacenter. - 1. **Workload Trace**: A dataset that characterizes the historical runtime behavior of virtual machines in the - workload over time. - 2. **Load Sampling Fraction**: The percentage of the workload that should be simulated (e.g., 10% of virtual - machines in the workload trace). -3. **Environment**: - 1. **Topology**: one of the topologies of the project to use for the scenario. - 2. **Scheduler**: the algorithm that decides on which hosts the virtual machines should be placed. -4. **Operational Phenomena**: - 1. **Failures**: a flag to enable stochastic host failures during simulation. - 2. **Performance interference**: a flag to enable performance interference between virtual machines (only available - for a subset of traces). - -Once you have created the scenario, it will be enqueued for simulation. Usually the results of the simulation should be -available within one minute after creation. However, if there are lots of queued simulation jobs, it might take a bit -longer. - -
- Creating a new scenario in OpenDC -
Creating a new scenario in OpenDC. The user can select the topology, workload, and operational phenomena.
-
- -## Analyze Results - -After creating scenarios, the scenario table will show whether the simulation is still pending, completed successfully, -or failed for some reason. If the scenario was simulated successfully, its results will become visible on the ‘Results’ -tab as shown in [Figure 2](#analysis). - -
- Analysis of results reported by OpenDC -
Plots and visual summaries generated by OpenDC comparing different scenarios.
-
- -This tab will show the selected metrics for the portfolio and allow you to compare their values for different scenarios. diff --git a/site/old_tutorials/3-whats-next.md b/site/old_tutorials/3-whats-next.md deleted file mode 100644 index 7c021119..00000000 --- a/site/old_tutorials/3-whats-next.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -description: How to supercharge your designs and experiments with OpenDC. ---- - -# What's next? - -Congratulations! You have just learned how to design and experiment with a (virtual) datacenter in OpenDC. What's next? - -- Follow one of the [tutorials](/docs/category/tutorials) using OpenDC. -- Check the [advanced guides](/docs/category/advanced-guides) for more complex material. -- Read about [existing work using OpenDC](/community/research). -- Get involved in the [OpenDC Community](/community/support). diff --git a/site/old_tutorials/_category_.json b/site/old_tutorials/_category_.json deleted file mode 100644 index 169f7a27..00000000 --- a/site/old_tutorials/_category_.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "label": "Getting Started", - "position": 2, - "link": { - "type": "generated-index", - "description": "10 minutes to learn the most important concepts of OpenDC." - } -} -- cgit v1.2.3