From f37a1bce0e0b0b6ea3f488d7bdaf8bd05bffb123 Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Mon, 27 Jul 2020 19:07:56 +0200 Subject: Improve docs --- .editorconfig | 2 +- CONTRIBUTING.md | 6 ++--- README.md | 19 +++++++------- api/README.md | 12 ++++----- api/opendc/models/prefab.py | 2 ++ api/requirements.txt | 6 ++--- frontend/README.md | 28 ++++++++------------ .../app/results/PortfolioResultsComponent.js | 6 ++++- frontend/src/containers/modals/NewScenarioModal.js | 5 +++- opendc-api-spec.yml | 30 ++++++++++++---------- 10 files changed, 58 insertions(+), 58 deletions(-) diff --git a/.editorconfig b/.editorconfig index b0ec2ebd..3ccb008a 100644 --- a/.editorconfig +++ b/.editorconfig @@ -15,6 +15,6 @@ insert_final_newline = true [*.md] trim_trailing_whitespace = false -# Ensure YAML formatting is according to standard +# Ensure YAML formatting adheres to standard [*.{yml,yaml}] indent_size = 2 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2f7fc184..7ed01f8a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,11 +10,9 @@ Contact us at 📧[opendc@atlarge-research.com](mailto:opendc@atlarge-research.c ## 🐞 Want to report a bug or suggest a feature? -Please file an issue! First, have a look if someone has already filed an issue addressing your concern. +Please file an issue! First, have a look if someone has already filed an issue addressing your concern. If there already is such an issue, feel free to comment on the issue to show your support for it, or to add additional information that might be helpful. You can also just react with a thumbs-up 👍 to the issue, to indicate that you'd be interested in its resolution. This can help us prioritize what we spend our development time on. -If there already is such an issue, feel free to comment on the issue to show your support for it, or to add additional information that might be helpful. You can also just react with a thumbs-up 👍 to the issue, to indicate that you'd be interested in its resolution. This can help us prioritize what we spend our development time on. - -If you can't find an issue that fits your problem or feature request, open a new one. Describe actual and expected behavior, and be as detailed as you can. We'll get back to you asap. +If you can't find an issue that fits your problem or feature request, open a new one. Describe actual and expected behavior, and be as detailed as you can. We'll get back to you asap! ## 💻 Want to contribute code? diff --git a/README.md b/README.md index 0ca14867..0db109b0 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,13 @@ OpenDC is an open-source simulator for datacenters aimed at both research and ed ![opendc-frontend-construction](misc/artwork/opendc-frontend-construction.png) -Users can construct datacenters (see above) and define experiments to see how these datacenters perform under different workloads and schedulers (see below). +Users can construct datacenters (see above) and define portfolios of scenarios (experiments) to see how these datacenters perform under different workloads and schedulers (see below). ![opendc-frontend-simulation](misc/artwork/opendc-frontend-simulation.png) -The simulator is accessible both as a ready-to-use website hosted by Delft University of Technology at [opendc.org](https://opendc.org), and as source code that users can run locally on their own machine. +The simulator is accessible both as a ready-to-use website hosted by us at [opendc.org](https://opendc.org), and as source code that users can run locally on their own machine, through Docker. -OpenDC is a project by the [@Large Research Group](http://atlarge-research.com). +OpenDC is a project by the [@Large Research Group](https://atlarge-research.com). ## Architecture @@ -29,13 +29,13 @@ OpenDC consists of four components: a Kotlin [simulator](/simulator), a MongoDB OpenDC Component Diagram

-On the frontend, users can construct a topology by specifying a datacenter's rooms, racks and machines, and create experiments to see how a workload trace runs on that topology. The frontend communicates with the web server over SocketIO, through a custom REST request/response layer. For example, the frontend might make a `GET` request to `/api/v1/users/{userId}`, but this request is completed via SocketIO, not plain HTTP requests. Note that the API itself can also be accessed by HTTP. +On the frontend, users can construct a topology by specifying a datacenter's rooms, racks and machines, and create scenarios to see how a workload trace runs on that topology. The frontend communicates with the web server over SocketIO, through a custom REST request/response layer. For example, the frontend might make a `GET` request to `/api/v1/users/{userId}`, but this request is completed via SocketIO, not plain HTTP requests. Note that the API itself can also be accessed by HTTP. The (Swagger/OpenAPI compliant) API spec specifies what requests the frontend can make to the web server. To view this specification, go to the [Swagger Editor](https://editor.swagger.io/) and paste in our [opendc-api-spec.yml](opendc-api-spec.yml). -The web server receives API requests and processes them in the database. When the frontend requests to run a new experiment, the web server adds it to the `experiments` table in the database and sets is `state` as `QUEUED`. +The web server receives API requests and processes them in the database. When the frontend requests to run a new scenario, the web server adds it to the `scenarios` collection in the database and sets its `state` as `QUEUED`. -The simulator monitors the database for `QUEUED` experiments, and simulates them as they are submitted. It writes the resulting `machine_states` to the database, which the frontend can then again retrieve via the web server. +The simulator monitors the database for `QUEUED` scenarios, and simulates them as they are submitted. It writes the results to parquet files, which in turn are analysed and aggregated by a Spark pipeline. This pipeline then write the aggregated summary to the database, which the frontend can then again retrieve via the web server. ## Setup @@ -69,7 +69,7 @@ git clone https://github.com/atlarge-research/opendc.git cd opendc/ ``` -In the directory you just entered, you need to set up a set of environment variables. To do this, create a file called `.env` in the `opendc` folder. In this file, replace `your-google-oauth-client-id` with your `client_id` from the OAuth client ID you created and put the path of this `opendc` folder in the place of `/your/path/to/opendc`. For a standard setup, you can leave the other settings as-is. +In the directory you just entered, you need to set up a set of environment variables. To do this, create a file called `.env` in the `opendc` folder. In this file, replace `your-google-oauth-client-id` with your `client_id` from the OAuth client ID you created and put the path of this `opendc` folder where this placeholder is currently listed: `/your/path/to/opendc`. For a standard setup, you can leave the other settings as-is. ```.env MONGO_INITDB_ROOT_USERNAME=root @@ -84,8 +84,7 @@ OPENDC_ROOT_DIR=/your/path/to/opendc OPENDC_SERVER_BASE_URL=http://localhost:8081 ``` -Afterwards, you should also create a `traces/` directory in which you place the VM and workflow traces you want to -experiment with. +We provide a list of default traces for you to experiment with. If you want to add others, place them in the `traces` directory and add entries to the database (see also [the database folder](database/mongo-init-opendc-db.sh)) If you plan to publicly deploy, please also tweak the other settings. In that case, also check the `docker-compose.yml` for further instructions. @@ -99,4 +98,4 @@ docker-compose build docker-compose up ``` -Wait a few seconds and open `http://localhost:8081` in your browser to use OpenDC. +Wait a few seconds and open `http://localhost:8081` in your browser to use OpenDC. We recommend Google Chrome for the best development experience. diff --git a/api/README.md b/api/README.md index 4e8110d0..182cd803 100644 --- a/api/README.md +++ b/api/README.md @@ -66,18 +66,18 @@ Clone OpenDC and follow the [instructions in the main repository](../) to set up **Important:** Be sure to set up environment variables according to those instructions, in a `.env` file. -In `api/static/index.html`, add your own `OAUTH_CLIENT_ID` in `content=` on line `2`. +If you want to test REST calls manually, add your own `OAUTH_CLIENT_ID` in `content=` on line `2` in `api/static/index.html`. #### Set up the database -You can selectively run only the database services from the standard OpenDC `docker-compose` setup: +You can selectively run only the database services from the standard OpenDC `docker-compose` setup (in the root directory): ```bash docker-compose build mongo mongo-express docker-compose up mongo mongo-express ``` -This will set you up with a running MongoDB instance and a visual inspection tool running on [localhost:8082](http://localhost:8082), with which you can view and manipulate the database. +This will set you up with a running MongoDB instance and a visual inspection tool running on [localhost:8082](http://localhost:8082), with which you can view and manipulate the database. Add the simulator images to the command lists above if you want to test simulation capabilities, as well. ### Local Development @@ -92,10 +92,10 @@ When editing the web server code, restart the server (`CTRL` + `c` followed by ` #### Code Style -To format all files, run `format.sh` in this directory (uses `yapf` internally). +To format all files, run `format.sh` in this directory. The script uses `yapf` internally to format everything automatically. -To check if code style is up to modern standards, run `check.sh` in this directory (uses `pylint` internally). +To check if code style is up to modern standards, run `check.sh` in this directory. The script uses `pylint` internally. #### Testing -Run `pytest` in this directory to run all tests. +Run `pytest opendc` in this directory to run all tests. diff --git a/api/opendc/models/prefab.py b/api/opendc/models/prefab.py index 70910c4a..edf1d4c4 100644 --- a/api/opendc/models/prefab.py +++ b/api/opendc/models/prefab.py @@ -16,6 +16,8 @@ class Prefab(Model): """ user = User.from_google_id(google_id) + # TODO(Jacob) add special handling for OpenDC-provided prefabs + #try: print(self.obj) diff --git a/api/requirements.txt b/api/requirements.txt index 140a046f..0f8c3fcf 100644 --- a/api/requirements.txt +++ b/api/requirements.txt @@ -1,5 +1,5 @@ flask==1.1.2 -flask-socketio==4.3.0 +flask-socketio==4.3.1 oauth2client==4.1.3 eventlet==0.25.2 flask-compress==1.5.0 @@ -9,7 +9,7 @@ six==1.15.0 pymongo==3.10.1 yapf==0.30.0 pytest==5.4.3 -pytest-mock==3.1.1 +pytest-mock==3.2.0 pytest-env==0.6.2 pylint==2.5.3 -python-dotenv==0.13.0 +python-dotenv==0.14.0 diff --git a/frontend/README.md b/frontend/README.md index 14b2479f..ae556b09 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -1,16 +1,10 @@

- OpenDC -
- OpenDC Frontend + OpenDC +
+ OpenDC Frontend

- Collaborative Datacenter Simulation and Exploration for Everybody -

- -

- Build Status - License: MIT - styled with prettier
+ Collaborative Datacenter Simulation and Exploration for Everybody

The user-facing component of the OpenDC stack, allowing users to build and interact with their own (virtual) datacenters. Built in *React.js* and *Redux*, with the help of `create-react-app`. @@ -32,7 +26,9 @@ yarn First, you need to have a Google OAuth client ID set up. Check the [documentation of the main OpenDC repo](https://github.com/atlarge-research/opendc) if you're not sure how to do this. Once you have such an ID, you need to set it as environment variable `REACT_APP_OAUTH_CLIENT_ID`. One way of doing this is to create an `.env` file with content `REACT_APP_OAUTH_CLIENT_ID=YOUR_ID` (`YOUR_ID` without quotes), in the root directory of this repo. -Once you've set this variable, you're ready to start the development server: +Once you've set this variable, start the OpenDC `docker-compose` setup. See the root README for instructions on this. + +Now, you're ready to start the development server: ```bash yarn start @@ -46,12 +42,10 @@ To compile everything for camera-ready deployment, use the following command: yarn build ``` -**Note:** Perhaps this goes without saying, but for any functionality beyond visiting the entry page, a server backend running in the background is necessary. The easiest way to do this is to have an OpenDC docker container running, see [the main repo](https://github.com/atlarge-research/opendc) for more information on how to do this. - ## Architecture -The codebase follows a standard React.js structure, with static assets being contained in the `public` folder, while dynamic components and their styles are contained in `src`. The app uses client-side routing (with `react-router`), meaning that the only HTML file needed to be served is a `index.html` file. +The codebase follows a standard React.js structure, with static assets being contained in the `public` folder, while dynamic components and their styles are contained in `src`. The app uses client-side routing (with `react-router`), meaning that the only HTML file needing to be served is a `index.html` file. ### Pages @@ -61,9 +55,7 @@ All pages are represented by a component in the `src/pages` directory. There are **Projects.js** - Overview of projects of the user (`/projects`) -**App.js** - Main application, with datacenter construction and simulation UI (`/projects/:projectId` and `/projects/:projectId/experiments/:experimentId`) - -**Experiments.js** - Overview of experiments of the current project (`/projects/:projectId/experiments`) +**App.js** - Main application, with datacenter construction and simulation UI (`/projects/:projectId` and `/projects/:projectId/portfolios/:portfolioId`) **Profile.js** - Profile of the current user (`/profile`) @@ -77,7 +69,7 @@ Even the canvas (the main component of the app) is built using React components, ### State Management -Almost all state is kept in a central Redux store. State is kept there in an immutable form, only to be modified through actions being dispatched. These actions are contained in the `src/actions` folder, and the reducers (managing how state is updated according to dispatched actions) are located in `src/reducers`. If you're not familiar with the Redux approach to state management, have a look at their [official documentation](http://redux.js.org/). +Almost all state is kept in a central Redux store. State is kept there in an immutable form, only to be modified through actions being dispatched. These actions are contained in the `src/actions` folder, and the reducers (managing how state is updated according to dispatched actions) are located in `src/reducers`. If you're not familiar with the Redux approach to state management, have a look at their [official documentation](https://redux.js.org/). ### API Interaction diff --git a/frontend/src/components/app/results/PortfolioResultsComponent.js b/frontend/src/components/app/results/PortfolioResultsComponent.js index 35dba603..10b189a6 100644 --- a/frontend/src/components/app/results/PortfolioResultsComponent.js +++ b/frontend/src/components/app/results/PortfolioResultsComponent.js @@ -54,7 +54,11 @@ const PortfolioResultsComponent = ({ portfolio, scenarios }) => {

{METRIC_NAMES[metric]}

- + approx(tick)} diff --git a/frontend/src/containers/modals/NewScenarioModal.js b/frontend/src/containers/modals/NewScenarioModal.js index c46d0a13..225ae321 100644 --- a/frontend/src/containers/modals/NewScenarioModal.js +++ b/frontend/src/containers/modals/NewScenarioModal.js @@ -15,7 +15,10 @@ const mapStateToProps = (state) => { return { show: state.modals.newScenarioModalVisible, currentPortfolioId: state.currentPortfolioId, - currentPortfolioScenarioIds: state.currentPortfolioId !== '-1' && state.objects.portfolio[state.currentPortfolioId] ? state.objects.portfolio[state.currentPortfolioId].scenarioIds : [], + currentPortfolioScenarioIds: + state.currentPortfolioId !== '-1' && state.objects.portfolio[state.currentPortfolioId] + ? state.objects.portfolio[state.currentPortfolioId].scenarioIds + : [], traces: Object.values(state.objects.trace), topologies, schedulers: Object.values(state.objects.scheduler), diff --git a/opendc-api-spec.yml b/opendc-api-spec.yml index 3009ab03..671d4760 100644 --- a/opendc-api-spec.yml +++ b/opendc-api-spec.yml @@ -1,6 +1,6 @@ swagger: '2.0' info: - version: 1.0.0 + version: 2.0.0 title: OpenDC API description: 'OpenDC is an open-source datacenter simulator for education, featuring real-time online collaboration, diverse simulation models, and detailed performance feedback statistics.' host: opendc.org @@ -703,19 +703,6 @@ paths: description: Prefab not found. definitions: - Prefab: - type: object - properties: - _id: - type: string - name: - type: string - datetimeCreated: - type: string - format: dateTime - datetimeLastEdited: - type: string - format: dateTime Scheduler: type: object properties: @@ -865,6 +852,8 @@ definitions: properties: state: type: string + results: + type: object trace: type: object properties: @@ -919,3 +908,16 @@ definitions: type: string authorizationLevel: type: string + Prefab: + type: object + properties: + _id: + type: string + name: + type: string + datetimeCreated: + type: string + format: dateTime + datetimeLastEdited: + type: string + format: dateTime -- cgit v1.2.3 From 8522bd8acbf694120586a9ede4036d874a0a14f6 Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Mon, 27 Jul 2020 19:10:29 +0200 Subject: Add a fixme disable rule --- api/.pylintrc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/api/.pylintrc b/api/.pylintrc index f25e4fc2..7fe24187 100644 --- a/api/.pylintrc +++ b/api/.pylintrc @@ -64,7 +64,8 @@ disable=duplicate-code, missing-module-docstring, invalid-name, bare-except, - too-few-public-methods + too-few-public-methods, + fixme # Enable the message, report, category or checker with the given id(s). You can # either give multiple identifier separated by comma (,) or put this option -- cgit v1.2.3 From ceaddeb0f6927e0f99f7b140aba6a85dac0106fb Mon Sep 17 00:00:00 2001 From: Georgios Andreadis Date: Tue, 28 Jul 2020 11:36:31 +0200 Subject: Correct typo in Flavor.kt --- .../src/main/kotlin/com/atlarge/opendc/compute/core/Flavor.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simulator/opendc/opendc-compute/src/main/kotlin/com/atlarge/opendc/compute/core/Flavor.kt b/simulator/opendc/opendc-compute/src/main/kotlin/com/atlarge/opendc/compute/core/Flavor.kt index ff5060de..a49d3abf 100644 --- a/simulator/opendc/opendc-compute/src/main/kotlin/com/atlarge/opendc/compute/core/Flavor.kt +++ b/simulator/opendc/opendc-compute/src/main/kotlin/com/atlarge/opendc/compute/core/Flavor.kt @@ -25,7 +25,7 @@ package com.atlarge.opendc.compute.core /** - * Flavors define the compute and memory capacity of [Server] instance. o put it simply, a flavor is an available + * Flavors define the compute and memory capacity of [Server] instance. To put it simply, a flavor is an available * hardware configuration for a server. It defines the size of a virtual server that can be launched. */ public data class Flavor( -- cgit v1.2.3