Revamp web server instructions

8 files changed, 62 insertions, 98 deletions

diff --git a/README.md b/README.md
index eed60765..14156eff 100644
--- a/README.md
+++ b/README.md
@@ -23,7 +23,7 @@ OpenDC is a project by the [@Large Research Group](http://atlarge-research.com).
 
 ## Architecture
 
-OpenDC consists of four components: a Kotlin simulator, a MongoDB database, a Python Flask web server, and a React.js frontend.
+OpenDC consists of four components: a Kotlin [simulator](/simulator), a MongoDB database, a Python Flask [web server](/web-server), and a React.js [frontend](/frontend), each in their own subdirectories.
 
 <p align="center">
     <img src="misc/artwork/opendc-component-diagram.png" alt="OpenDC Component Diagram">
diff --git a/web-server/README.md b/web-server/README.md
index 6c07c29f..ff50cb36 100644
--- a/web-server/README.md
+++ b/web-server/README.md
@@ -1,12 +1,14 @@
 <h1 align="center">
-  <img src="../misc/artwork/logo.png" width="100" alt="OpenDC">
-  <br>
-  OpenDC Web Server
+    <img src="../misc/artwork/logo.png" width="100" alt="OpenDC">
+    <br>
+    OpenDC Web Server
 </h1>
 <p align="center">
-Collaborative Datacenter Simulation and Exploration for Everybody
+    Collaborative Datacenter Simulation and Exploration for Everybody
 </p>
 
+<br>
+
 The OpenDC web server is the bridge between OpenDC's frontend and database. It is built with Flask/SocketIO in Python and implements the OpenAPI-compliant [OpenDC API specification](../opendc-api-spec.yml).
 
 This document explains a high-level view of the web server architecture ([jump](#architecture)), and describes how to set up the web server for local development ([jump](#setup-for-local-development)).
@@ -23,107 +25,69 @@ The OpenDC API is implemented by the `Main Server Loop`, which is the only compo
 
 The `Util` package handles several miscellaneous tasks:
 
-* `REST`: Parses SocketIO messages into `Request` objects, and calls the appropriate `API` endpoint to get a `Response` object to return to the `Main Server Loop`.
-* `Param Checker`: Recursively checks whether required `Request` parameters are present and correctly typed.
+* `Database API`: Wraps database access functionality used by `Models` to read themselves from/write themselves into the database.
 * `Exceptions`: Holds definitions for exceptions used throughout the web server.
-* `Database API`: Wraps SQLite functionality used by `Models` to read themselves from/ write themselves into the database.
+* `Parameter Checker`: Recursively checks whether required `Request` parameters are present and correctly typed.
+* `REST`: Parses SocketIO and HTTP messages into `Request` objects, and calls the appropriate `API` endpoint to get a `Response` object to return to the `Main Server Loop`.
 
 ### API Package
 
-The `API` package contains the logic for the HTTP methods in each API endpoint. Packages are structured to mirror the API: the code for the endpoint `GET simulations/authorizations`, for example, would be located at the `Endpoint` inside the `authorizations` package, inside the `simulations` package (so at `api/simulations/authorizations/endpoint.py`).
+The `API` package contains the logic for the HTTP methods in each API endpoint. Packages are structured to mirror the API: the code for the endpoint `GET api/simulations`, for example, would be located at the `endpoint.py` inside the `simulations` package (so at `api/simulations/endpoint.py`).
 
-An `Endpoint` contains methods for each HTTP method it supports, which takes a request as input (such as `def GET(request):`). Typically, such a method checks whether the parameters were passed correctly (using the `Param Checker`); fetches some model from the database; checks whether the data exists and is accessible by the user who made the request; possibly modifies this data and writes it back to the database; and returns a JSON representation of the model.
+An `endpoint.py` file contains methods for each HTTP method it supports, which takes a request as input (such as `def GET(request):`). Typically, such a method checks whether the parameters were passed correctly (using the `Parameter Checker`); fetches some model from the database; checks whether the data exists and is accessible by the user who made the request; possibly modifies this data and writes it back to the database; and returns a JSON representation of the model.
 
-The `REST` component dynamically imports the appropriate method from the appropriate `Endpoint`, according to request it receives, and executes it.
+The `REST` component dynamically imports the appropriate method from the appropriate `endpoint`, according to request it receives, and executes it.
 
 ### Models Package
 
-The `Models` package contains the logic for mapping Python objects to their database representations. This involves an abstract `model` which has methods to `read`, `insert`, `update` and `delete` objects. Extensions of `model`, such as a `User` or `Simulation`, specify some metadata such as their tabular representation in the database and how they map to a JSON object, which the code in `model` uses in the database interaction methods.
+The `models` package contains the logic for mapping Python objects to their database representations. This involves an abstract `model` which has generic CRUD operations. Extensions of `model`, such as a `User` or `Simulation`, specify some more specific operations and their collection metadata.
 
-`Endpoint`s import these `Models` and use them to execute requests.
+`Endpoint`s import these `models` and use them to execute requests.
 
 ## Setup for Local Development
 
-The following steps will guide you through setting up the OpenDC web server locally for development. To test individual endpoints, edit `static/index.html`. This guide was tested and developed on Windows 10.
+The following steps will guide you through setting up the OpenDC web server locally for development. To test individual endpoints, edit `static/index.html`.
 
 ### Local Setup
 
 #### Install requirements
 
-Make sure you have Python 2.7 installed (if not, get it [here](https://www.python.org/)), as well as pip (if not, get it [here](https://pip.pypa.io/en/stable/installing/)). Then run the following to install the requirements.
+Make sure you have Python 3.7+ installed (if not, get it [here](https://www.python.org/)), as well as pip (if not, get it [here](https://pip.pypa.io/en/stable/installing/)). Then run the following to install the requirements.
 
 ```bash
-python setup.py install
+pip install -r requirements.txt
 ```
 
-The web server also requires MariaDB >= 10.1. Instructions to install MariaDB can be found [here](https://mariadb.com/kb/en/mariadb/getting-installing-and-upgrading-mariadb/). The Docker image can be found [here](https://hub.docker.com/_/mariadb/). 
-
-#### Get the code
-
-Clone both this repository and the main OpenDC repository, from the same base directory.
-
-```bash
-git clone https://github.com/atlarge-research/opendc-web-server.git
-git clone https://github.com/atlarge-research/opendc.git
-```
-
-#### Set up the database
-
-The database can be rebuilt by using the `schema.sql` file from main opendc repository. 
+The web server also requires a running MongoDB instance. We recommend setting this up through docker, by running `docker-compose build` and `docker-compose up` in the [`mongodb` directory](../mongodb) of the main OpenDC repository.
 
-#### Configure OpenDC
+#### Get and configure the code
 
-Create a file `config.json` in `opendc-web-server`, containing:
+Clone OpenDC and follow the [instructions in the main repository](../) to set up a Google OAuth ID and environment variables.
 
-```json
-{
-	"ROOT_DIR": "BASE_DIRECTORY",
-	"OAUTH_CLIENT_ID": "OAUTH_CLIENT_ID",
-	"FLASK_SECRET": "FLASK_SECRET",
-	"MYSQL_DATABASE": "opendc",
-	"MYSQL_USER": "opendc",
-	"MYSQL_PASSWORD": "opendcpassword",
-	"MYSQL_HOST": "127.0.0.1",
-	"MYSQL_PORT": 3306
-}
-```
-
-Make the following replacements:
-* Replace `BASE_DIRECTORY` with the base directory in which you cloned `opendc` and `opendc-web-server`.
-* Replace `OAUTH_CLIENT_ID` with your OAuth client ID (see the [OpenDC README](https://github.com/atlarge-research/opendc#preamble)).
-* Replace `FLASK_SECRET`, come up with some string.
-* Replace the `MYSQL_*` variables with the correct settings for accessing the MariaDB database that was just created. 
+**Important:** Be sure to set up environment variables according to those instructions, in a `.env` file.
 
 In `opendc-web-server/static/index.html`, add your own `OAUTH_CLIENT_ID` in `content=` on line `2`.
 
-#### Set up Postman and OpenDC account
-
-To easily make HTTP requests to the web server, we recommend Postman (get it [here](https://www.getpostman.com/)).
-
-Once Postman is installed and set up, `Import` the OpenDC requests collection (`OpenDC.postman_collection.json`). In the `Collections` tab, expand `OpenDC` and click `Create New User`. This should open the request in the `Builder` pane.
-
-Navigate to `http://localhost:8081/my-auth-token` and copy the authentication token on this page to your clipboard. In the Postman `Builder` pane, navigate to the `Headers (2)` tab, and paste the authentication token as value for the `auth-token` header. (This token expires every hour - refresh the auth token page to get a new token.) 
-
-(Optional: navigate to the `Body` tab and change the email address to the gmail address you used to get an authentication token.)
+#### Set up the database
 
-Click `Send` in Postman to send your request and see the server's response. If it's a `200`, your account is set up!
+Run `docker-compose build` and `docker-compose up` in the [`mongodb` directory](../mongodb) of the main OpenDC repository to get a database setup up and running.
 
 ### Local Development
 
 Run the server.
 
 ```bash
-cd opendc-web-server
-python main.py config.json
+cd web-server
+python main.py
 ```
 
-To try a different query, use the Postman `Builder` to edit the method, path, body, query parameters, etc. `Create New Simulation` is provided as an additional example.
-
-When editing the web server code, restart the server (`CTRL` + `c` followed by `python main.py config.json` in the console running the server) to see the result of your changes.
+When editing the web server code, restart the server (`CTRL` + `c` followed by `python main.py` in the console running the server) to see the result of your changes.
 
 #### Code Style
 
-To format all files, run `format.sh` in this directory.
+To format all files, run `format.sh` in this directory (uses `yapf` internally).
+
+To check if code style is up to modern standards, run `check.sh` in this directory (uses `pylint` internally).
 
 #### Testing
 
diff --git a/web-server/conftest.py b/web-server/conftest.py
index 05172b04..1f4831b8 100644
--- a/web-server/conftest.py
+++ b/web-server/conftest.py
@@ -1,3 +1,6 @@
+"""
+Configuration file for all unit tests.
+"""
 import pytest
 
 from main import FLASK_CORE_APP
diff --git a/web-server/main.py b/web-server/main.py
index 94696e8d..412bf3f9 100644
--- a/web-server/main.py
+++ b/web-server/main.py
@@ -8,11 +8,14 @@ from flask import Flask, request, send_from_directory, jsonify
 from flask_compress import Compress
 from oauth2client import client, crypt
 from flask_cors import CORS
+from dotenv import load_dotenv
 
 from opendc.models.user import User
 from opendc.util import rest, path_parser, database
 from opendc.util.exceptions import AuthorizationTokenError, RequestInitializationError
 
+load_dotenv()
+
 TEST_MODE = "OPENDC_FLASK_TESTING" in os.environ
 
 # Specify the directory of static assets
@@ -190,4 +193,5 @@ def _process_message(message):
 
 
 if __name__ == '__main__':
+    print("Web server started on 8081")
     SOCKET_IO_CORE.run(FLASK_CORE_APP, host='0.0.0.0', port=8081)
diff --git a/web-server/opendc/api/v2/topologies/topologyId/endpoint.py b/web-server/opendc/api/v2/topologies/topologyId/endpoint.py
index 824bb12b..a4f71ed6 100644
--- a/web-server/opendc/api/v2/topologies/topologyId/endpoint.py
+++ b/web-server/opendc/api/v2/topologies/topologyId/endpoint.py
@@ -6,7 +6,6 @@ from opendc.models.topology import Topology
 from opendc.util.rest import Response
 
 
-
 def GET(request):
     """Get this Topology."""
 
@@ -19,15 +18,10 @@ def GET(request):
 
     return Response(200, 'Successfully retrieved topology.', topology.obj)
 
+
 def PUT(request):
     """Update this topology"""
-    request.check_required_parameters(path={'topologyId': 'int'},
-                                      body={
-                                          'topology': {
-                                              'name': 'string',
-                                              'rooms': {}
-                                          }
-                                      })
+    request.check_required_parameters(path={'topologyId': 'int'}, body={'topology': {'name': 'string', 'rooms': {}}})
     topology = Topology.from_id(request.params_path['topologyId'])
 
     topology.check_exists()
@@ -41,6 +35,7 @@ def PUT(request):
 
     return Response(200, 'Successfully updated topology.', topology.obj)
 
+
 def DELETE(request):
     """Delete this topology"""
     request.check_required_parameters(path={'topologyId': 'int'})
diff --git a/web-server/opendc/api/v2/topologies/topologyId/test_endpoint.py b/web-server/opendc/api/v2/topologies/topologyId/test_endpoint.py
index ac0d02aa..fa88c497 100644
--- a/web-server/opendc/api/v2/topologies/topologyId/test_endpoint.py
+++ b/web-server/opendc/api/v2/topologies/topologyId/test_endpoint.py
@@ -49,18 +49,15 @@ def test_get_topology_no_authorizations(client, mocker):
 PUT /topologies/{topologyId}
 '''
 
+
 def test_update_topology_missing_parameter(client):
     assert '400' in client.put('/api/v2/topologies/1').status
 
+
 def test_update_topology_non_existent(client, mocker):
     mocker.patch.object(DB, 'fetch_one', return_value=None)
-    assert '404' in client.put('/api/v2/topologies/1',
-                                json={
-                                    'topology': {
-                                        'name': 'test_topology',
-                                        'rooms': {}
-                                    }
-                                }).status
+    assert '404' in client.put('/api/v2/topologies/1', json={'topology': {'name': 'test_topology', 'rooms': {}}}).status
+
 
 def test_update_topology_not_authorized(client, mocker):
     mocker.patch.object(DB,
@@ -74,13 +71,13 @@ def test_update_topology_not_authorized(client, mocker):
                             }]
                         })
     mocker.patch.object(DB, 'update', return_value={})
-    assert '403' in client.put('/api/v2/topologies/1',
-                               json={
-                                   'topology': {
-                                       'name': 'updated_topology',
-                                       'rooms': {}
-                                   }
-                               }).status
+    assert '403' in client.put('/api/v2/topologies/1', json={
+        'topology': {
+            'name': 'updated_topology',
+            'rooms': {}
+        }
+    }).status
+
 
 def test_update_topology(client, mocker):
     mocker.patch.object(DB,
@@ -95,19 +92,19 @@ def test_update_topology(client, mocker):
                         })
     mocker.patch.object(DB, 'update', return_value={})
 
-    assert '200' in client.put('/api/v2/topologies/1',
-                               json={
-                                   'topology': {
-                                       'name': 'updated_topology',
-                                       'rooms': {}
-                                   }
-                               }).status
+    assert '200' in client.put('/api/v2/topologies/1', json={
+        'topology': {
+            'name': 'updated_topology',
+            'rooms': {}
+        }
+    }).status
 
 
 '''
 DELETE /topologies/{topologyId}
 '''
 
+
 def test_delete_topology(client, mocker):
     mocker.patch.object(DB,
                         'fetch_one',
@@ -126,7 +123,7 @@ def test_delete_topology(client, mocker):
     res = client.delete('/api/v2/topologies/1')
     assert '200' in res.status
 
+
 def test_delete_nonexistent_topology(client, mocker):
     mocker.patch.object(DB, 'fetch_one', return_value=None)
     assert '404' in client.delete('/api/v2/topologies/1').status
-
diff --git a/web-server/opendc/models/simulation.py b/web-server/opendc/models/simulation.py
index dbe1e800..86aa4726 100644
--- a/web-server/opendc/models/simulation.py
+++ b/web-server/opendc/models/simulation.py
@@ -17,8 +17,8 @@ class Simulation(Model):
         :param edit_access: True when edit access should be checked, otherwise view access.
         """
         user = User.from_google_id(google_id)
-        authorizations = list(
-            filter(lambda x: str(x['simulationId']) == str(self.get_id()), user.obj['authorizations']))
+        authorizations = list(filter(lambda x: str(x['simulationId']) == str(self.get_id()),
+                                     user.obj['authorizations']))
         if len(authorizations) == 0 or (edit_access and authorizations[0]['authorizationLevel'] == 'VIEW'):
             raise ClientError(Response(403, "Forbidden from retrieving simulation."))
 
diff --git a/web-server/requirements.txt b/web-server/requirements.txt
index b95d3145..d55ccacd 100644
--- a/web-server/requirements.txt
+++ b/web-server/requirements.txt
@@ -12,3 +12,4 @@ pytest==5.4.3
 pytest-mock==3.1.1
 pytest-env==0.6.2
 pylint==2.5.3
+python-dotenv==0.13.0