diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 138 |
1 files changed, 114 insertions, 24 deletions
@@ -1,39 +1,129 @@ -<a href="https://opendc.org/"> - <img src="https://opendc.org/img/logo.svg" alt="OpenDC logo" title="OpenDC" align="right" height="100" /> -</a> +## _Sunfish_: Enabling Predictive Analytics for Datacenters Through Digital Twinning -# OpenDC +_Sunfish_ is a system that creates the pipeline between the Datacenter and its Digital Twin. -Collaborative Datacenter Simulation and Exploration for Everybody +## Dependencies -[](/LICENSE.txt) -[](https://opendc.org/learn/category/documentation) -[](https://github.com/atlarge-research/opendc/releases) -[](https://github.com/atlarge-research/opendc/actions/workflows/build.yml) +_Sunfish_ is not a monolithic program. The entire ecosystem consists of: ------ +1. [OpenDC](https://opendc.org) version 2.4 +2. [Confluent Kafka](https://docs.confluent.io/kafka/overview.html) confluent version v4.39.0 +3. [PostgreSQL](https://www.postgresql.org/) version 18.4 +4. [Redis](https://redis.io/) version 9.1.1 +5. [Grafana](https://grafana.com/) version 13.1.0 -This repository is the home of the OpenDC project, a free and open-source platform for cloud datacenter simulation. +Be aware the project will only work with this OpenDC (_i.e.,_ in this repository) as there are hidden Maven dependencies. -## Latest Release +## Schema Registry +The Confluent ecosystem contains the Schema Registry, which enables seamless schema of the data interchange formats. +_Sunfish_ uses ProtoBuf (see `schema.proto`) file. +Each change to the `schema.proto` file must be updated with: +```bash +cd resources/experiments/ +protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/ schema.proto +curl -X DELETE http://localhost:8081/subjects/postgres-topic-value +``` -All releases of OpenDC can be found [here](https://github.com/atlarge-research/opendc/releases). -The latest release will be marked _latest_. +## Confluent Configuration +First, see https://www.confluent.io/installation/. +Confluent JDBC sink and source (includes Postgres connector) files: +https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc -## Documentation +By default, the Kafka topic is named `postgres_topic`. -You can find the OpenDC documentation [on the website](https://opendc.org). -The documentation is divided into several sections: +The scripts to start the Confluent Kafka (assuming installation in `opt/confluent`) instance are listed below. First, start Kafka Connect: -* [Getting Started](https://opendc.org/learn/category/getting-started) -* [Tutorials](https://opendc.org/learn/category/tutorials) -* [Where to Get Support](https://opendc.org/community/support) -* [Contributing Guide](https://opendc.org/community/contributing) +``` +/opt/confluent/bin/connect-standalone \ +/opt/confluent/etc/kafka/connect-standalone.properties \ +/opt/confluent/share/confluent-common/connectors/sink-jdbc.properties \ +/opt/confluent/share/confluent-common/connectors/sink-redis.properties +``` -You find information about the research projects and publications related to OpenDC in the [research section](https://opendc.org/research/publications). +Then, format Kafka Storage: -The source code for the website is located in the [gh-pages branch](https://github.com/atlarge-research/opendc/tree/gh-pages). -Push to that branch to update the website. +``` +/opt/confluent/bin/kafka-storage format -t \ +2vi2WtHxQAOPyXb1Bj1Jvw -c /opt/confluent/etc/kafka/server.properties \ +--standalone > /dev/null 2>&1; echo 0 +``` + +Afterwards, start Kafka Broker: + +``` +/opt/confluent/bin/kafka-server-start /opt/confluent/etc/kafka/server.properties +``` + +Then create a new Kafka Topic (_e.g.,_ "posgres-topic"): + +``` +kafka-topics --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 +``` + +Lastly, run the Schema Registry: +``` +/opt/confluent/bin/schema-registry-start \ +/opt/confluent/etc/schema-registry/schema-registry.properties +``` + +At this point, ensure PostgreSQL, Redis and Grafana are up and running on the system, on their default port configurations. +To see metrics flow seamlessly into PostgreSQL, once can run the following command (_i.e.,_ assuming the database name is "opendc"): + +``` +sudo su postgres; psql -d opendc +``` + +Additionally, to interact with the Redis cache, we suggest to use the excellent redis-cli tool: + +``` +redis-cli -h localhost -p 6379 +``` + +The following commands can be used to list the contents of the cache (_i.e.,_ assuming stream name "postgres_topic"), and to clear the cache respectively: + +``` +XRANGE postgres_topic - + +XTRIM postgres_topic MAXLEN 0 +``` + +For the predictive analytics in our implementation it is useful to create a separate consumer group: + +``` +XGROUP CREATE mystream mygroup 0 +``` + +Lastly, each time you change the database schema, you must run (_i.e.,_ assuming `schema.proto` lives in `resources/experiments`): + +``` +cd resources/experiments/; +protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/ schema.proto +``` + +_Sunfish_ acts as the digital twin and the modified OpenDC (this repository) acts as the physical datacenter. +Both contain modifications. +Additionally: create a `.venv` in `python_scripts` and `http_server` and install all the dependencies. +Then, run respectively the HTTP Server and the Python Scripts. +Afterwards, start IntellijIDEA in this repository and run some experiments. + +## Useful commands + +To access a certain PostgreSQL database _e.g.,_ called "opendc": +`psql -d opendc` + +This command selects everything from the topic which Kafka writes to. +`SELECT * FROM postgres_topic;` + +Connect to Redis server via an excellent command-line tool: +`redis-cli -h localhost -p 6379` +Very useful tutorial https://redis.io/docs/latest/develop/data-types/streams/ +See also https://redis.io/docs/latest/develop/tools/cli/ + +The below command returns all items in the Redis stream +`XRANGE postgres_topic - +` +`XTRIM postgres_topic MAXLEN 0` + +To create a group for consumers to read from stream: +`XGROUP CREATE mystream mygroup 0` ## Contributing |