diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 74 |
1 files changed, 74 insertions, 0 deletions
@@ -31,6 +31,80 @@ https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc By default, the Kafka topic is named `postgres_topic`. +The scripts to start the Confluent Kafka (assuming installation in `opt/confluent`) instance are listed below. First, start Kafka Connect: + +``` +/opt/confluent/bin/connect-standalone \ +/opt/confluent/etc/kafka/connect-standalone.properties \ +/opt/confluent/share/confluent-common/connectors/sink-jdbc.properties \ +/opt/confluent/share/confluent-common/connectors/sink-redis.properties +``` + +Then, format Kafka Storage: + +``` +/opt/confluent/bin/kafka-storage format -t \ +2vi2WtHxQAOPyXb1Bj1Jvw -c /opt/confluent/etc/kafka/server.properties \ +--standalone > /dev/null 2>&1; echo 0 +``` + +Afterwards, start Kafka Broker: + +``` +/opt/confluent/bin/kafka-server-start /opt/confluent/etc/kafka/server.properties +``` + +Then create a new Kafka Topic (_e.g.,_ "posgres-topic"): + +``` +kafka-topics --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 +``` + +Lastly, run the Schema Registry: +``` +/opt/confluent/bin/schema-registry-start \ +/opt/confluent/etc/schema-registry/schema-registry.properties +``` + +At this point, ensure PostgreSQL, Redis and Grafana are up and running on the system, on their default port configurations. +To see metrics flow seamlessly into PostgreSQL, once can run the following command (_i.e.,_ assuming the database name is "opendc"): + +``` +sudo su postgres; psql -d opendc +``` + +Additionally, to interact with the Redis cache, we suggest to use the excellent redis-cli tool: + +``` +redis-cli -h localhost -p 6379 +``` + +The following commands can be used to list the contents of the cache (_i.e.,_ assuming stream name "postgres_topic"), and to clear the cache respectively: + +``` +XRANGE postgres_topic - + +XTRIM postgres_topic MAXLEN 0 +``` + +For the predictive analytics in our implementation it is useful to create a separate consumer group: + +``` +XGROUP CREATE mystream mygroup 0 +``` + +Lastly, each time you change the database schema, you must run (_i.e.,_ assuming `schema.proto` lives in `resources/experiments`): + +``` +cd resources/experiments/; +protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/ schema.proto +``` + +_Sunfish_ acts as the digital twin and the modified OpenDC (this repository) acts as the physical datacenter. +Both contain modifications. +Additionally: create a `.venv` in `python_scripts` and `http_server` and install all the dependencies. +Then, run respectively the HTTP Server and the Python Scripts. +Afterwards, start IntellijIDEA in this repository and run some experiments. + ## Useful commands To access a certain PostgreSQL database _e.g.,_ called "opendc": |