blob: 63c06068d85db607a1da2410063fdca900705590 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
|
## _Sunfish_: Enabling Predictive Analytics for Datacenters Through Digital Twinning
_Sunfish_ is a system that creates the pipeline between the Datacenter and its Digital Twin.
## Dependencies
_Sunfish_ is not a monolithic program. The entire ecosystem consists of:
1. [OpenDC](https://opendc.org) version 2.4
2. [Confluent Kafka](https://docs.confluent.io/kafka/overview.html) confluent version v4.39.0
3. [PostgreSQL](https://www.postgresql.org/) version 18.4
4. [Redis](https://redis.io/) version 9.1.1
5. [Grafana](https://grafana.com/) version 13.1.0
Be aware the project will only work with this OpenDC (_i.e.,_ in this repository) as there are hidden Maven dependencies.
## Schema Registry
The Confluent ecosystem contains the Schema Registry, which enables seamless schema of the data interchange formats.
_Sunfish_ uses ProtoBuf (see `schema.proto`) file.
Each change to the `schema.proto` file must be updated with:
```bash
cd resources/experiments/
protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/ schema.proto
curl -X DELETE http://localhost:8081/subjects/postgres-topic-value
```
## Confluent Configuration
First, see https://www.confluent.io/installation/.
Confluent JDBC sink and source (includes Postgres connector) files:
https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc
By default, the Kafka topic is named `postgres_topic`.
## Useful commands
To access a certain PostgreSQL database _e.g.,_ called "opendc":
`psql -d opendc`
This command selects everything from the topic which Kafka writes to.
`SELECT * FROM postgres_topic;`
Connect to Redis server via an excellent command-line tool:
`redis-cli -h localhost -p 6379`
Very useful tutorial https://redis.io/docs/latest/develop/data-types/streams/
See also https://redis.io/docs/latest/develop/tools/cli/
The below command returns all items in the Redis stream
`XRANGE postgres_topic - +`
`XTRIM postgres_topic MAXLEN 0`
To create a group for consumers to read from stream:
`XGROUP CREATE mystream mygroup 0`
## Contributing
Questions, suggestions and contributions are welcome and appreciated!
Please refer to the [contributing guidelines](CONTRIBUTING.md) for more details.
## License
OpenDC is distributed under the MIT license. See [LICENSE.txt](/LICENSE.txt).
|
