## _Sunfish_: Enabling Predictive Analytics for Datacenters Through Digital Twinning

_Sunfish_ is a system that creates the pipeline between the Datacenter and its Digital Twin.

## Dependencies

_Sunfish_ is not a monolithic program. The entire ecosystem consists of:

1. [OpenDC](https://opendc.org) version 2.4
2. [Confluent Kafka](https://docs.confluent.io/kafka/overview.html) confluent version v4.39.0 
3. [PostgreSQL](https://www.postgresql.org/) version 18.4
4. [Redis](https://redis.io/) version 9.1.1
5. [Grafana](https://grafana.com/) version 13.1.0

Be aware the project will only work with this OpenDC (_i.e.,_ in this repository) as there are hidden Maven dependencies.

## Schema Registry
The Confluent ecosystem contains the Schema Registry, which enables seamless schema of the data interchange formats.
_Sunfish_ uses ProtoBuf (see `schema.proto`) file.
Each change to the `schema.proto` file must be updated with:
```bash
cd resources/experiments/
protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/  schema.proto
curl -X DELETE http://localhost:8081/subjects/postgres-topic-value
```

## Confluent Configuration
First, see https://www.confluent.io/installation/.
Confluent JDBC sink and source (includes Postgres connector) files: 
https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc

By default, the Kafka topic is named `postgres_topic`.

The scripts to start the Confluent Kafka (assuming installation in `opt/confluent`) instance are listed below. First, start Kafka Connect:

```
/opt/confluent/bin/connect-standalone \
/opt/confluent/etc/kafka/connect-standalone.properties \
/opt/confluent/share/confluent-common/connectors/sink-jdbc.properties \
/opt/confluent/share/confluent-common/connectors/sink-redis.properties
```

Then, format Kafka Storage:

```
/opt/confluent/bin/kafka-storage format -t \
2vi2WtHxQAOPyXb1Bj1Jvw -c /opt/confluent/etc/kafka/server.properties \
--standalone > /dev/null 2>&1; echo 0
```

Afterwards, start Kafka Broker:

```
/opt/confluent/bin/kafka-server-start /opt/confluent/etc/kafka/server.properties        
```

Then create a new Kafka Topic (_e.g.,_ "posgres-topic"):

```
kafka-topics --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```

Lastly, run the Schema Registry:
```
/opt/confluent/bin/schema-registry-start \
/opt/confluent/etc/schema-registry/schema-registry.properties
```

At this point, ensure PostgreSQL, Redis and Grafana are up and running on the system, on their default port configurations.
To see metrics flow seamlessly into PostgreSQL, once can run the following command (_i.e.,_ assuming the database name is "opendc"):

```
sudo su postgres; psql -d opendc 
```

Additionally, to interact with the Redis cache, we suggest to use the excellent redis-cli tool:

```
redis-cli -h localhost -p 6379
```

The following commands can be used to list the contents of the cache (_i.e.,_ assuming stream name "postgres_topic"), and to clear the cache respectively:

```
XRANGE postgres_topic - +
XTRIM postgres_topic MAXLEN 0
```
	
For the predictive analytics in our implementation it is useful to create a separate consumer group:

```
XGROUP CREATE mystream mygroup 0                
```

Lastly, each time you change the database schema, you must run (_i.e.,_ assuming `schema.proto` lives in `resources/experiments`):

```
cd resources/experiments/;
protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/  schema.proto
```
	
_Sunfish_ acts as the digital twin and the modified OpenDC (this repository) acts as the physical datacenter.
Both contain modifications.
Additionally: create a `.venv` in `python_scripts` and `http_server` and install all the dependencies.
Then, run respectively the HTTP Server and the Python Scripts.
Afterwards, start IntellijIDEA in this repository and run some experiments.

## Useful commands

To access a certain PostgreSQL database _e.g.,_ called "opendc":
`psql -d opendc`

This command selects everything from the topic which Kafka writes to.
`SELECT * FROM postgres_topic;`

Connect to Redis server via an excellent command-line tool:
`redis-cli -h localhost -p 6379`
Very useful tutorial https://redis.io/docs/latest/develop/data-types/streams/
See also https://redis.io/docs/latest/develop/tools/cli/

The below command returns all items in the Redis stream
`XRANGE postgres_topic - +`
`XTRIM postgres_topic MAXLEN 0`

To create a group for consumers to read from stream:
`XGROUP CREATE mystream mygroup 0`

## Contributing

Questions, suggestions and contributions are welcome and appreciated!
Please refer to the [contributing guidelines](CONTRIBUTING.md) for more details.

## License

OpenDC is distributed under the MIT license. See [LICENSE.txt](/LICENSE.txt).