diff options
Diffstat (limited to 'shell_scripts')
| -rwxr-xr-x | shell_scripts/kafka_connect.sh | 4 | ||||
| -rw-r--r-- | shell_scripts/readme.md | 13 | ||||
| -rw-r--r-- | shell_scripts/useful_commands.sh | 19 |
3 files changed, 4 insertions, 32 deletions
diff --git a/shell_scripts/kafka_connect.sh b/shell_scripts/kafka_connect.sh index 3f22b6de..fecc8f63 100755 --- a/shell_scripts/kafka_connect.sh +++ b/shell_scripts/kafka_connect.sh @@ -1,2 +1,6 @@ #!/bin/bash /opt/confluent/bin/connect-standalone /opt/confluent/etc/kafka/connect-standalone.properties /opt/confluent/share/confluent-common/connectors/sink-jdbc.properties /opt/confluent/share/confluent-common/connectors/sink-redis.properties +# Considerable effort has gone into filtering messages in situ. Please refer to: +# https://forum.confluent.io/t/invalid-value-for-filter-in-smt/10046 +# https://github.com/json-path/JsonPath +# https://docs.confluent.io/kafka-connectors/transforms/current/filter-confluent.html#properties diff --git a/shell_scripts/readme.md b/shell_scripts/readme.md deleted file mode 100644 index 2a0842b1..00000000 --- a/shell_scripts/readme.md +++ /dev/null @@ -1,13 +0,0 @@ -You need to run this each time you change `schema.proto` -```bash -cd resources/experiments/ -protoc --java_out=/home/matt/git/opendc/opendc-common/src/main/java/ schema.proto -curl -X DELETE http://localhost:8081/subjects/postgres-topic-value -``` -Confluent JDBC sink and source (includes Postgres connector) files: -https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc - -Be aware the project will only work with this OpenDC as there are hidden Maven dependencies. -Kafka topic should be named `postgres_topic`. - -Confluent local (see https://www.confluent.io/installation/): diff --git a/shell_scripts/useful_commands.sh b/shell_scripts/useful_commands.sh deleted file mode 100644 index 8ff604e2..00000000 --- a/shell_scripts/useful_commands.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash - -doas su postgres -# In this project we only use the ``opendc'' database. -psql -d opendc - -# Queries -# Selects everything from the topic which Kafka writes to. -SELECT * FROM postgres_topic; - -# Redis CLI -# Connect to Redis -redis-cli -h localhost -p 6379 -# Very useful tutorial https://redis.io/docs/latest/develop/data-types/streams/ -# https://redis.io/docs/latest/develop/tools/cli/ -# Returns all items in the Redis stream -XRANGE postgres_topic - + -XTRIM postgres_topic MAXLEN 0 -# https://codesignal.com/learn/courses/mastering-redis-for-high-performance-applications-with-java-and-jedis-1/lessons/redis-streams-with-java |