We need to sort out kafka

[slifty]

Task

Description

We're using Kafka for managing our data streams, but right now we're just using whatever comes out of the box from kafkajs, without much clarity on exactly how it works and how we interact with it.

Specifically:

1. Data lives in kafka for a while; we probably don't want very long retention policies.
2. When a new TV Kitchen worker connects, we don't want it to read historic messages (I believe that kafkaJS has a fromBeginning that probably defaults to false, but we should confirm).
3. Are we supposed to be tagging messages in any special way?

I don't know what code will come out of this issue, but I suspect we have some configurations to make (at the very least to our kafka service + documentation).

Relevant Resources / Research

• kafkajs

Related Issues

This will likely impact how we interact with kafka in the countertop, so #23 is a related issue.

[slifty]

Another example of a problem: when I start / restart during testing it seems that consumers are not automatically de-registering after dying. This means that messages are being sent to the now-dead consumers instead of the active consumer, so the active consumer just sits there silently.

[chriszs]

I set KAFKA_LOG_RETENTION_BYTES in the Docker Compose just now as test to see whether that improved Kafka performance. Seems like what we may want here. The alternative is setting retention by hours.