Representing a message broker in DataGalaxy gives you a robust documentation solution for event management.
To understand our approach, we take Kafka as an example. In simple terms, Kafka sits between message producers (who push information to Kafka) and message consumers (who pull information from Kafka).
How to represent the broker in the dictionary
We can therefore consider Kafka as a data hub, which is why we use the dictionary to represent Kafka. To do so:
- As messages are often in JSON format, we recommend using NoSQL databases.
- Topics are represented by directories.
- Messages are then represented by files.
Screenshot: topics as directories and messages as files in the dictionary. (Re-add this image here — the original Freshdesk link expires after a few days.)
You can manage the versions of the messages by including a numbering in the file names.
How to trace producers and consumers
You can then identify the producers' publications and the consumers' retrievals through data processing. This makes it possible to identify:
- All consumers of a publication
- All producers used by a consumer
This works the same way for other message brokers: RabbitMQ, Red…