We are sending data through Kafka cluster having multiple partitions using round-robin and consume from Kafka with multiple consumers.
In case of maintenance or other disruption, causing queueu buildup, consumers may resume reading untransferred data from multiple partitions at different speeds and send them to Kapacitor.
What would be the effect of such unordered data points,
what kind of nodes may generate or miss alerts,
are there any recommendations or thoughts on how to avoid this?
I found that in Kafka
(not related to InfluxDB, but still wanted to share if somebody is interested in this setup)
there is a Producer settings “partitioner” to assign partition based on the key hash.
So if I split a line from line protocol by the first space I get measurement+keys as Kafka message Key and the rest as message value, Kafka Producer will consistently assign the same partition for the series.
If that’s not enough, it is possible to have a broader key, for example the measurement alone, so that all data for the measurement will go through the same partition and thus be ordered.
The only downside here is that if the number of keys is limited and amounts of data are not evenly distributed, the size of partitions may be skewed and space utilization between Kafka servers become uneven.
I am still interested in how would Kapacitor alerts be affected when a Task checks based on streams from different measurements. However with the above approach it will be a rare case.
