Batch Processing vs. Stream Processing: What’s the Difference?

InfluxDB 1 Kapacitor
1

Originally published at: Batch Processing vs. Stream Processing: What's the Difference? | InfluxData

If you’ve read DevRel Katy Farmer’s stellar post, Kapacitor and Continuous Queries: How To Decide Which Tool You Need, then you know that when our community talks, we listen. So, in alignment with that view and in honor of our very own Kapacitor Koala, let’s tackle another common community issue that has come to our attention: when should we use batch processing versus stream processing in our Kapacitor tasks?

[caption id=“attachment_214447” align=“aligncenter” width=“308”] Our famous Kapacitor Koala[/caption]

Now, if you’ve no vague idea what Kapacitor is, I recommend doing a little light reading on it here and here just to get you up to speed. Kapacitor, the final component of our TICK Stack, offers several capabilities such as data transformation, downsampling, and alerting. Kapacitor uses its own DSL, called TICKscript, which allows you to define certain tasks, which can then be executed on your data—essentially, it’s processing your data for you.

Here’s where it gets tricky though: how do you choose whether to process your data as a batch task or streaming task?

Batch Tasks

Let’s discuss batch tasks first. A batch is a collection of data points that have been grouped together within a specific time interval. Another term often used for this is a window of data. When running a batch task, Kapacitor queries InfluxDB periodically, thereby avoiding having to buffer much of your data in RAM. There are several cases where batch processing is the way to go:
  • Performing aggregate functions such as finding the mean, maximum, or minimum of a set interval of data.
  • Cases where alerting doesn’t need to run on every single data point (since state changes will probably not happen that often). You don’t want to be inundated with alerts!
  • Downsampling of your data takes a large collection of data points and only retains the most significant data (so you can still view overall trends in the data).
  • Cases where a little extra latency won’t severely impact your operation.
  • Cases with a super-high throughput InfluxDB instance since Kapacitor cannot process data as quickly as it can be written to InfluxDB (this occurs more frequently with InfluxEnterprise clusters).

Stream Tasks

On the other side, we have stream tasks. Stream tasks create subscriptions to InfluxDB so that every data point written to InfluxDB is also written to Kapacitor. One should note though that stream tasks use a high percentage of available memory, so memory availability is a key factor to take into consideration. Here’s where stream processing is most ideal:
  • If you want to transform each individual data point in real time (technically, this could also be run with a batch process but there’s latency to consider).
  • Cases where lowest possible latency is paramount to the operation. If alerts need to be triggered immediately, for example, running a stream task will ensure the least possible delay.
  • Cases in which InfluxDB is handling high volume query load and you may want to alleviate some of the query pressure from InfluxDB.
  • Stream tasks understand time by the data’s timestamps; there are no race conditions for when exactly a given point will make it into a window or not. With batch tasks, on the other hand, it is possible for a data point to arrive late and be left out of its relevant window.
Another advantage some might see with writing stream tasks is the ease of use in having to define the task using only Kapacitor’s TICKscript, without having to delve into writing queries for InfluxDB. If you are comfortable with writing both, however, it’s probably going to be in your best interest to go with batch processing most of the time since it uses a lot less memory. An additional factor to consider is that Kapacitor is not limited to use only with InfluxDB. For example, if you want to send data straight from Telegraf over to Kapacitor, that will have to be done as a streaming task.

Key Takeaways

  • Batch tasks query InfluxDB periodically, use limited memory, but can place additional query load on InfluxDB.
  • Batch tasks are best used for performing aggregate functions on your data, downsampling, and processing large temporal windows of data.
  • Stream tasks subscribe to writes from InfluxDB placing additional write load on Kapacitor, but can reduce query load on InfluxDB.
  • Stream tasks are best used for cases where low latency is integral to the operation.
When our community talks, we listen.
We’d love to hear how your batch and stream tasks are going! Send us your comments, questions, issues, and blog ideas on our community site and feel free to reach out to us on Twitter:

@InfluxDB
@mschae16

1 Like
2

Stream tasks fail for a particular alerting application that seems like it would be common: Alerting on a stateDuration triggered by an infrequently sampled data point such as a Boolean that is written to the database only on change. Let’s say you have a condition like, “Alert me whenever this Boolean has been true for over 10 seconds” If the data points that Boolean depends on are only written on change, or at a frequency less than the resolution you want to alert at, then the alert will never trip. The Boolean will become true, causing the stateDuration node with a unit of 1s to emit “duration=0” and then since the data is not sampled after this until the next change, the alert based on a condition like “duration > 10” won’t fire at the actual 10 second mark, but only when the next data point comes in after 10 seconds. If that data happens to reset the Boolean to false, no alert will happen.

In this case I just have to use a batch task and query at a frequency higher than the resolution I want to alert at.

I have written about this issue here on the forum and here as an open issue on github and received no response about the correct way to work around this subtle but significant limitation of stream tasks and the stateDuration node.

3

Hi Adam,

I apologize for the delay in responding to your issue. We will take a look into this issue and respond as soon as we can.

Margo

4

Thanks for the response. I’ll add comments on my previously mentioned thread.

5

Hi Adam,

There are a few people looking into your issue on gitHub, if you’d like to join the conversation there.

Thanks!

6

Embedded devices commonly used at the edge with little memory and fragile storage really shine a light on the tradeoffs. Assuming there is enough memory to use the stream tasks, will doing so result in less writes to disk for something like the application David Simmons mentions in his blog post about architecting IOT gateway devices?

How does Kapacitor handle a network interruption while forwarding data?