How to calculate values written per second for specific database

,
1

Hi, is there a way to calculate the values written per second for a specific database?
I’d like to figure out what the avg value written per second is for a 5 min interval so that I can set up a kapacitor tick script to alert when a certain threshold is reached.

Thank you.

2

If you are using Telegraf, you can enable the internal input plugin to report how many “metrics” Telegraf is writing to each output. You could then have Kapacitor sum up the value across hosts and generate the alert.

3

What if I have multiple Telegraf containers writing to the same Kafka output, then would I need to enable the internal plugin in all the telegraf.conf files?

4

Did some testing…
Test Setup:

  • 2 Telegraf containers collecting data from routers and sending to Kafka output.
  • Each telegraf.conf file is configured to collect data from 5-6 routers.
  • One InfluDB container that consumes data from the Kafka topic that Telegraf is producing to.

I enabled the [[inputs.internal]] plugin on one container and now see the following measurements:
> show measurements
name: measurements
name
----
interfaces
internal_agent
internal_gather
internal_write

Ran the following query to calculate the rate of metrics written per minute:

> select non_negative_difference(last("metrics_written")) from "internal_write" where time >= now() - 10m group by time(1m)
name: internal_write
time                 non_negative_difference
----                 -----------------------
2019-11-20T13:12:00Z 7047724
2019-11-20T13:13:00Z 7352
2019-11-20T13:14:00Z 7352
2019-11-20T13:15:00Z 7352
2019-11-20T13:16:00Z 7352
2019-11-20T13:17:00Z 7352
2019-11-20T13:19:00Z 6673940
2019-11-20T13:20:00Z 7176
2019-11-20T13:22:00Z 4044360

I believe this gives me the metrics written per minute. The huge spike in the rate I believe is caused by how many metrics Telegarf counted at the moment it was writing to the output, which is Kafka in my case. Not really sure how to smooth those numbers out.

I then enabled the [[inputs.internal]] plugin on my second Telegraf container, which is also writing to the same Kafka output, but I did not see the rate increase dramatically from what it was when I only enabled the plugin on the first container.

I hope I’m testing this correctly. If someone at Influx can confirm and/or correct this testing that would be awesome.

Thank you.

5

What if I have multiple Telegraf containers writing to the same Kafka output, then would I need to enable the internal plugin in all the telegraf.conf files?

6

Actually, I think I may have misspoke. I did another round of testing.

  1. Enabled [[inputs-internal]] plugin on my first Telegraf container which is pulling data from 2 routers. I noticed the rate was around 670 writes per min.
  2. I then enabled the [[inputs-internal]] plugin on my second Telegraf container which is pulling same data but from two different routers. I noticed the rate spike to 678,000 writes per min.

Both Telegraf containers were using the same hostname and they are both writing to the same Kafka output topic.

I then made the hostname unique in both Telegraf containers and noticed that when I select the appropriate host in Grafana I see a more appropriate writes per minute rate for each host. I’m just wondering why I can’t leave the hostname the same for both containers and see the write per minute rate double instead of spike by 1K.