Outputs.Cloudwatch write_statistics confusion

Telegraf
1

Outputs.Cloudwatch

I’m pushing metrics to Cloudwatch with the telegraf plugin outputs.cloudwatch. I have the telegraf agent installed on many machines.

write_statistics

I enabled write_statistics option per recommended to batch results, however, I’m really not happy with the resulting stored values in Cloudwatch.

For example, a memory metric is now stored with multiple entries like: max_memory_mean, max_memory_max,max_memory_sum etc.

aggregation plugin and write_statistics

I’m already using the aggregation plugin to capture metrics every 15 seconds or so, and then get the mean/median value from the aggregator to send 1x a minute up to Cloudwatch. This is an aggregated value. However, for Cloudwatch, I think it would just show up as a single metric at a point in time. I don’t really want to send the same value as average, max, minimum etc. Not only is it confusing, but really all I want is to use the single point.

my main questions

  1. If I disable the write_statistics, and use the aggregation plugin to calculate the value I want, is that acceptable to write the data point?
  2. If I use the write_statistics setting, is there anyway for it come through as a single max_memory type metric, rather than 4 different types of calculated metrics, when I’ve already done the aggregation to get the 1 value I want?
2

I think the answer for both of these questions currently is no, but both make sense as improvements. Do you think you could open two new feature request issues on the Telegraf GitHub?

3

That sounds good. I’ll try and do that today.
So basically, I’m sending raw metrics right now, but aws doesn’t know any better that I’m actually calculating the aggregation myself.

Am I correct in assuming it still uses an efficient batching of the metrics to send in as few api calls as possible when I send metrics without write statistics? I have the output plugin set to drop all fields except the _mean so i’m accomplishing most of the filtering I need right now based on that at least.

4

The plugin makes calls using PutMetricData, which has a limit of up to 20 different AWS metrics. It does make one call per 20 metrics, but there is onne aspect where I believe the plugin could be improved. If there are multiple values per metric it should be possible to write them together as the same metric in the request, up to 150 values, but instead each value is written separately. It seems that this could be helpful if you are writing less frequently than the collection interval, and you have several intervals worth of data.