Capture metrics for all running instances of a container


#1

I have a microservice running in kubernetes as a SpringBoot 2 application. Right now I’m pushing using statsd to Telegraf.

My configuration allows for unique identifiers per container:
monitor.instance: ${spring.application.name}-${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}

Problem is, when I set up a chart to measure, say Memory of the container, I need to reference some other instance identifier and group by the monitor.instance as each time a container recreates, the instance id will be regenerated.

Currently my example query is:

SELECT mean(“value”) AS “mean_value” FROM “telegraf”.“autogen”.“jvm_memory_used” WHERE time > now() - 1h AND “app_instance”=‘api-ea486d5fcdf261f711f0a9fb2d38513c’ AND “statistic”=‘value’ GROUP BY :interval: FILL(null)

Of course on recreation of the container (redeploy, crash, scale, etc) the instance identifier is considered archived.

How do I ensure that I capture metrics (for CPU) for all running instances of a container?