Telegraf not posting data to multiple Databases on Same Influxdb

Hi,

I am passing all the collected metrics from JVM to jvmstats and Docker Metrics to dockerstats which is hosted on same Influxdb instance. Below are my configurations: I am facing below issues:

  1. If I say SHOW MEASUREMENTS on dockerstats it is empty. But in jvmstats it shows all including docker measurements
  2. Only docker metrics make out to the dockerstats or I can do the same using jvmstats as the measurements are available here.
  3. JVM metrics is getting dropped or not made it to the database.

###############################################################################

OUTPUTS - JVM

###############################################################################

Configuration for influxdb server to send metrics to

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

Multiple urls can be specified as part of the same cluster,

this means that only ONE of the urls will be written to each interval.

urls = [“udp://localhost:8089”] # UDP endpoint example

urls = [“http://HOST:PORT”] # required

The target database for metrics (telegraf will create it if not exists).

database = “jvmstats” # required

Retention policy to write to.

retention_policy = “”

Precision of writes, valid values are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.

note: using “s” precision greatly improves InfluxDB compression.

precision = “s”

Write timeout (for the InfluxDB client), formatted as a string.

If not provided, will default to 5s. 0s means no timeout (not recommended).

timeout = “5s”
username = “admin”
password = “admin”

Set the user agent for HTTP POSTs (can be useful for log differentiation)

user_agent = “telegraf”

Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)

udp_payload = 512

Optional SSL Config

ssl_ca = “/etc/telegraf/ca.pem”

ssl_cert = “/etc/telegraf/cert.pem”

ssl_key = “/etc/telegraf/key.pem”

Use SSL but skip chain & host verification

insecure_skip_verify = false

namedrop = [“container_*”,“docker_”,“engine”]

###############################################################################

OUTPUTS - Docker

###############################################################################

Configuration for influxdb server to send metrics to

[[outputs.influxdb]]

The full HTTP or UDP endpoint URL for your InfluxDB instance.

Multiple urls can be specified as part of the same cluster,

this means that only ONE of the urls will be written to each interval.

urls = [“udp://localhost:8089”] # UDP endpoint example

urls = [“http://HOST:PORT”] # required

The target database for metrics (telegraf will create it if not exists).

database = “dockerstats” # required

Retention policy to write to.

retention_policy = “”

Precision of writes, valid values are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.

note: using “s” precision greatly improves InfluxDB compression.

precision = “s”

Write timeout (for the InfluxDB client), formatted as a string.

If not provided, will default to 5s. 0s means no timeout (not recommended).

timeout = “5s”
username = “admin”
password = “admin”

Set the user agent for HTTP POSTs (can be useful for log differentiation)

user_agent = “telegraf”

Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)

udp_payload = 512

Optional SSL Config

ssl_ca = “/etc/telegraf/ca.pem”

ssl_cert = “/etc/telegraf/cert.pem”

ssl_key = “/etc/telegraf/key.pem”

Use SSL but skip chain & host verification

insecure_skip_verify = false

namedrop = [“jolokia*”]

Any help is appreciated.

Thanks in advance
Karthik

@kpalanivelu Without seeing your input configuration I can’t make a determination as to what is going on here. Have you tried splitting this config into 2 telegraf instances?

I tried running two instances of telegraf with two configs. It works. I am wondering, can I do it with one conf file.

Inputs for your review:

###############################################################################

INPUT PLUGINS

###############################################################################

# Read metrics about docker containers

[[inputs.docker]]
endpoint = "ENV"
container_names = []
timeout = "5s"
perdevice = true
total = false

###############################################################################

INPUT PLUGINS

###############################################################################

# Read JMX metrics through Jolokia

[[inputs.jolokia]]
context = “/read”

[[inputs.jolokia.servers]]
name = "IP"
host = "CONTAINER"
port = “8080”

## List of metrics collected on above servers

## Each metric consists in a name, a jmx path and either

## a pass or drop slice attribute.

## This collect all heap memory usage metrics.

[[inputs.jolokia.metrics]]
name = "heap_memory_usage"
mbean = "java.lang:type=Memory"
attribute = “Verbose,ObjectPendingFinalizationCount,NonHeapMemoryUsage”

## This collect thread counts metrics.

[[inputs.jolokia.metrics]]
name = "thread_count"
mbean = "java.lang:type=Threading"
attribute = “TotalStartedThreadCount,ThreadCount,DaemonThreadCount,PeakThreadCount”

## This collect number of class loaded/unloaded counts metrics.

[[inputs.jolokia.metrics]]
name = "class_count"
mbean = "java.lang:type=ClassLoading"
attribute = “LoadedClassCount,UnloadedClassCount,TotalLoadedClassCount”

## This collect number of GC.

[[inputs.jolokia.metrics]]
name = "GC-MarkSweep"
mbean = "java.lang:name=PS MarkSweep,type=GarbageCollector"
attribute = “LastGcInfo,CollectionCount,CollectionTime,Valid”

## This collect number of GC.

[[inputs.jolokia.metrics]]
name = "GC-Scavenge"
mbean = "java.lang:name=PS Scavenge,type=GarbageCollector"
attribute = “LastGcInfo,CollectionCount,CollectionTime,Valid”

Please advise if this is the recommended approach for two different input plugins to same output different databases:

nohup /telegraf/usr/bin/telegraf -config /opt/telegraf/telegraf.conf > /dev/null 2>/var/log/telegraf/telegraf.log &
nohup /telegraf/usr/bin/telegraf -config /opt/telegraf/telegraf-docker.conf > /dev/null 2>/var/log/telegraf/telegraf-docker.log &

It should not be required to use two instances, what version of Telegraf are you using?

I am using 1.1.2 of the telegraf