Jolokia2_proxy: request canceled (Client.Timeout exceeded while awaiting headers)

Telegraf
1

Running Jolokia 1.6.x in proxy mode under Tomcat 8.5.x. Here’s my jolokia2_proxy input configuration:

[[inputs.jolokia2_proxy]]

## Proxy agent
url = "http://localhost:8080/jolokia"
username = "agent"
password = "<password>"
response_timeout = "60s"

  # target-1
  [[inputs.jolokia2_proxy.target]]
  url = "service:jmx:rmi:///jndi/rmi://10.1.1.2:xxxx/jmxrmi"
  ...
  # target-16
  [[inputs.jolokia2_proxy.target]]
  url = "service:jmx:rmi:///jndi/rmi://10.1.1.2:xxxx/jmxrmi"

## Add metrics to read
[[inputs.jolokia2_proxy.metric]]
  name  = "java_runtime"
  mbean = "java.lang:type=Runtime"
  paths = ["Uptime"]

[[inputs.jolokia2_proxy.metric]]
  name  = "java_memory"
  mbean = "java.lang:type=Memory"
  paths = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"]

[[inputs.jolokia2_proxy.metric]]
  name     = "java_garbage_collector"
  mbean    = "java.lang:name=*,type=GarbageCollector"
  paths    = ["CollectionTime", "CollectionCount"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metric]]
  name  = "java_last_garbage_collection"
  mbean = "java.lang:name=*,type=GarbageCollector"
  paths = ["LastGcInfo"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metrics]]
  name  = "java_threading"
  mbean = "java.lang:type=Threading"
  paths = ["TotalStartedThreadCount", "ThreadCount", "DaemonThreadCount", "PeakThreadCount"]

[[inputs.jolokia2_proxy.metrics]]
  name  = "java_class_loading"
  mbean = "java.lang:type=ClassLoading"
  paths = ["LoadedClassCount", "UnloadedClassCount", "TotalLoadedClassCount"]

[[inputs.jolokia2_proxy.metrics]]
  name     = "java_memory_pool"
  mbean    = "java.lang:name=*,type=MemoryPool"
  paths    = ["Usage", "PeakUsage", "CollectionUsage"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metrics]]
  name     = "econnect_queue"
  mbean    = "eConnect:name=*,type=WorkerQueue"
  paths    = ["DequeueCount", "EnqueueCount", "QueueSize"]
  tag_keys = ["name"]

My [agent] interval value is also 60s but I’m still seeing the following errors:

2019-05-24T06:31:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:@localhost:8080/jolokia/read: net/http: timeout awaiting response headers
2019-05-24T06:32:00Z W! [agent] input “inputs.jolokia2_proxy” did not complete within its interval
2019-05-24T06:32:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:@localhost:8080/jolokia/read: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019-05-24T06:33:00Z W! [agent] input “inputs.jolokia2_proxy” did not complete within its interval
2019-05-24T06:33:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:***@localhost:8080/jolokia/read: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

I’m guessing my interval values are too low. Is there a way I can see how long each batch takes so I can adjust the interval value properly?

2

I’m not sure, 60 seconds is quite generous, is it working at all? You can set the response_timeout very high, enable the internal plugin, and watch the internal_gather measurement for timing of the plugin.

3

My bad. Apparently, the firewall was blocking the requests to the instances. Once we added the correct rules, timeout wasn’t an issue anymore. Thanks for all your help!