Jolokia2_proxy: request canceled (Client.Timeout exceeded while awaiting headers)

Running Jolokia 1.6.x in proxy mode under Tomcat 8.5.x. Here’s my jolokia2_proxy input configuration:

[[inputs.jolokia2_proxy]]

## Proxy agent
url = "http://localhost:8080/jolokia"
username = "agent"
password = "<password>"
response_timeout = "60s"

  # target-1
  [[inputs.jolokia2_proxy.target]]
  url = "service:jmx:rmi:///jndi/rmi://10.1.1.2:xxxx/jmxrmi"
  ...
  # target-16
  [[inputs.jolokia2_proxy.target]]
  url = "service:jmx:rmi:///jndi/rmi://10.1.1.2:xxxx/jmxrmi"

## Add metrics to read
[[inputs.jolokia2_proxy.metric]]
  name  = "java_runtime"
  mbean = "java.lang:type=Runtime"
  paths = ["Uptime"]

[[inputs.jolokia2_proxy.metric]]
  name  = "java_memory"
  mbean = "java.lang:type=Memory"
  paths = ["HeapMemoryUsage", "NonHeapMemoryUsage", "ObjectPendingFinalizationCount"]

[[inputs.jolokia2_proxy.metric]]
  name     = "java_garbage_collector"
  mbean    = "java.lang:name=*,type=GarbageCollector"
  paths    = ["CollectionTime", "CollectionCount"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metric]]
  name  = "java_last_garbage_collection"
  mbean = "java.lang:name=*,type=GarbageCollector"
  paths = ["LastGcInfo"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metrics]]
  name  = "java_threading"
  mbean = "java.lang:type=Threading"
  paths = ["TotalStartedThreadCount", "ThreadCount", "DaemonThreadCount", "PeakThreadCount"]

[[inputs.jolokia2_proxy.metrics]]
  name  = "java_class_loading"
  mbean = "java.lang:type=ClassLoading"
  paths = ["LoadedClassCount", "UnloadedClassCount", "TotalLoadedClassCount"]

[[inputs.jolokia2_proxy.metrics]]
  name     = "java_memory_pool"
  mbean    = "java.lang:name=*,type=MemoryPool"
  paths    = ["Usage", "PeakUsage", "CollectionUsage"]
  tag_keys = ["name"]

[[inputs.jolokia2_proxy.metrics]]
  name     = "econnect_queue"
  mbean    = "eConnect:name=*,type=WorkerQueue"
  paths    = ["DequeueCount", "EnqueueCount", "QueueSize"]
  tag_keys = ["name"]

My [agent] interval value is also 60s but I’m still seeing the following errors:

2019-05-24T06:31:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:***@localhost:8080/jolokia/read: net/http: timeout awaiting response headers
2019-05-24T06:32:00Z W! [agent] input “inputs.jolokia2_proxy” did not complete within its interval
2019-05-24T06:32:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:***@localhost:8080/jolokia/read: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2019-05-24T06:33:00Z W! [agent] input “inputs.jolokia2_proxy” did not complete within its interval
2019-05-24T06:33:00Z E! [inputs.jolokia2_proxy]: Error in plugin: Post hxxp://agent:***@localhost:8080/jolokia/read: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

I’m guessing my interval values are too low. Is there a way I can see how long each batch takes so I can adjust the interval value properly?

I’m not sure, 60 seconds is quite generous, is it working at all? You can set the response_timeout very high, enable the internal plugin, and watch the internal_gather measurement for timing of the plugin.

My bad. Apparently, the firewall was blocking the requests to the instances. Once we added the correct rules, timeout wasn’t an issue anymore. Thanks for all your help!