Using inputs.http plugin getting Previous collection has not completed; scheduled collection skipped

1

Hi InfluxDB Community,

I’m reaching out for some help or guidance regarding an issue I’m experiencing with the inputs.http plugin in Telegraf.
I’m using multiple input configuration files to collect data from different IPs via Redfish APIs. Here’s a simplified example of my plugin config:
This is my first config file:

[[inputs.http]]
  alias = "http-api-redfish_bios4"
  urls = ["https://[ip1]/redfish/v1/systems/1/bios/settings", "https://[ip1]/json/overview"]
  method = "GET"
  username = "********"
  password = "********"
  data_format = "json"
  tags = { http-api-tag = "http-api-redfish_bios4", nename="xyz"}
  json_string_fields = ["*"]
  insecure_skip_verify = true
  interval = "5m"
  timeout = "60s"

[[outputs.elasticsearch]]
  alias = "http-api-output-redfish_bios4"
  tagpass = { http-api-tag = ["http-api-redfish_bios4"]}
  urls = ["http://of-opensearch-master:9200"]
  index_name = "redfish-%H"
  username = "********"
  password = "********"
  metric_batch_size = 100

This is my Second config file

[[inputs.http]]
  alias = "http-api-redfish_bios6"
  urls = ["https://[ip2]/redfish/v1/systems/1/bios/settings","https://[ip2]/json/overview"]
  method = "GET"
  ## Credentials for the Redfish API.
  username = "********"
  password = "********"
  data_format = "json"
  tags = { http-api-tag = "http-api-redfish_bios6", nename="abc"}
  json_string_fields = ["*"]
  insecure_skip_verify = true
  interval = "5m"
  timeout = "60s"

  [[outputs.elasticsearch]]
  alias = "http-api-output-redfish_bios6"
  tagpass = { http-api-tag = ["http-api-redfish_bios6"]}
  urls = ["http://of-opensearch-master:9200"]
  index_name = "redfish-%H"
  username = "********"
  password = "********"
  metric_batch_size = 100

Occasionally, I encounter the following warning and debug logs

2025-04-14T15:47:29Z D! [inputs.http::http-api-redfish_bios4] Previous collection has not completed; scheduled collection skipped
2025-04-14T15:47:29Z W! [inputs.http::http-api-redfish_bios4] Collection took longer than expected; not complete after interval of 5m
2025-04-14T15:47:29Z D! [inputs.http::http-api-redfish_bios6] Previous collection has not completed; scheduled collection skipped
2025-04-14T15:47:29Z W! [inputs.http::http-api-redfish_bios6] Collection took longer than expected; not complete after interval of 5m

Once the issue come in the first config file. It impacts all the config file and give the same logs for all config files. And all the files stop writing data to Opensearch

When this happens, it appears to block or delay data collection from other IPs as well, and as a result, metrics aren’t being written to Opensearch as expected.

My Questions:

  1. Is there a way to isolate the inputs so that if one API call takes too long, it doesn’t affect others?
  2. Would running each input in a separate Telegraf instance or container help fix the issue?
  3. Are there any known optimizations or recommended best practices to avoid this issue?
  4. Are there any known limitations with the inputs.http plugin that might be causing this issue, especially when dealing with multiple IPs?
  5. Does Telegraf supports parallel execution through the use of inputs.http plugins and configurations.

Thanks in Advance.

2

  1. Yes there are ways to isolate inputs. The most effective approach is to run multiple Telegraf instances. This is a common practice for handling different types of metrics or when dealing with potential performance issues.

  2. Yes,running each input in a separate Telegraf instance would help fix the issue. This is supported by the documentation and is considered a best practice for certain scenarios. see this example: telegraf/plugins/inputs/vsphere/README.md at master · influxdata/telegraf · GitHub

  3. Several optimizations can help:

  • Adjust timeouts: Your current timeout is 60s, which might not be enough for some API calls.

  • Collection jitter: Adding collection jitter can help distribute the load: collection_jitter = “30s”

  • Increase interval: If 5m isn’t enough for your API calls to complete, consider increasing it.

  • Split configurations: As mentioned in a community post, splitting configurations across multiple Telegraf instances is a common solution: Multiple Agents SNMP

  1. When a plugin’s collection takes longer than its interval, it can disrupt metric collection and result in missed samples. This appears to be what you’re experiencing. A similar issue was reported with the inputs.http plugin where cookie authentication failures would crash the entire agent: Inputs.http cookie failure

  2. Telegraf does support parallel execution through multiple plugin instances, but there are limitations. Each plugin instance runs independently, but they all share the same process resources.

So to summarize, try the following:

  1. Split your configurations into separate Telegraf instances, each handling a subset of your API endpoints.

  2. Adjust timeouts and intervals based on the actual response times of your endpoints.

  3. Consider using collection jitter to distribute the load.

  4. Monitor each instance’s performance using Telegraf’s internal metrics.