I’m unsure if this is a bug or if I’m overlooking something obvious. I’ve examined the issue in great detail and have searched github issues, this forum, and the general internet.
I should not be seeing this in the logs when I can connect and get data from the API by other means. I’m near certain influxdb is not an issue here as other plugins are writing successfully.
** anticipating question: have you tried a different API? Yes, several. This one was chosen to demo because it’s totally open / no key required (and very interesting content). All other APIs used this way have undergone/ passed/failed same rigorous testing shown below in same exact way
This is my input definition. An API (api.artic.edu) request, expecting json, interval at 35 seconds:
[[inputs.http]]
urls = [ "https://api.artic.edu/api/v1/artworks/search?q=cats" ]
interval = "35s"
use_system_proxy = true
timeout = "15s"
data_format = "json"
I see this in my logs regularly. I can reproduce consistently. Alternating dial tcp and context deadline exceeded:
Dec 10 12:05:10 ingest001 telegraf[10718]: 2023-12-10T20:05:10Z E! [inputs.http] Error in plugin: [url=https://api.artic.edu/api/v1/artworks/search?q=cats]: Get "https://api.artic.edu/api/v1/artworks/search?q=cats": dial tcp 108.138.85.28:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
Dec 10 12:05:45 ingest001 telegraf[10718]: 2023-12-10T20:05:45Z E! [inputs.http] Error in plugin: [url=https://api.artic.edu/api/v1/artworks/search?q=cats]: Get "https://api.artic.edu/api/v1/artworks/search?q=cats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Dec 10 12:06:20 ingest001 telegraf[10718]: 2023-12-10T20:06:20Z E! [inputs.http] Error in plugin: [url=https://api.artic.edu/api/v1/artworks/search?q=cats]: Get "https://api.artic.edu/api/v1/artworks/search?q=cats": dial tcp 108.138.85.25:443: i/o timeout (Client.Timeout exceeded while awaiting headers)
Dec 10 12:06:55 ingest001 telegraf[10718]: 2023-12-10T20:06:55Z E! [inputs.http] Error in plugin: [url=https://api.artic.edu/api/v1/artworks/search?q=cats]: Get "https://api.artic.edu/api/v1/artworks/search?q=cats": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
From the same system (ingest001) I can get data from the API
# curl -v 'https://api.artic.edu/api/v1/artworks/search?q=cats'
* Uses proxy env variable no_proxy == 'localhost, .somewhere.io'
* Uses proxy env variable https_proxy == 'http://proxy.somewhere.io:8118'
* Trying 192.168.1.30:8118...
* Connected to (nil) (192.168.1.30) port 8118 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api.artic.edu:443
> CONNECT api.artic.edu:443 HTTP/1.1
> Host: api.artic.edu:443
> User-Agent: curl/7.81.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< date: Sun, 10 Dec 2023 20:40:51 GMT
< transfer-encoding: chunked
* Ignoring Transfer-Encoding in CONNECT 200 response
< set-cookie: SERVERID=proxy01; path=/
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
[ ... ]
< content-type: application/json
< server: Apache/2.4.58 () OpenSSL/1.0.2k-fips
< access-control-allow-origin: *
< date: Sun, 10 Dec 2023 20:40:52 GMT
< cache-control: no-cache, private
< etag: "02c1253400c6505230e952333558acaf"
< vary: Accept-Encoding
< x-cache: RefreshHit from cloudfront
< via: 1.1 55545918b0c914bb8f5282930649df4c.cloudfront.net (CloudFront)
< x-amz-cf-pop: IAD12-P2
< x-amz-cf-id: JH2v7mX2QhjDsAqf-bvLMBzG93kyFojE17eoYpBpAr6WBds3W47ULg==
<
{"preference":null,"pagination":{"total":7346,"limit":10,"offset":0,"total_pages":735,"current_page":1},"data":[{"_score":135.74696,"thumbnail":{"alt_text":"A bronze lion, deep green and muscular, looks out in the distance from its pedestal in front of the Art Institute of Chicago." [ ... ]
I can run a telegraf --test and get data:
# telegraf --debug --test --config testing_http_temp_trash.conf
2023-12-10T20:16:42Z I! Loading config: testing_http_temp_trash.conf
2023-12-10T20:16:42Z I! Starting Telegraf 1.28.5 brought to you by InfluxData the makers of InfluxDB
2023-12-10T20:16:42Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
2023-12-10T20:16:42Z I! Loaded inputs: http
2023-12-10T20:16:42Z I! Loaded aggregators:
2023-12-10T20:16:42Z I! Loaded processors:
2023-12-10T20:16:42Z I! Loaded secretstores:
2023-12-10T20:16:42Z W! Outputs are not used in testing mode!
2023-12-10T20:16:42Z I! Tags enabled: host=ingest001
2023-12-10T20:16:42Z D! [agent] Initializing plugins
2023-12-10T20:16:42Z D! [agent] Starting service inputs
2023-12-10T20:16:42Z D! [agent] Stopping service inputs
2023-12-10T20:16:42Z D! [agent] Input channel closed
2023-12-10T20:16:42Z D! [agent] Stopped Successfully
> http,host=ingest001,url=https://api.artic.edu/api/v1/artworks/search?q\=cats data_0__score=135.76062,data_0_id=656,data_0_thumbnail_height=5620,data_0_thumbnail_width=8430,data_1__score=119.26675,data_1_id=117241,data_1_thumbnail_height=7661,data_1_thumbnail_width=6486,data_2__score=115.99253,data_2_id=45259,data_2_thumbnail_ [ ... ]
testing_http_temp_trash.conf
[[inputs.http]]
urls = [ "https://api.artic.edu/api/v1/artworks/search?q=cats" ]
interval = "35s"
use_system_proxy = true
timeout = "15s"
data_format = "json"
telegraf.conf
[global_tags]
[agent]
interval = "10s"
debug = false
hostname = "ingest001.somewhere.io"
round_interval = true
flush_interval = "10s"
flush_jitter = "0s"
collection_jitter = "0s"
metric_batch_size = 1000
metric_buffer_limit = 10000
quiet = false
logfile = ""
omit_hostname = false
[[outputs.influxdb]]
urls = ["https://influxdb.somewhere.io:8086"]
username = "influx_telegraf"
password = "hellothere!"
insecure_skip_verify = true
database = "telegraf"
