Hi,
Is it possible to specify the HTTP version in the InfluxDB output plugin?
Interesting question!
The plugin uses HTTP/1.1 by default. What’s the rationale for changing it?
I don’t see any way in the default behaviour of the plugin to change it.
One possible workaround however, would be to send the requests via a proxy, and translate to some other HTTP version there?
## HTTP Proxy override, if unset values the standard proxy environment
## variables are consulted to determine which proxy, if any, should be used.
# http_proxy = "http://corporate.proxy:3128"
Thanks @popey for the suggestion.
Let me give more context on this.
We have hosted our InfluxDB server in AWS environment and recently enabled WAF protection for our Load Balancer. After WAF was enabled we saw a spike in 400 error with message,
400 Bad Request: invalid byte in chunk length
While debugging this issue with AWS help, they told about a know issue when WAF is enabled
→ Using ALB + WAF
AND
→ using HTTP/2 version on the front end and sending request with chunked transfer encoding without specifying the chunk size.In other words, client using HTTP/2 connection on the front end without specifying Content-Length header in request may send request to backend using chunked transfer encoding without specifying the chunk size. This happens intermittently and may lead to backend sending 400 error and sometimes ALB 504 error(when backend keeps waiting for more data eventually leading to ALB 504 error). The issue is only seen when clients use HTTP/2 to send POST request with data payload but missing Content-Length header.
I tried to check the Telegraf source code to confirm the HTTP version used, but I couldn’t find it. Neither I could confirm chunked Transfer encoding was used.
AWS suggested three options
Disable HTTP/2 on ALB and/or configure client to use HTTP/1.1 only.
Configure client to include Content-Length header in requests having data payload, if possible.
If clients only use HTTP/2 and HTTP/1.1 is not required by any client, use HTTP/2 on backend and change the target group protocol to HTTP/2. Having end to end HTTP/2 shall not have this issue.
This question was to check if it is possible to enforce HTTP 1.1 version from telegraf end.
Instead we disabled HTTP v2 support in Load Balancer. After we disabled the HTTP Version 2 support 400 errors stopped.
Thanks for the explanation.
I think telegraf/http.go at master · influxdata/telegraf · GitHub is where the transport used by the InfluxDB output plugin is set. Looks to my (new, and untrained) eye that it’s using http.Transport which implies HTTP/1.1 rather than http2.transport which may imply HTTP/2. Perhaps further down the stack in one of the vendored modules, it’s using http2 by default?
It looks like it might be based on the net/http package. I found this blurb in the docs.
"Starting with Go 1.6, the http package has transparent support for the HTTP/2 protocol when using HTTPS. Programs that must disable HTTP/2 can do so by setting Transport.TLSNextProto (for clients) or Server.TLSNextProto (for servers) to a non-nil, empty map. Alternatively, the following GODEBUG environment variables are currently supported:
GODEBUG=http2client=0 # disable HTTP/2 client support
GODEBUG=http2server=0 # disable HTTP/2 server support
GODEBUG=http2debug=1 # enable verbose HTTP/2 debug logs
GODEBUG=http2debug=2 # ... even more verbose, with frame dumps
The GODEBUG variables are not covered by Go’s API compatibility promise.
The http package’s Transport and Server both automatically enable HTTP/2 support for simple configurations. To enable HTTP/2 for more complex configurations, to use lower-level HTTP/2 features, or to use a newer version of Go’s http2 package, import golang.org/x/net/http2 directly and use its ConfigureTransport and/or ConfigureServer functions. Manually configuring HTTP/2 via the golang.org/x/net/http2 package takes precedence over the net/http package’s built-in HTTP/2 support."
Let us know if you are still having problems.
1 Like
