Telegraf http output plugin connection reset

Telegraf
1

Hello Folks,

I have telegraf running as a pod using a client certificate which expires every 1 hour. this cert is mounted as secret to my telegraf deployment. telegraf sends the metrics using http output plugin to envoy based proxy where the client cert validation and TLS termination occurs.

I’m observing the following behaviour

  1. when the cert rotates the secret is updated and the same is visible in the container file system
  2. For some reason, the proxy pod is restarted, the flow of metrics from telegraf to proxy is interrupted with expired cert error
  3. at this point, the certs in the secret mounted on telegraf is updated with a valid cert, the cert is also reflected in the container file system

The only path forward seems to be to restart telegraf pod. Is there a way to force telegraf to reset the connection or is there a better way of handling this instead of restarting the pod.

2

Any chance you share the redacted configuration of your http output? How do you pass in the certificate?

3 
[outputs]
  [outputs.http]
    url = "someurl"
    method = "POST"
    tls_ca = "/etc/http-plugin-tls-config/ca.crt"
    tls_cert = "/etc/http-plugin-tls-config/tls.crt"
    tls_key = "/etc/http-plugin-tls-config/tls.key"
    tls_server_name = "foo.bar.svc.*.*"
    [outputs.http.headers]
      content-type = "text/plain; charset=utf-8"

tls.crt, tls.key are part of a secret mounted in the telegraf deployment

volumes:
- name: tls-config
secret:
defaultMode: 420
secretName: tls-config

mountPath: /etc/http-plugin-tls-config
name: tls-config
readOnly: true