Why is telegraf logging a 401 response from elasticsearch when using a user-authenticated SSL connection?

Telegraf
1

I am configuring the elasticsearch input for telegraf (version 1.25.0), and running into an issue where telegraf is continually logging

[inputs.elasticsearch] Error in plugin: elasticsearch: API responded with status-code 401, expected 200

The configuration for the elasticsearch input plugin looks like this:

# Read stats from one or more Elasticsearch servers or clusters
[[inputs.elasticsearch]]
  ## specify a list of one or more Elasticsearch servers
  # you can add username and password to your url to use basic authentication:
  # servers = ["http://user:pass@localhost:9200"]
  servers = ["https://elastic.server:9200"]

  ## Timeout for HTTP requests to the elastic search server(s)
  http_timeout = "5s"

  ## When local is true (the default), the node will read only its own stats.
  ## Set local to false when you want to read the node stats from all nodes
  ## of the cluster.
  local = false

  ## Set cluster_health to true when you want to also obtain cluster health stats
  cluster_health = true

  ## Adjust cluster_health_level when you want to also obtain detailed health stats
  ## The options are
  ##  - indices (default)
  ##  - cluster
  # cluster_health_level = "indices"

  ## Set cluster_stats to true when you want to also obtain cluster stats from the
  ## Master node.
  cluster_stats = true

  ## node_stats is a list of sub-stats that you want to have gathered. Valid options
  ## are "indices", "os", "process", "jvm", "thread_pool", "fs", "transport", "http",
  ## "breaker". Per default, all stats are gathered.
  # node_stats = ["jvm", "http"]

  ## Optional TLS Config
  tls_ca = "/etc/telegraf/ca.crt"
  tls_cert = "/etc/telegraf/clientcert.crt"
  tls_key = "/etc/telegraf/clientcert.key"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  username = "telegraf_user"
  password = "telegraf_user_password"

I have also attempted putting the user/password information in the url configured in the servers setting.

I have verified that I can manually connect to elasticsearch using the provider user info and certificates (via curl).

Is there a way for me to get insight into what exact calls telegraf is making so I can debug this a little bit better?

The curl command I am trying is:

curl --cacert /etc/telegraf/ca.crt --cert /etc/telegraf/clientcert.crt --key /etc/telegraf/clientcert.key https://telegraf_user:telefraf_user_password@elkdevn1.dev.oati.local:9200/_nodes/_local/name

To which I get a response like:

{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elkcluster","nodes":{"nodeitentifier":{"name":"elastic.server","transport_address":"10.100.2.128:9300","host":"10.100.2.128","ip":"10.100.2.128","version":"7.16.2","build_flavor":"default","build_type":"rpm","build_hash":"2b937c44140b6559905130a8650c64dbd0879cfb","roles":["data","ingest","master"],"attributes":{"xpack.installed":"true","transform.node":"false"}}}}
2

Is there a way for me to get insight into what exact calls telegraf is making so I can debug this a little bit better?

Telegraf is connecting, but getting told by elasticsearch that it is unauthorized (401). If it were something with the certs my thought is that you would get a different error message, focused around the connection.

Did you get the same 401 with the username password in the URL and not in the URL? You used https for both as well I ask since your commented out one is http.

If you use --user <username:password> with curl does that work?