Telegraf attempts and fails writing to [http://localhost:8086], but localhost isn't in config

Telegraf
,
1

Hello,

I am having a trivial bug that I wanted to preset. :wave: I am using telegraf (1.25.3, but same issue on 1.22.1) on Windows 10.

I write to a remote influxdb via http configured in the outputs section. As shown below local host is not configured as an output, unless I am missing something. However in the logs, I constantly get failures to write to localhost.

The weirdest part is that the data ends up in the right influx database and shows just fine in Grafana. I have no idea where the localhost is coming from.

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply surround
# them with ${}. For strings the variable must be within quotes (ie, "${STR_VAR}"),
# for numbers and booleans they should be plain (ie, ${INT_VAR}, ${BOOL_VAR})


# Global tags can be specified here in key="value" format.
[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  # user = "$USER"


# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## Maximum number of unwritten metrics per output.  Increasing this value
  ## allows for longer periods of output downtime without dropping metrics at the
  ## cost of higher maximum memory usage.
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Log at debug level.
   debug = true
  ## Log only error level messages.
  # quiet = false

  ## Log target controls the destination for logs and can be one of "file",
  ## "stderr" or, on Windows, "eventlog".  When set to "file", the output file
  ## is determined by the "logfile" setting.
   logtarget = "file"

  ## Name of the file to be logged to when using the "file" logtarget.  If set to
  ## the empty string then logs are written to stderr.
   logfile = "C:\\Program Files\\Telegraf\\telegraf.log"

  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.  Logs are rotated only when
  ## written to, if there is no log activity rotation may be delayed.
  # logfile_rotation_interval = "0d"

  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  # logfile_rotation_max_size = "0MB"

  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  # logfile_rotation_max_archives = 5

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################


# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
      urls = ["http://10.0.20.45:8086"]

  ## The target database for metrics; will be created as needed.
  ## For UDP url endpoint database needs to be configured on server side.
   database = "BattleStation"

  ## If true, no CREATE DATABASE queries will be sent.  Set to true when using
  ## Telegraf with a user without permissions to create databases or when the
  ## database already exists.
   skip_database_creation = true

  ## Name of existing retention policy to write to.  Empty string writes to
  ## the default retention policy.  Only takes effect when using HTTP.
   retention_policy = ""

  ## Timeout for HTTP messages.
  # timeout = "5s"

  ## HTTP Basic Auth
   username = "*********"
   password = "*******"

  ## HTTP User-Agent
   user_agent = "telegraf"

  ## UDP payload size is the maximum packet size to send.
   udp_payload = "512B"

2023-03-07T21:44:02Z I! Starting Telegraf 1.25.3
2023-03-07T21:44:02Z I! Available plugins: 210 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2023-03-07T21:44:02Z I! Loaded inputs: win_perf_counters (3x)
2023-03-07T21:44:02Z I! Loaded aggregators: 
2023-03-07T21:44:02Z I! Loaded processors: 
2023-03-07T21:44:02Z I! Loaded secretstores: 
2023-03-07T21:44:02Z I! Loaded outputs: influxdb (3x)
2023-03-07T21:44:02Z I! Tags enabled: host=Battlestation
2023-03-07T21:44:02Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"Battlestation", Flush Interval:10s
2023-03-07T21:44:02Z D! [agent] Initializing plugins
2023-03-07T21:44:02Z D! [agent] Connecting outputs
2023-03-07T21:44:02Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-07T21:44:02Z D! [agent] Successfully connected to outputs.influxdb
2023-03-07T21:44:02Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-07T21:44:02Z D! [agent] Successfully connected to outputs.influxdb
2023-03-07T21:44:02Z D! [agent] Attempting connection to [outputs.influxdb]
2023-03-07T21:44:02Z D! [agent] Successfully connected to outputs.influxdb
2023-03-07T21:44:02Z D! [agent] Starting service inputs
2023-03-07T21:44:12Z D! [outputs.influxdb] Wrote batch of 429 metrics in 23.0855ms
2023-03-07T21:44:12Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:12Z D! [outputs.influxdb] Wrote batch of 429 metrics in 29.6106ms
2023-03-07T21:44:12Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:14Z E! [outputs.influxdb] When writing to [http://localhost:8086]: failed doing req: Post "http://localhost:8086/write?db=BattleStation": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2023-03-07T21:44:14Z D! [outputs.influxdb] Buffer fullness: 429 / 10000 metrics
2023-03-07T21:44:14Z E! [agent] Error writing to outputs.influxdb: could not write any address
2023-03-07T21:44:22Z D! [outputs.influxdb] Wrote batch of 429 metrics in 19.6201ms
2023-03-07T21:44:22Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:22Z D! [outputs.influxdb] Wrote batch of 429 metrics in 26.2594ms
2023-03-07T21:44:22Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:24Z E! [outputs.influxdb] When writing to [http://localhost:8086]: failed doing req: Post "http://localhost:8086/write?db=BattleStation": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2023-03-07T21:44:24Z D! [outputs.influxdb] Buffer fullness: 858 / 10000 metrics
2023-03-07T21:44:24Z E! [agent] Error writing to outputs.influxdb: could not write any address
2023-03-07T21:44:32Z D! [outputs.influxdb] Wrote batch of 429 metrics in 20.4369ms
2023-03-07T21:44:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:32Z D! [outputs.influxdb] Wrote batch of 429 metrics in 20.9518ms
2023-03-07T21:44:32Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:34Z E! [outputs.influxdb] When writing to [http://localhost:8086]: failed doing req: Post "http://localhost:8086/write?db=BattleStation": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2023-03-07T21:44:34Z D! [outputs.influxdb] Buffer fullness: 1287 / 10000 metrics
2023-03-07T21:44:34Z E! [agent] Error writing to outputs.influxdb: could not write any address
2023-03-07T21:44:42Z D! [outputs.influxdb] Wrote batch of 429 metrics in 22.7816ms
2023-03-07T21:44:42Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:42Z D! [outputs.influxdb] Wrote batch of 429 metrics in 27.2103ms
2023-03-07T21:44:42Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2023-03-07T21:44:44Z E! [outputs.influxdb] When writing to [http://localhost:8086]: failed doing req: Post "http://localhost:8086/write?db=BattleStation": dial tcp [::1]:8086: connectex: No connection could be made because the target machine actively refused it.
2

Telegraf sees three influxdb configuration stanzas and is attempting to use all three.

How are you running influxdb? via service? are you providing a list of config files?

3

I have telegraf running as a service on Windows 10 and influx running on a separate Ubuntu machine.

Telegraf runs with the following config "C:\Program Files\telegraf\telegraf.exe" --config "C:\Program Files\Telegraf\telegraf.conf" --config-directory "C:\Program Files\telegraf" --service-name telegraf

I had a ‘old’ folder in the telegraf directory, from a previous update, that had another config file in it. I removed that and now I am down to influxdb 2x and the local host issue is gone.

4

–config “C:\Program Files\Telegraf\telegraf.conf” --config-directory “C:\Program Files\telegraf”

You are pointing the config-directory param at the same directory that your config is in, which means it will read that file twice.

5

That fixed it.

I guess I used this to set up the service and thought they were both necessary. telegraf/WINDOWS_SERVICE.md at master · influxdata/telegraf · GitHub

I ended up having to uninstall the service, reboot and reinstall with only one config flag. I tried to use sc config telegraf binPath= “C:\Program Files\Telegraf\telegraf.conf” --config-directory “C:\Program Files\telegraf”

But it kept erroring out.

6

There is a slight difference in that on that page it shows the config.d directory:

--config-directory C:\"Program Files"\Telegraf\telegraf.d

This is similar to what we use on the unix/linux systems:

$ tree /etc/telegraf/
/etc/telegraf/
├── telegraf.conf
├── telegraf.conf.sample
└── telegraf.d