[solved] Why is $HOSTNAME not replaced with its content when using in global tags?

influxdb
#1

Hey all over there,
I have an unexpected behavior when using $HOSTNAME as a tag in telegraf:
So $HOSTNAME is used as string, although other env vars are replaced by their content.

root@docker2:~# head /etc/telegraf/telegraf.conf
[global_tags]
  datacenter = "$DATACENTER"
  fqdn = "$FQDN"
  location = "$LOCATION"
  engine = "$HOSTNAME"

# Configuration for telegraf agent
[agent]
  interval = "15s"
  round_interval = true
root@docker2:~# telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d/ --input-filter cpu --test
* Plugin: inputs.cpu, Collection 1
* Plugin: inputs.cpu, Collection 2
> cpu,location=development,host=docker2,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,cpu=cpu0 usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_idle=98.03921568448419,usage_iowait=0,usage_system=1.9607843136879006,usage_nice=0 1507314594000000000
> cpu,cpu=cpu1,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2 usage_system=0,usage_iowait=0,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_idle=100,usage_nice=0,usage_softirq=0,usage_steal=0 1507314594000000000
> cpu,engine=$HOSTNAME,cpu=cpu2,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2 usage_nice=0,usage_iowait=0,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_system=0,usage_steal=0,usage_idle=100,usage_softirq=0 1507314594000000000
> cpu,host=docker2,cpu=cpu3,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development usage_nice=0,usage_system=0,usage_idle=100,usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_iowait=0 1507314594000000000
> cpu,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2,cpu=cpu-total usage_iowait=0,usage_user=0,usage_idle=99.50248756460951,usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_system=0.4975124378339072,usage_nice=0 1507314594000000000
root@docker2:~# echo $HOSTNAME
docker2

root@docker2:~# telegraf version
Telegraf v1.4.1 (git: release-1.4 2de7aa23d7d3c3bcf639d417129ab7c17d83399b)

Telegraf is running natively on the docker host here and not in a docker container.

//edited:

When using without " , an error occures, which is expected, of course:

root@docker2:~# head /etc/telegraf/telegraf.conf
[global_tags]
  datacenter = "$DATACENTER"
  fqdn = "$FQDN"
  location = "$LOCATION"
  engine = $HOSTNAME

# Configuration for telegraf agent
[agent]
  interval = "15s"
  round_interval = true
root@docker2:~# telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d/ --input-filter cpu --test
2017/10/06 20:43:33 E! Error parsing /etc/telegraf/telegraf.conf, toml: line 5: parse error

//edited:

Just commented the (very old) issue here: https://github.com/influxdata/telegraf/issues/29

#2

Oh, the variables are only replaced with its content when using the --test parameter in telegraf.
When writing to influxdb the result looks like this:

> select * from cpu
name: cpu
time                cpu       datacenter  engine    fqdn  host    location  usage_guest usage_guest_nice usage_idle        usage_iowait         usage_irq usage_nice usage_softirq       usage_steal usage_system       usage_user
----                ---       ----------  ------    ----  ----    --------  ----------- ---------------- ----------        ------------         --------- ---------- -------------       ----------- ------------       ----------
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.26540922865132 0                    0         0          0.03368137420019253 0           1.3304142809011912 0.37049511620077763
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.44908968300591 0                    0         0          0                   0           1.0114632501673122 0.5394470667558743
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.80370624566348 0                    0         0          0.06863417982145581 0           1.3726835964306767 0.7549759780344533
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.18059299188765 0                    0         0          0                   0           0.8760107816712416 0.9433962264135333
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.91891891889553 0                    0         0          0                   0           0.8108108108106821 0.2702702702703554
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                96.98836413413888 0.06844626967828618  0         0          0.13689253935666965 0           1.8480492813139702 0.9582477754953258
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.25268817207876 0                    0         0          0.06720430107518667 0           1.5456989247308213 0.13440860215037334
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.65319865321746 0                    0         0          0                   0           1.0774410774414966 0.26936026936018276
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.73195876287505 0                    0         0          0.20618556701027585 0           1.1683848797253522 0.8934707903754529
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.51451721808925 0                    0         0          0                   0           1.2153950033739886 0.27008777852755306
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.51752021562523 0                    0         0          0.06738544474393776 0           1.3477088948789469 0.06738544474427292
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.52577319587279 0                    0         0          0.0687285223366935  0           1.5807560137455132 0.824742268040322
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.11320754717721 0                    0         0          0                   0           1.6846361185987793 0.20215633423166965
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.61766689142466 0                    0         0          0.03371544167229919 0           1.0451786918409394 0.30343897505026146
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.22982216133302 0.034199726402183134 0         0          0.1538987688097998  0           1.6928864569068256 0.8891928864539426
#3

I just checked and it works for me in either case. Double check that those environment variables are set when running without --test, don’t forget that if it is running as a service it will have a different environment.

#4

Hi daniel,
thanks for your answer.

Good advice with the environment, just rebootet the machine because the env vars were set within the last uptime.
I had set them in /etc/profile.d/XY and was sure they were read from there…

Just checking the env vars with the telegraf user:

root@docker1:~# su telegraf 
root@docker1:~# getent passwd telegraf 
telegraf:x:999:998::/etc/telegraf:/bin/false
root@docker1:~# sudo -u telegraf whoami
telegraf
root@docker1:~# sudo -u telegraf echo $SHELL
/bin/bash
root@docker1:~# sudo -u telegraf echo $HOSTNAME
docker1
root@docker1:~# sudo -u telegraf echo $FQDN
docker1.mbr.intern
root@docker1:~# sudo -u telegraf echo $LOCATION
development

Seems like the environment within systemd is REALLY different, as I am reading here:

I have never heard about that. At least, $HOSTNAME had to be there, I thought.

@daniel Could you please tell me:
When you were testing it, where did you define your env vars before? Is telegraf also running with systemd at your side?

//edited: the env-file for telegraf seems to be

root@docker1:/lib/systemd/system# grep Environment . -R | grep telegraf
./telegraf.service:EnvironmentFile=-/etc/default/telegraf

If we have to define our vars (e.g. for tags) there, this should be documented. What do you think?
(maybe I am wrong the whole time. Just searching for the way to define globals vars in ubuntu and use them with telegraf…)

#5

I added a mention of this file to the configuration documentation.

1 Like
#6

thanks, daniel !