[solved] Why is $HOSTNAME not replaced with its content when using in global tags?

rdxmb · October 6, 2017, 6:38pm

Hey all over there,
I have an unexpected behavior when using $HOSTNAME as a tag in telegraf:
So $HOSTNAME is used as string, although other env vars are replaced by their content.

root@docker2:~# head /etc/telegraf/telegraf.conf
[global_tags]
  datacenter = "$DATACENTER"
  fqdn = "$FQDN"
  location = "$LOCATION"
  engine = "$HOSTNAME"

# Configuration for telegraf agent
[agent]
  interval = "15s"
  round_interval = true
root@docker2:~# telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d/ --input-filter cpu --test
* Plugin: inputs.cpu, Collection 1
* Plugin: inputs.cpu, Collection 2
> cpu,location=development,host=docker2,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,cpu=cpu0 usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_idle=98.03921568448419,usage_iowait=0,usage_system=1.9607843136879006,usage_nice=0 1507314594000000000
> cpu,cpu=cpu1,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2 usage_system=0,usage_iowait=0,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_idle=100,usage_nice=0,usage_softirq=0,usage_steal=0 1507314594000000000
> cpu,engine=$HOSTNAME,cpu=cpu2,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2 usage_nice=0,usage_iowait=0,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_system=0,usage_steal=0,usage_idle=100,usage_softirq=0 1507314594000000000
> cpu,host=docker2,cpu=cpu3,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development usage_nice=0,usage_system=0,usage_idle=100,usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_user=0,usage_iowait=0 1507314594000000000
> cpu,engine=$HOSTNAME,datacenter=mbr,fqdn=docker2.mbr.intern,location=development,host=docker2,cpu=cpu-total usage_iowait=0,usage_user=0,usage_idle=99.50248756460951,usage_irq=0,usage_softirq=0,usage_steal=0,usage_guest=0,usage_guest_nice=0,usage_system=0.4975124378339072,usage_nice=0 1507314594000000000
root@docker2:~# echo $HOSTNAME
docker2

root@docker2:~# telegraf version
Telegraf v1.4.1 (git: release-1.4 2de7aa23d7d3c3bcf639d417129ab7c17d83399b)

Telegraf is running natively on the docker host here and not in a docker container.

//edited:

When using without " , an error occures, which is expected, of course:

root@docker2:~# head /etc/telegraf/telegraf.conf
[global_tags]
  datacenter = "$DATACENTER"
  fqdn = "$FQDN"
  location = "$LOCATION"
  engine = $HOSTNAME

# Configuration for telegraf agent
[agent]
  interval = "15s"
  round_interval = true
root@docker2:~# telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d/ --input-filter cpu --test
2017/10/06 20:43:33 E! Error parsing /etc/telegraf/telegraf.conf, toml: line 5: parse error

//edited:

Just commented the (very old) issue here: Hostname is not passed as a tag · Issue #29 · influxdata/telegraf · GitHub

rdxmb · October 6, 2017, 7:11pm

Oh, the variables are only replaced with its content when using the --test parameter in telegraf.
When writing to influxdb the result looks like this:

> select * from cpu
name: cpu
time                cpu       datacenter  engine    fqdn  host    location  usage_guest usage_guest_nice usage_idle        usage_iowait         usage_irq usage_nice usage_softirq       usage_steal usage_system       usage_user
----                ---       ----------  ------    ----  ----    --------  ----------- ---------------- ----------        ------------         --------- ---------- -------------       ----------- ------------       ----------
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.26540922865132 0                    0         0          0.03368137420019253 0           1.3304142809011912 0.37049511620077763
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.44908968300591 0                    0         0          0                   0           1.0114632501673122 0.5394470667558743
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.80370624566348 0                    0         0          0.06863417982145581 0           1.3726835964306767 0.7549759780344533
1507316925000000000 cpu3      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.18059299188765 0                    0         0          0                   0           0.8760107816712416 0.9433962264135333
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.91891891889553 0                    0         0          0                   0           0.8108108108106821 0.2702702702703554
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                96.98836413413888 0.06844626967828618  0         0          0.13689253935666965 0           1.8480492813139702 0.9582477754953258
1507316925000000000 cpu2      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.25268817207876 0                    0         0          0.06720430107518667 0           1.5456989247308213 0.13440860215037334
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.65319865321746 0                    0         0          0                   0           1.0774410774414966 0.26936026936018276
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.73195876287505 0                    0         0          0.20618556701027585 0           1.1683848797253522 0.8934707903754529
1507316925000000000 cpu1      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.51451721808925 0                    0         0          0                   0           1.2153950033739886 0.27008777852755306
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.51752021562523 0                    0         0          0.06738544474393776 0           1.3477088948789469 0.06738544474427292
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.52577319587279 0                    0         0          0.0687285223366935  0           1.5807560137455132 0.824742268040322
1507316925000000000 cpu0      $DATACENTER $HOSTNAME $FQDN docker1 $LOCATION 0           0                98.11320754717721 0                    0         0          0                   0           1.6846361185987793 0.20215633423166965
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker3 $LOCATION 0           0                98.61766689142466 0                    0         0          0.03371544167229919 0           1.0451786918409394 0.30343897505026146
1507316925000000000 cpu-total $DATACENTER $HOSTNAME $FQDN docker2 $LOCATION 0           0                97.22982216133302 0.034199726402183134 0         0          0.1538987688097998  0           1.6928864569068256 0.8891928864539426

daniel · October 6, 2017, 9:27pm

I just checked and it works for me in either case. Double check that those environment variables are set when running without --test, don’t forget that if it is running as a service it will have a different environment.

rdxmb · October 6, 2017, 10:40pm

Hi daniel,
thanks for your answer.

Good advice with the environment, just rebootet the machine because the env vars were set within the last uptime.
I had set them in /etc/profile.d/XY and was sure they were read from there…

Just checking the env vars with the telegraf user:

root@docker1:~# su telegraf 
root@docker1:~# getent passwd telegraf 
telegraf:x:999:998::/etc/telegraf:/bin/false
root@docker1:~# sudo -u telegraf whoami
telegraf
root@docker1:~# sudo -u telegraf echo $SHELL
/bin/bash
root@docker1:~# sudo -u telegraf echo $HOSTNAME
docker1
root@docker1:~# sudo -u telegraf echo $FQDN
docker1.mbr.intern
root@docker1:~# sudo -u telegraf echo $LOCATION
development

Seems like the environment within systemd is REALLY different, as I am reading here:

github.com/coreos/bugs

Setting global env vars that take affect by systemd units?

opened 09:03PM - 09 Aug 16 UTC

closed 09:12PM - 15 Aug 16 UTC

asaf

kind/question component/systemd area/usability team/os

Hey, I probably miss something about Systemd here, I have a systemd file such:… ``` bash sudo cat /etc/systemd/system/flanneld.service.d/50-network-config.conf [Service] ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }' ``` Which fails to start due to: ``` bash Aug 09 20:42:30 core-01 etcdctl[1184]: Error: malformed HTTP response "\x15\x03\x01\x00\x02\x02" Aug 09 20:42:30 core-01 systemd[1]: flanneld.service: Control process exited, code=exited status=4 ``` This fails because `ETCDCTL_***` env vars are missing, I put the relevant etcdctl config params in `/etc/profile.d/01-etcd.sh`, also tried in `/etc/environment`, ``` bash cat /etc/environment export ETCDCTL_CA_FILE=/etc/certs/ca.pem export ETCDCTL_CERT_FILE=/etc/certs/coreos.pem export ETCDCTL_KEY_FILE=/etc/certs/coreos-key.pem export ETCDCTL_ENDPOINT=https://127.0.0.1:2379 ``` These exported fine in interactive session and works just fine: ``` bash core@core-01 ~ $ etcdctl ls /coreos.com ``` But seems like these env vars have no effect in systemd units, I also tried to put these vars in `/etc/systemd/system.conf` as a value of the `DefaultEnvironment` field. Is this a bug ? alternatively, what is the official approach for setting up env vars that should take affect on systemd units globally? Thanks. ### CoreOS Version ``` NAME=CoreOS ID=coreos VERSION=1122.0.0 VERSION_ID=1122.0.0 BUILD_ID=2016-07-27-0739 PRETTY_NAME="CoreOS 1122.0.0 (MoreOS)" ANSI_COLOR="1;32" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://github.com/coreos/bugs/issues" ``` ### Environment Using official vagrant file to raise the VMs

I have never heard about that. At least, $HOSTNAME had to be there, I thought.

@daniel Could you please tell me:
When you were testing it, where did you define your env vars before? Is telegraf also running with systemd at your side?

//edited: the env-file for telegraf seems to be

root@docker1:/lib/systemd/system# grep Environment . -R | grep telegraf
./telegraf.service:EnvironmentFile=-/etc/default/telegraf

If we have to define our vars (e.g. for tags) there, this should be documented. What do you think?
(maybe I am wrong the whole time. Just searching for the way to define globals vars in ubuntu and use them with telegraf…)

daniel · October 7, 2017, 12:00am

I added a mention of this file to the configuration documentation.

rdxmb · October 9, 2017, 9:11am

thanks, daniel !
…

Topic		Replies	Views
There is a way to dynamically add global tags on telegraf data? Telegraf telegraf	1	697	July 8, 2020
Show tag is not showing all the tag values Telegraf	5	1373	May 8, 2019
Multiple inputs.http Telegraf	2	3197	May 8, 2019
One Extra columns gets added in the measurement while using file plugin	4	334	February 4, 2021
Telegraf: Cannot Get strings.replace To Work Telegraf telegraf	2	1719	October 21, 2019

[solved] Why is $HOSTNAME not replaced with its content when using in global tags?

Related Topics