Telegraf service on windows 11 only send partially data, works if executed manually

Hi, I am pretty new to Telegraf/InfluxDB/Grafana and I am getting into it using my own PC, so I am pretty new to this. I gather some information and report it into Influx to build a dashboard.

Telegraf is running as a service and reporting some data including a [[input.exec]] plugin with a ps1 script.

.\telegraf.exe --service install --config ‘C:\Program Files\InfluxData\telegraf\telegraf-1.23.4\telegraf.conf’

The script contains WMI data and is reporting some of the data but not all. I can see the script is executed by telegraf and running once a second.

The strange thing is that telegraf is sending all data from the script if running manually:

.\telegraf.exe --config ‘C:\Program Files\InfluxData\telegraf\telegraf-1.23.4\telegraf.conf’

is is not when running as a service. There is only a single ps1 script that is used so I assume it is not a permission challenge.

How can I best debug this behavior ?

Script part that only works manually:

## Custom stuff here

$ohmf = 'c:\Program Files\InfluxData\telegraf\telegraf-1.23.4\exe\ohm.txt'
$ohm = Get-CimInstance -ClassName Sensor -Namespace 'root\OpenHardwareMonitor' > $ohmf

# CPU temp package: /amdcpu/0/temperature/0
$cputemp=Select-String $ohmf -Pattern "/amdcpu/0/temperature/0" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$cputempvalue=$cputemp.split(":",5)[-1] -replace '\s','' -replace ',','.'

# CPU temp CCD: /amdcpu/0/temperature/4
$cpuccd=Select-String $ohmf -Pattern "/amdcpu/0/temperature/4" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$cpuccdvalue=$cpuccd.split(":",5)[-1] -replace '\s','' -replace ',','.'

# CPU power: /amdcpu/0/power/0
$cpupower=Select-String $ohmf -Pattern "/amdcpu/0/power/0" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$cpupowervalue=$cpupower.split(":",5)[-1] -replace '\s','' -replace ',','.'

# CPU power cores: /amdcpu/0/power/1
$cpupc=Select-String $ohmf -Pattern "/amdcpu/0/power/1" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$cpupcvalue=$cpupc.split(":",5)[-1] -replace '\s','' -replace ',','.'

# GPU Core temp: /atigpu/0/temperature/0
$gputemp=Select-String $ohmf -Pattern "/atigpu/0/temperature/0" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$gputempvalue=$gputemp.split(":",5)[-1] -replace '\s','' -replace ',','.'

# GPU HotSPot: /atigpu/0/temperature/9
$gpuhotspot=Select-String $ohmf -Pattern "/atigpu/0/temperature/9" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$gpuhotspotvalue=$gpuhotspot.split(":",5)[-1] -replace '\s','' -replace ',','.'

# GPU power: /atigpu/0/power/0
$gpupower=Select-String $ohmf -Pattern "/atigpu/0/power/0" -Context 0,10 | out-string -stream | select -first 1 -skip 10
$gpupowervalue=$gpupower.split(":",5)[-1] -replace '\s','' -replace ',','.'

echo "$($metric),host=$($si.Name),metric=cputemp value=$($cputempvalue)"
echo "$($metric),host=$($si.Name),metric=cpucd value=$($cpuccdvalue)"
echo "$($metric),host=$($si.Name),metric=cpupower value=$($cpupowervalue)"
echo "$($metric),host=$($si.Name),metric=cpupc value=$($cpupcvalue)"
echo "$($metric),host=$($si.Name),metric=gputemp value=$($gputempvalue)"
echo "$($metric),host=$($si.Name),metric=gpuhotspot value=$($gpuhotspotvalue)"
echo "$($metric),host=$($si.Name),metric=gpupower value=$($gpupowervalue)"

That’s the first thing I’d check…
When it runs manually it’s using your user… but which user is running the service? does it have proper permission? (the easiest way is to impersonate it)

It is running as System but I switched to my account to test it without success. Also I would be surprised if a single ps1 is having “partially” permission issues. All information are collected from WMI using Get-CimInstance.

Also the output script “ohm.txt” is populated with correct and new information every second when the service is running so the script is executed and all metrics are gathered and written correctly on the local disk but not into the InfluxDB instance.

Can you share your input and output config then?
I’d expect some output errors tho.

Here is what a test gives me.

Is there any good way to get output from telegraf when running as a service ? I cannot find any log / output and the “logfile” option in telegraf.conf always gives me a syntax error.

PS C:\Program Files\InfluxData\telegraf\telegraf-1.23.4> .\telegraf.exe --config .\telegraf_custom.conf --test
.\telegraf.exe : 2022-09-08T07:12:11Z I! Starting Telegraf 1.23.4
In Zeile:1 Zeichen:1
+ .\telegraf.exe --config .\telegraf_custom.conf --test
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022-09-08T07:1...Telegraf 1.23.4:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2022-09-08T07:12:11Z I! Loaded inputs: exec
2022-09-08T07:12:11Z I! Loaded aggregators: 
2022-09-08T07:12:11Z I! Loaded processors: 
2022-09-08T07:12:11Z W! Outputs are not used in testing mode!
2022-09-08T07:12:11Z I! Tags enabled: host=phil-pc
2022-09-08T07:12:11Z D! [agent] Initializing plugins
2022-09-08T07:12:11Z D! [agent] Starting service inputs
2022-09-08T07:12:12Z D! [agent] Stopping service inputs
2022-09-08T07:12:12Z D! [agent] Input channel closed
2022-09-08T07:12:12Z D! [agent] Stopped Successfully
> winsys,host=PHIL-PC,metric=domain strval="WORKGROUP" 1662621132000000000
> winsys,host=PHIL-PC,metric=mem_total value=31.92 1662621132000000000
> winsys,host=PHIL-PC,metric=mem_free value=26.07 1662621132000000000
> winsys,host=PHIL-PC,metric=mem_used value=5.85 1662621132000000000
> winsys,host=PHIL-PC,metric=mem_used_percent value=18.3270676691729 1662621132000000000
> winsys,device=C:,host=PHIL-PC,metric=disk_total value=243.42 1662621132000000000
> winsys,device=C:,host=PHIL-PC,metric=disk_free value=186.87 1662621132000000000
> winsys,device=C:,host=PHIL-PC,metric=disk_used value=56.55 1662621132000000000
> winsys,device=C:,host=PHIL-PC,metric=disk_used_percent value=23.2314518116835 1662621132000000000
> winsys,device=D:,host=PHIL-PC,metric=disk_total value=687.37 1662621132000000000
> winsys,device=D:,host=PHIL-PC,metric=disk_free value=557.65 1662621132000000000
> winsys,device=D:,host=PHIL-PC,metric=disk_used value=129.72 1662621132000000000
> winsys,device=D:,host=PHIL-PC,metric=disk_used_percent value=18.8719321471697 1662621132000000000
> winsys,device=E:,host=PHIL-PC,metric=disk_total value=465.75 1662621132000000000
> winsys,device=E:,host=PHIL-PC,metric=disk_free value=465.5 1662621132000000000
> winsys,device=E:,host=PHIL-PC,metric=disk_used value=0.25 1662621132000000000
> winsys,device=E:,host=PHIL-PC,metric=disk_used_percent value=0.0536768652710682 1662621132000000000
> winsys,host=PHIL-PC,metric=windows strval="Microsoft Windows 11 Pro" 1662621132000000000
> winsys,host=PHIL-PC,metric=cputemp value=49.75 1662621132000000000
> winsys,host=PHIL-PC,metric=cpucd value=55.25 1662621132000000000
> winsys,host=PHIL-PC,metric=cpupower value=37.58681 1662621132000000000
> winsys,host=PHIL-PC,metric=cpupc value=9.543147 1662621132000000000
> winsys,host=PHIL-PC,metric=gputemp value=33 1662621132000000000
> winsys,host=PHIL-PC,metric=gpuhotspot value=40 1662621132000000000
> winsys,host=PHIL-PC,metric=gpupower value=9 1662621132000000000

PS C:\Program Files\InfluxData\telegraf\telegraf-1.23.4>

see logtarget and logfile here

put the path in single quotes to avoid escaping errors, also the user must have permission to write the specified log file

okay, so the service collects a different value as telegraf.exe when running as a user. This is why I do not get the full response.

Will have a look why I get a different result running the same code as a service and from user space.

.\telegraf.exe --service install --service-name telegraf-2 --config ‘C:\Program Files\InfluxData\telegraf\telegraf-1.23.4\telegraf_custom.conf’

2022-09-08T07:47:46Z I! Loaded inputs: exec
2022-09-08T07:47:46Z I! Loaded aggregators: 
2022-09-08T07:47:46Z I! Loaded processors: 
2022-09-08T07:47:46Z I! Loaded outputs: influxdb
2022-09-08T07:47:46Z I! Tags enabled: host=phil-pc
2022-09-08T07:47:46Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"phil-pc", Flush Interval:10s
2022-09-08T07:47:46Z D! [agent] Initializing plugins
2022-09-08T07:47:46Z D! [agent] Connecting outputs
2022-09-08T07:47:46Z D! [agent] Attempting connection to [outputs.influxdb]
2022-09-08T07:47:46Z D! [agent] Successfully connected to outputs.influxdb
2022-09-08T07:47:46Z D! [agent] Starting service inputs
2022-09-08T07:47:47Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:48Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:49Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:50Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:51Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:52Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:53Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"
2022-09-08T07:47:54Z E! [inputs.exec] Error in plugin: metric parse error: expected field at 19:43: "winsys,host=PHIL-PC,metric=cputemp value=Temperature"