Redfish input fails with "forbidden" when run normally, but succeeds when run with "telegraf --debug --once"

Telegraf
1

When using the following redfish conf file:

[[inputs.redfish]]
  address = "https://host1.ourdomain.com/redfish/v1/"
  username = "root"
  password = "ourpassword"
  computer_system_id="System.Embedded.1"
  insecure_skip_verify = true

I get the following error in /var/log/messages:

Nov  3 09:54:00 host01 telegraf[3848243]: 2023-11-03T13:54:00Z E! [inputs.redfish] Error in plugin: Get "https://host1.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden

Granted, the certs on that host are expired and thatā€™s why weā€™re using ā€œinsecure_skip_verify = trueā€

However, if I use the same redfish conf file and run it with ā€œtelegraf --debug --onceā€ it succeeds in pulling the metrics from the node.

Can anyone help me understand why it succeeds with ā€œtelegraf --debug --onceā€ but fails normally?

Thanks.

2

This message is literally saying you are getting a 403 from that URL when running. Meaning Telegraf is starting correctly, making a request to the redfish endpoint, but getting a 403 back from the server itself.

but fails normally?

Define normally. Is this running as a service? Windows? Linux? What version? Are you using the same config file and ensuring you are using the same credentials?

3

Thanks for the response, @jpowers .

To answer your questions:

What I meant by ā€œnormallyā€ is running as a service. Itā€™s running on Linux. Telegraf version 1.28.1. Iā€™m using the same conf file on the same machine, so it get the same credentials. The only difference is that it fails when it runs as a service and succeeds when it is run with ā€œtelegraf --debug --onceā€

4

Also, if I do ā€œcurl -kā€ to the url from the same machine with the same credentials, it succeeds.

5

The only difference is that it fails when it runs as a service and succeeds when it is run with ā€œtelegraf --debug --onceā€

Then clearly there is a difference :wink: itā€™s just a matter of what.

  • The telegraf service runs as the telegraf user versus whatever use you are running as. Can that user run the telegraf command successfully?
  • Are you using a proxy locally, but not for the system? this is probably what I would guess as your proxy is probably saying access denied/forbidden versus the redfish endpoint
6

Thank you. I will check those things and reply once I have an update.

1 Like
7

I liked the proxy idea, but that doesnā€™t seem to be it. Neither the root user (who I had been running telegraf --debug --once as) nor the telegraf user have a proxy set. I donā€™t see any proxy set on the system, and our proxy itself isnā€™t showing any signs of these URLs in its logs.

When I login to the telegraf user account and run telegraf --debug --once it succeeds, just as it had been succeeding for the root user. So I believe that means itā€™s not user account related.

8

When I login to the telegraf user account

You should not be able to log in as the telegraf user as it is a service account, not a user account.

I am at the point of guessing without logs and your config so, can you provide the following:

  1. your config with secrets removed
  2. the full log when running as a service (i.e. journalctl --no-pager --unit telegraf)
  3. the full log when running as the telegraf user with once (i.e. sudo -u telegraf /usr/bin/telegraf --once ...)
  4. the full log when running as the telegraf user without once (i.e. sudo -u telegraf /usr/bin/telegraf ...)
9

Sorry for the confusion - I had temporarily unlocked the telegraf user account so that I could login and manually run the command once. I misunderstood your request - my mistake. I have since put the account back the way it was.

I have followed your steps. Here are the results. They seem consistent with what I had been seeing before.

[global_tags]

[agent]
  hostname = "host1"
  omit_hostname = false
  interval = "1m"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "1m"
  flush_jitter = "0s"
  precision = ""
  logfile = "/var/log/telegraf/telegraf.log"
  debug = false
  quiet = false

[[inputs.redfish]]
  address = "https://host1-obm.ourdomain.com/redfish/v1/"
  username = "root"
  password = "ourrootpassword"
  computer_system_id="System.Embedded.1"
  insecure_skip_verify = true

[[outputs.file]]
files = ["/tmp/telegraf-debugging"]

[root@host1 telegraf]# systemctl start telegraf
[root@host1 telegraf]# cat /var/log/telegraf/telegraf.log 
2023-11-03T19:02:24Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
2023-11-03T19:02:24Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
2023-11-03T19:02:24Z I! Loaded inputs: redfish
2023-11-03T19:02:24Z I! Loaded aggregators: 
2023-11-03T19:02:24Z I! Loaded processors: 
2023-11-03T19:02:24Z I! Loaded secretstores: 
2023-11-03T19:02:24Z I! Loaded outputs: file
2023-11-03T19:02:24Z I! Tags enabled: host=host1
2023-11-03T19:02:24Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
2023-11-03T19:03:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
[root@host1 telegraf]# cat /tmp/telegraf-debugging
[root@host1 telegraf]#

[root@host1 telegraf]# systemctl stop telegraf
[root@host1 telegraf]# rm /var/log/telegraf/telegraf.log 
rm: remove regular file '/var/log/telegraf/telegraf.log'? y
[root@host1 telegraf]# sudo -u telegraf /usr/bin/telegraf --once
[root@host1 telegraf]# cat /var/log/telegraf/telegraf.log 
2023-11-03T19:04:27Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
2023-11-03T19:04:27Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
2023-11-03T19:04:27Z I! Loaded inputs: redfish
2023-11-03T19:04:27Z I! Loaded aggregators: 
2023-11-03T19:04:27Z I! Loaded processors: 
2023-11-03T19:04:27Z I! Loaded secretstores: 
2023-11-03T19:04:27Z I! Loaded outputs: file
2023-11-03T19:04:27Z I! Tags enabled: host=host1
2023-11-03T19:04:28Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-11-03T19:04:28Z I! [agent] Stopping running outputs
[root@host1 telegraf]# cat /tmp/telegraf-debugging                                                                                                                                                                                           
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=CPU1\ Temp,state=Enabled upper_threshold_critical=103,lower_threshold_critical=3,reading_celsius=48 1699038268000000000             
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=CPU2\ Temp,state=Enabled lower_threshold_critical=3,reading_celsius=44,upper_threshold_critical=103 1699038268000000000             
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=2,name=System\ Board\ Inlet\ Temp,state=Enabled lower_threshold_critical=-7,reading_celsius=22,upper_threshold_critical=42 1699038268000000000
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=3,name=System\ Board\ Exhaust\ Temp,state=Enabled reading_celsius=38,upper_threshold_critical=80,lower_threshold_critical=3 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=System\ Board\ Fan1A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038268000000000                                     
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=System\ Board\ Fan1B,state=Enabled reading_rpm=7560i,lower_threshold_critical=480i 1699038268000000000                                     
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=2,name=System\ Board\ Fan2A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038268000000000                                     
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=3,name=System\ Board\ Fan2B,state=Enabled reading_rpm=7560i,lower_threshold_critical=480i 1699038268000000000                                     
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=4,name=System\ Board\ Fan3A,state=Enabled reading_rpm=8160i,lower_threshold_critical=480i 1699038268000000000                                     
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=5,name=System\ Board\ Fan3B,state=Enabled lower_threshold_critical=480i,reading_rpm=7800i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=6,name=System\ Board\ Fan4A,state=Enabled lower_threshold_critical=480i,reading_rpm=8040i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=7,name=System\ Board\ Fan4B,state=Enabled reading_rpm=7680i,lower_threshold_critical=480i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=8,name=System\ Board\ Fan5A,state=Enabled reading_rpm=8160i,lower_threshold_critical=480i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=9,name=System\ Board\ Fan5B,state=Enabled lower_threshold_critical=480i,reading_rpm=7680i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=10,name=System\ Board\ Fan6A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=11,name=System\ Board\ Fan6B,state=Enabled reading_rpm=7680i,lower_threshold_critical=480i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=12,name=System\ Board\ Fan7A,state=Enabled reading_rpm=8160i,lower_threshold_critical=480i 1699038268000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=13,name=System\ Board\ Fan7B,state=Enabled reading_rpm=7560i,lower_threshold_critical=480i 1699038268000000000
redfish_power_powercontrol,address=host1-obm.ourdomain.com,host=host1,member_id=0,name=System\ Power\ Control power_consumed_watts=255,power_requested_watts=1176.71069335938,average_consumed_watts=255,interval_in_min=1i,min_consumed_watts=253,power_allocated_watts=840,power_available_watts=0,power_capacity_watts=1120,max_consumed_watts=281 1699038268000000000
redfish_power_powersupplies,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=PS1\ Status,state=Enabled power_output_watts=113.125,line_input_voltage=204,last_power_output_watts=113.125,power_capacity_watts=800,power_input_watts=126.5 1699038268000000000
redfish_power_powersupplies,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=PS2\ Status,state=Enabled power_input_watts=127.25,power_output_watts=113.375,line_input_voltage=208,last_power_output_watts=113.375,power_capacity_watts=800 1699038268000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=39,name=CPU1\ VCORE\ VR,state=Enabled reading_volts=1.77 1699038268000000000                                                                    
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=40,name=CPU2\ VCORE\ VR,state=Enabled reading_volts=1.77 1699038268000000000                                                                    
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=41,name=CPU1\ MEM0123\ VR,state=Enabled reading_volts=1.21 1699038268000000000                                                                  
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=42,name=CPU1\ MEM4567\ VR,state=Enabled reading_volts=1.21 1699038268000000000                                                                  
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=43,name=CPU2\ MEM0123\ VR,state=Enabled reading_volts=1.21 1699038268000000000                                                                  
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=44,name=CPU2\ MEM4567\ VR,state=Enabled reading_volts=1.21 1699038268000000000                                                                  
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=45,name=PS1\ Voltage\ 1,state=Enabled reading_volts=204 1699038268000000000                                                                     
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=46,name=PS2\ Voltage\ 2,state=Enabled reading_volts=208 1699038268000000000                                                                     
[root@host1 telegraf]#

[root@host1 telegraf]# rm /var/log/telegraf/telegraf.log 
rm: remove regular file '/var/log/telegraf/telegraf.log'? y
[root@host1 telegraf]# rm /tmp/telegraf-debugging
rm: remove regular file '/tmp/telegraf-debugging'? y
[root@host1 telegraf]# sudo -u telegraf /usr/bin/telegraf
2023-11-03T19:11:45Z I! Loading config: /etc/telegraf/telegraf.conf

[From another terminal]
[root@host1 telegraf]# cat /var/log/telegraf/telegraf.log 
2023-11-03T19:11:45Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
2023-11-03T19:11:45Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
2023-11-03T19:11:45Z I! Loaded inputs: redfish
2023-11-03T19:11:45Z I! Loaded aggregators: 
2023-11-03T19:11:45Z I! Loaded processors: 
2023-11-03T19:11:45Z I! Loaded secretstores: 
2023-11-03T19:11:45Z I! Loaded outputs: file
2023-11-03T19:11:45Z I! Tags enabled: host=host1
2023-11-03T19:11:45Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
[root@host1 telegraf]# cat /tmp/telegraf-debugging
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=CPU1\ Temp,state=Enabled reading_celsius=48,upper_threshold_critical=103,lower_threshold_critical=3 1699038721000000000
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=CPU2\ Temp,state=Enabled lower_threshold_critical=3,reading_celsius=44,upper_threshold_critical=103 1699038721000000000
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=2,name=System\ Board\ Inlet\ Temp,state=Enabled reading_celsius=22,upper_threshold_critical=42,lower_threshold_critical=-7 1699038721000000000
redfish_thermal_temperatures,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=3,name=System\ Board\ Exhaust\ Temp,state=Enabled lower_threshold_critical=3,reading_celsius=39,upper_threshold_critical=80 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=System\ Board\ Fan1A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=System\ Board\ Fan1B,state=Enabled lower_threshold_critical=480i,reading_rpm=7560i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=2,name=System\ Board\ Fan2A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=3,name=System\ Board\ Fan2B,state=Enabled lower_threshold_critical=480i,reading_rpm=7680i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=4,name=System\ Board\ Fan3A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=5,name=System\ Board\ Fan3B,state=Enabled lower_threshold_critical=480i,reading_rpm=7800i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=6,name=System\ Board\ Fan4A,state=Enabled lower_threshold_critical=480i,reading_rpm=8040i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=7,name=System\ Board\ Fan4B,state=Enabled lower_threshold_critical=480i,reading_rpm=7680i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=8,name=System\ Board\ Fan5A,state=Enabled lower_threshold_critical=480i,reading_rpm=8280i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=9,name=System\ Board\ Fan5B,state=Enabled lower_threshold_critical=480i,reading_rpm=7800i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=10,name=System\ Board\ Fan6A,state=Enabled reading_rpm=8160i,lower_threshold_critical=480i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=11,name=System\ Board\ Fan6B,state=Enabled lower_threshold_critical=480i,reading_rpm=7680i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=12,name=System\ Board\ Fan7A,state=Enabled lower_threshold_critical=480i,reading_rpm=8160i 1699038721000000000
redfish_thermal_fans,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=13,name=System\ Board\ Fan7B,state=Enabled reading_rpm=7560i,lower_threshold_critical=480i 1699038721000000000
redfish_power_powercontrol,address=host1-obm.ourdomain.com,host=host1,member_id=0,name=System\ Power\ Control interval_in_min=1i,min_consumed_watts=253,power_allocated_watts=840,power_available_watts=0,power_requested_watts=1176.71069335938,max_consumed_watts=281,power_capacity_watts=1120,power_consumed_watts=255,average_consumed_watts=255 1699038721000000000
redfish_power_powersupplies,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=0,name=PS1\ Status,state=Enabled power_input_watts=126.5,power_output_watts=113.5,line_input_voltage=204,last_power_output_watts=113.5,power_capacity_watts=800 1699038721000000000
redfish_power_powersupplies,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=1,name=PS2\ Status,state=Enabled power_input_watts=127.25,power_output_watts=113.875,line_input_voltage=208,last_power_output_watts=113.875,power_capacity_watts=800 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=39,name=CPU1\ VCORE\ VR,state=Enabled reading_volts=1.77 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=40,name=CPU2\ VCORE\ VR,state=Enabled reading_volts=1.77 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=41,name=CPU1\ MEM0123\ VR,state=Enabled reading_volts=1.21 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=42,name=CPU1\ MEM4567\ VR,state=Enabled reading_volts=1.21 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=43,name=CPU2\ MEM0123\ VR,state=Enabled reading_volts=1.21 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=44,name=CPU2\ MEM4567\ VR,state=Enabled reading_volts=1.21 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=45,name=PS1\ Voltage\ 1,state=Enabled reading_volts=204 1699038721000000000
redfish_power_voltages,address=host1-obm.ourdomain.com,health=OK,host=host1,member_id=46,name=PS2\ Voltage\ 2,state=Enabled reading_volts=208 1699038721000000000
[root@host1 telegraf]#
10

Please set debug to true and re-run the service test

[agent]
  ...
  debug = true
11

Here you goā€¦

[root@host1 ~]# systemctl start telegraf
[root@host1 ~]# cat /var/log/telegraf/telegraf.log 
2023-11-03T20:22:56Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
2023-11-03T20:22:56Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
2023-11-03T20:22:56Z I! Loaded inputs: redfish
2023-11-03T20:22:56Z I! Loaded aggregators: 
2023-11-03T20:22:56Z I! Loaded processors: 
2023-11-03T20:22:56Z I! Loaded secretstores: 
2023-11-03T20:22:56Z I! Loaded outputs: file
2023-11-03T20:22:56Z I! Tags enabled: host=host1
2023-11-03T20:22:56Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
2023-11-03T20:22:56Z D! [agent] Initializing plugins
2023-11-03T20:22:56Z D! [agent] Connecting outputs
2023-11-03T20:22:56Z D! [agent] Attempting connection to [outputs.file]
2023-11-03T20:22:56Z D! [agent] Successfully connected to outputs.file
2023-11-03T20:22:56Z D! [agent] Starting service inputs
2023-11-03T20:23:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
[root@host1 ~]# cat /tmp/telegraf-debugging
[root@host1 ~]#

Thanks!

12

I want you to try three things with the service:

  1. Remove logfile from your config to stop logging to a file. This doesnā€™t actually capture some of the initial messages. You can then get the logs from journalctl --no-pager --unit telegraf. I wan you to verify the config file you are loading is correct:
root@j1:~# systemctl start telegraf
root@j1:~# journalctl --no-pager --unit telegraf
Nov 03 20:45:57 j1 systemd[1]: Starting Telegraf...
Nov 03 20:45:57 j1 telegraf[1738]: 2023-11-03T20:45:57Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 03 20:45:57 j1 telegraf[1738]: 2023-11-03T20:45:57Z I! Starting Telegraf 1.28.3 brought to you by InfluxData the makers of InfluxDB
Nov 03 20:45:57 j1 telegraf[1738]: 2023-11-03T20:45:57Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Nov 03 20:45:57 j1 telegraf[1738]: 2023-11-03T20:45:57Z I! Loaded inputs: mem
  1. For the redfish address, technically this asks for an address, not a URL, and the plugin itself will strip it down to the hostname. However, to ensure nothing is going wrong there, please use host1-obm.ourdomain.com
  2. Use an ip address. of the above system

Finally, if that all still does not work, you need to try the following debug build:

In 20-30mins from this message some artifacts will get attached as a comment to that PR. Download those and try it out as well.

13

Thank you @jpowers !

Here are the results that I got from these attempts:

[root@host1 telegraf]# systemctl stop telegraf
[root@host1 telegraf]# systemctl start telegraf
[root@host1 telegraf]# journalctl --no-pager --unit telegraf | tail -n 30
Nov 06 10:52:05 host1.ourdomain.com telegraf[527875]: 2023-11-06T15:52:05Z I! [agent] Hang on, flushing any cached metrics before shutdown
Nov 06 10:52:05 host1.ourdomain.com telegraf[527875]: 2023-11-06T15:52:05Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 10:52:05 host1.ourdomain.com telegraf[527875]: 2023-11-06T15:52:05Z I! [agent] Stopping running outputs
Nov 06 10:52:05 host1.ourdomain.com telegraf[527875]: 2023-11-06T15:52:05Z D! [agent] Stopped Successfully
Nov 06 10:52:05 host1.ourdomain.com systemd[1]: Stopping Telegraf...
Nov 06 10:52:05 host1.ourdomain.com systemd[1]: telegraf.service: Succeeded.
Nov 06 10:52:05 host1.ourdomain.com systemd[1]: Stopped Telegraf.
Nov 06 10:52:08 host1.ourdomain.com systemd[1]: Starting Telegraf...
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loaded inputs: redfish
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loaded aggregators:
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loaded processors:
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loaded secretstores:
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Loaded outputs: file
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! Tags enabled: host=host1
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z D! [agent] Initializing plugins
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z D! [agent] Connecting outputs
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z D! [agent] Attempting connection to [outputs.file]
Nov 06 10:52:08 host1.ourdomain.com systemd[1]: Started Telegraf.
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z D! [agent] Successfully connected to outputs.file
Nov 06 10:52:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:52:08Z D! [agent] Starting service inputs
Nov 06 10:53:00 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:53:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
Nov 06 10:53:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:53:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 10:54:00 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:54:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
Nov 06 10:54:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:54:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 10:55:00 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:55:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
Nov 06 10:55:08 host1.ourdomain.com telegraf[531170]: 2023-11-06T15:55:08Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
[root@host1 telegraf]#

Nov 06 10:59:41 host1.ourdomain.com systemd[1]: Stopped Telegraf.
Nov 06 10:59:41 host1.ourdomain.com systemd[1]: Starting Telegraf...
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loaded inputs: redfish
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loaded aggregators:
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loaded processors:
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loaded secretstores:
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Loaded outputs: file
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! Tags enabled: host=host1
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z D! [agent] Initializing plugins
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z D! [agent] Connecting outputs
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z D! [agent] Attempting connection to [outputs.file]
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z D! [agent] Successfully connected to outputs.file
Nov 06 10:59:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T15:59:41Z D! [agent] Starting service inputs
Nov 06 10:59:41 host1.ourdomain.com systemd[1]: Started Telegraf.
Nov 06 11:00:00 host1.ourdomain.com telegraf[532482]: 2023-11-06T16:00:00Z E! [inputs.redfish] Error in plugin: Get "/redfish/v1/Systems/System.Embedded.1": unsupported protocol scheme ""
Nov 06 11:00:41 host1.ourdomain.com telegraf[532482]: 2023-11-06T16:00:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 11:01:00 host1.ourdomain.com telegraf[532482]: 2023-11-06T16:01:00Z E! [inputs.redfish] Error in plugin: Get "/redfish/v1/Systems/System.Embedded.1": unsupported protocol scheme ""
[root@host1 telegraf]#

Nov 06 11:02:49 host1.ourdomain.com systemd[1]: Stopped Telegraf.
Nov 06 11:02:49 host1.ourdomain.com systemd[1]: Starting Telegraf...
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Starting Telegraf 1.28.1 brought to you by InfluxData the makers of InfluxDB
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 59 outputs, 5 secret-stores
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loaded inputs: redfish
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loaded aggregators:
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loaded processors:
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loaded secretstores:
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Loaded outputs: file
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! Tags enabled: host=host1
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z D! [agent] Initializing plugins
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z D! [agent] Connecting outputs
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z D! [agent] Attempting connection to [outputs.file]
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z D! [agent] Successfully connected to outputs.file
Nov 06 11:02:49 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:02:49Z D! [agent] Starting service inputs
Nov 06 11:02:49 host1.ourdomain.com systemd[1]: Started Telegraf.
Nov 06 11:03:00 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:03:00Z E! [inputs.redfish] Error in plugin: Get "/redfish/v1/Systems/System.Embedded.1": unsupported protocol scheme ""
[root@host1 telegraf]#

Using IP or FQDN:

[root@host1 telegraf]# systemctl restart telegraf
[root@host1 telegraf]# journalctl --no-pager --unit telegraf | tail -n 30
Nov 06 14:34:32 host1.ourdomain.com systemd[1]: telegraf.service: Succeeded.
Nov 06 14:34:32 host1.ourdomain.com systemd[1]: Stopped Telegraf.
Nov 06 14:34:32 host1.ourdomain.com systemd[1]: Starting Telegraf...
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Starting Telegraf 1.29.0-93471676 brought to you by InfluxData the makers of InfluxDB
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Available plugins: 241 inputs, 9 aggregators, 29 processors, 24 parsers, 60 outputs, 5 secret-stores
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loaded inputs: redfish
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loaded aggregators:
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loaded processors:
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loaded secretstores:
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Loaded outputs: file
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! Tags enabled: host=host1
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z D! [agent] Initializing plugins
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z D! [agent] Connecting outputs
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z D! [agent] Attempting connection to [outputs.file]
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z D! [agent] Successfully connected to outputs.file
Nov 06 14:34:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:34:32Z D! [agent] Starting service inputs
Nov 06 14:34:32 host1.ourdomain.com systemd[1]: Started Telegraf.
Nov 06 14:35:00 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:00Z I! [inputs.redfish] Calling GET on /redfish/v1/Systems/System.Embedded.1
Nov 06 14:35:00 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:00Z I! [inputs.redfish] username: root
Nov 06 14:35:00 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:00Z I! [inputs.redfish] password: OurPassword
Nov 06 14:35:00 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:00Z I! [inputs.redfish] error during the request
Nov 06 14:35:00 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:00Z E! [inputs.redfish] Error in plugin: Get "/redfish/v1/Systems/System.Embedded.1": unsupported protocol scheme ""
Nov 06 14:35:32 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:35:32Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Using URL:

[root@host1 telegraf]# journalctl --no-pager --unit telegraf | tail -n 30
Nov 06 14:38:59 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:38:59Z D! [agent] Input channel closed
Nov 06 14:38:59 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:38:59Z I! [agent] Hang on, flushing any cached metrics before shutdown
Nov 06 14:38:59 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:38:59Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 14:38:59 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:38:59Z I! [agent] Stopping running outputs
Nov 06 14:38:59 host1.ourdomain.com telegraf[595128]: 2023-11-06T19:38:59Z D! [agent] Stopped Successfully
Nov 06 14:38:59 host1.ourdomain.com systemd[1]: telegraf.service: Succeeded.
Nov 06 14:38:59 host1.ourdomain.com systemd[1]: Stopped Telegraf.
Nov 06 14:38:59 host1.ourdomain.com systemd[1]: Starting Telegraf...
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loading config: /etc/telegraf/telegraf.conf
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Starting Telegraf 1.29.0-93471676 brought to you by InfluxData the makers of InfluxDB
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Available plugins: 241 inputs, 9 aggregators, 29 processors, 24 parsers, 60 outputs, 5 secret-stores
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loaded inputs: redfish
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loaded aggregators:
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loaded processors:
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loaded secretstores:
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Loaded outputs: file
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! Tags enabled: host=host1
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"host1", Flush Interval:1m0s
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z D! [agent] Initializing plugins
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z D! [agent] Connecting outputs
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z D! [agent] Attempting connection to [outputs.file]
Nov 06 14:38:59 host1.ourdomain.com systemd[1]: Started Telegraf.
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z D! [agent] Successfully connected to outputs.file
Nov 06 14:38:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:38:59Z D! [agent] Starting service inputs
Nov 06 14:39:59 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:39:59Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
Nov 06 14:40:00 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:40:00Z I! [inputs.redfish] Calling GET on https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1
Nov 06 14:40:00 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:40:00Z I! [inputs.redfish] username: root
Nov 06 14:40:00 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:40:00Z I! [inputs.redfish] password: OurPassword
Nov 06 14:40:00 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:40:00Z I! [inputs.redfish] error during the request
Nov 06 14:40:00 host1.ourdomain.com telegraf[597130]: 2023-11-06T19:40:00Z E! [inputs.redfish] Error in plugin: Get "https://host1-obm.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden
[root@host1 telegraf]#
14

uhhh, I am officially out of ideas :\

Looking at other posts with 403 with respect to redfish it seems to point to a proxy:

15

I appreciate your efforts @jpowers

I checked again for any sign of a proxy on the node. I donā€™t see any. And I checked the proxy logs for any evidence of this node hitting it - again, nothing.

Could this message be telling us something that we hadnā€™t already considered?

Nov 06 11:03:00 host1.ourdomain.com telegraf[532547]: 2023-11-06T16:03:00Z E! [inputs.redfish] Error in plugin: Get "/redfish/v1/Systems/System.Embedded.1": unsupported protocol scheme ""

This is what replaced the Forbidden error message when I used a hostname or IP instead of a URL like you suggested.

16

The scheme message was about missing the https://, which apparently is still needed. Did you try:

address = ā€œhttps://host1-obm.ourdomain.comā€

17

Btw, you might not be using a proxy, but your traffic could still get intercepted by a proxy.

And I checked the proxy logs for any evidence of this node hitting it

Are you sure there isnā€™t another proxy further down the line? Are you in a corporate setting where your IT manages one?

18

Including the https:// yields the same Forbidden error:

Nov 08 15:44:00 host1.ourdomain.com telegraf[1934338]: 2023-11-08T20:44:00Z E! [inputs.redfish] Error in plugin: Get "https://host1.ourdomain.com/redfish/v1/Systems/System.Embedded.1": Forbidden

The only proxy in the path between the node running telegraf and the obm that serves redfish is our proxy. There may be proxies outside our network that are administered by central IT, but this traffic should never leave our network.

Iā€™m going to try running telegraf on another node to rule out something wrong with this node. Will report back if I find anything new.