Use InfluxDB's timestamp, instead of OS clock

Hello, I have to collect system status from remote server’s telegraf, but since this server dedicated to test, this server’s clock changes frequently.
So the collected data became chaotic, making unable to monitor this server at all.
I found I can actually grab timestamp from influxdb, and made this file

#!/bin/bash
timestamp=$(curl -I 10.10.91.130:8086/ping 2>/dev/null | grep Date: | sed ‘s/Date: //g’)

epoch=$(date -d “$timestamp” +%s)

epoch_ns=$(($epoch * 1000000000))
echo $epoch_ns

My idea was to run this file with exec plugin and then make starlark plugin replace timestamps… but I couldn’t progress any further from here.
Or, is there any better method to achieve this goal?

My company uses telegraf 1.23 and influxdb 1.8.10.

Where did you get hung up?
Maybe use the execd processor plugin? For more flexibility?
Can you share youre telegraf config?
and any logs? With debug = true in the agent portion of the config?

Thanks!

Hi, I actually had little progress.

First, I used this script to pull epoch nano timestamp from influxDB.

Timestamp.sh

#!/bin/bash
timestamp=$(curl -I 10.10.91.130:8086/ping 2>/dev/null | grep Date: | sed ‘s/Date: //g’)

epoch=$(date -d “$timestamp” +%s)

epoch_ns=$(($epoch * 1000000000))
#echo $epoch_ns > /etc/telegraf/script/time.txt
echo $epoch_ns

And then, this is my current config file.

[agent]
interval = “10s”
round_interval = true
metric_batch_size = 1_000
metric_buffer_limit = 10_000
collection_jitter = “0s”
flush_interval = “10s”
flush_jitter = “10s”
precision = “”
hostname = “test-db-1”
omit_hostname = false

[[outputs.influxdb]]
urls = [ “http://10.10.91.130:8086” ]
database = “Default”
retention_policy = “autogen”
write_consistency = “any”

[[inputs.cpu]]
percpu = false
totalcpu = true
collect_cpu_time = false
report_active = false

[[inputs.exec]]
commands = [“/etc/telegraf/script/timestamp.sh”]
timeout = “5s”
data_format = “value”
data_type = “integer”
name_override = “timestamp”

[[processors.starlark]]
source = ‘’’

def apply(metric):
if metric.name == “timestamp” and “value” in metric.fields:
metric.time = metric.fields[“value”]
return metric
‘’’

If I test this config, this will show new influxdb line, named timestamp, generated and it’s timestamp changed.
But I couldn’t change cpu data’s timestamp, already collected with telegraf.

root@shan-db-1:/etc/telegraf# telegraf --config ./telegraf.conf --test
2024-11-02T00:08:48Z I! Starting Telegraf 1.23.0
2024-11-02T00:08:48Z I! Loaded inputs: cpu exec
2024-11-02T00:08:48Z I! Loaded aggregators:
2024-11-02T00:08:48Z I! Loaded processors: starlark
2024-11-02T00:08:48Z W! Outputs are not used in testing mode!
2024-11-02T00:08:48Z I! Tags enabled: host=test-db-1

timestamp,host=test-db-1 value=1732001797000000000i 1732001797000000000
cpu,cpu=cpu-total,host=test-db-1 usage_guest=0,usage_guest_nice=0,usage_idle=98.99999997625127,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=0.9999999995334292,usage_user=0 1730506129000000000

I tried golbal variable, but it seems starlark doesn’t support it.
how can I use epoch timestamp collected from exec plugin, and use it to replace timestamp collected from other plugin?

Hi! Look what I got here!

I put these 2 plugin with the 6 lines of Timestamp.sh script, and it works well!
Telegraf now fetches the timestamp from influxDB, saves it as ‘timestamp’ metric, and then replace every metric.time before send data.

[[inputs.exec]]
commands = [“/etc/telegraf/script/timestamp.sh”]
timeout = “5s”
data_format = “value”
data_type = “integer”
name_override = “timestamp”

[[processors.starlark]]
order = 1
source = ‘’’
state = {“global_timestamp”:None}

def apply(metric):
if metric.name == “timestamp” and “value” in metric.fields:
state[“global_timestamp”] = metric.fields[“value”]
if state[“global_timestamp”] != None:
metric.time = state[“global_timestamp”]

return metric

‘’’

My first goal is achieved, but now I want put timestamp.sh inside the telegraf.conf if possible.
My company have solution to manage remote server’s telegraf.conf, but it has no ability to put the script to host.
So it’s better to keep everything in telegraf.conf, without seperated script.
Can anyone help me for this?

@shan1 the recently release Telegraf v1.33.0 has a new option for input plugins called time_source (see documentation) which can be used to override the timestamp of gathered metrics.

Please note: This option cannot be used with service inputs!

Hi Srebhan,
Isn’t this option just set the timing when the metrics timestamp decided?
It seems their timestamp source is still server’s OS and this option just decide when they collect them…