Convert metric time from timestamp

Telegraf
,
1

Im getting a json information from http plugin that looks like:

{"status":"ok","objects":[{"time":1639643372780,...},{"time":1639643372780,...},{"time":1639643363464,...},{"time":1639643361263,...}]}

and my telegraf config:

[[inputs.http]]
urls = ["myApiUrl"]
name_override = "myObjects"
data_format = "json"
tagexclude = [ "url", "host"]
json_query = "objects"
 json_time_key = "time"
 json_time_format = "unix"
 json_timezone = "Asia/Tehran"

this doesnt print anything on test because of json_time_key so after hours i figured it out :frowning_face:: not even error :frowning:

so to make it work i need to divide time /1000 for every object but i dont know how to do that

2 
json_time_format = "unix_ms"
3

i just came up with a solution using starlark plugin that looks like:

[[processors.starlark]]
  ## The Starlark source can be set as a string in this configuration file, or
  ## by referencing a file containing the script.  Only one source or script
  ## should be set at once.

  ## Source of the Starlark script.
  source = '''
load('time.star', 'time')
def apply(metric):
    mtime =  int(metric.fields['time'] / 1000)
    mtime = time.from_timestamp(mtime)
    metric.time = mtime.unix_nano
    return metric
'''
4

when i set json_time_format to "unix" or "unix_ms" test doesn’t show anything so i assume it doesn’t work

5

What do you mean by that?

7

this is the output on debug mode
telegraf --config ./telegraf.conf -test
2021-12-16T09:29:36Z I! Starting Telegraf 1.20.4
2021-12-16T09:29:36Z D! [agent] Initializing plugins
2021-12-16T09:29:36Z D! [agent] Starting service inputs
2021-12-16T09:29:37Z D! [agent] Stopping service inputs
2021-12-16T09:29:37Z D! [agent] Input channel closed
2021-12-16T09:29:37Z D! [agent] Processor channel closed
2021-12-16T09:29:37Z D! [agent] Stopped Successfully

8

You don’t need a starlark plugin for a standard unix timestamp in milliseconds.
I assume something else is wrong in your Telegraf config.

9

@Franky1 but only when i set json time format i get no output otherwise it works!
and now when i add metric.fields.clear() i get no output either!

  source = '''
load('time.star', 'time')
def apply(metric):
    mtime =  int(metric.fields['time'] / 1000)
    mtime = time.from_timestamp(mtime)
    metric.time = mtime.unix_nano
    metric.fields.clear()
    return metric
'''

is there any step by step log file to investigate more?

10

so when i metric.fields.clear() or metric.fields.pop('time') or metric.tags.pop('time') everything blows up and i get no output but when i metric.tags.clear() it just simply removes tags and i get output!

11

I would add this to your telegraf config for better debugging:

[agent]
  # other agent configs here
  # ....
  debug = true
  logtarget = "file"
  logfile = "debug.log"

# file output only for debugging
[[outputs.file]]
  files = ["debug.txt"]
  influx_sort_fields = true

Forget about starlark, you don’t need it for a unix timestamp.
Something else doesn’t work.

12

still nothing :frowning_face:

root@ebrahim-G751JY:/etc/telegraf# systemctl start telegraf             
root@ebrahim-G751JY:/etc/telegraf# systemctl stop telegraf 
root@ebrahim-G751JY:/etc/telegraf# cat telegraf.debug.txt 

root@ebrahim-G751JY:/etc/telegraf# cat telegraf.debug.log 
2021-12-16T11:01:53Z D! [agent] Initializing plugins
2021-12-16T11:01:53Z D! [agent] Starting service inputs
2021-12-16T11:01:54Z D! [agent] Stopping service inputs
2021-12-16T11:01:54Z D! [agent] Input channel closed
2021-12-16T11:01:54Z D! [agent] Processor channel closed
2021-12-16T11:01:54Z D! [agent] Stopped Successfully
root@ebrahim-G751JY:/etc/telegraf# telegraf --config telegraf.conf -test
2021-12-16T11:02:12Z I! Starting Telegraf 1.20.4
root@ebrahim-G751JY:/etc/telegraf# cat telegraf.debug.log               
2021-12-16T11:01:53Z D! [agent] Initializing plugins
2021-12-16T11:01:53Z D! [agent] Starting service inputs
2021-12-16T11:01:54Z D! [agent] Stopping service inputs
2021-12-16T11:01:54Z D! [agent] Input channel closed
2021-12-16T11:01:54Z D! [agent] Processor channel closed
2021-12-16T11:01:54Z D! [agent] Stopped Successfully
2021-12-16T11:02:12Z D! [agent] Initializing plugins
2021-12-16T11:02:12Z D! [agent] Starting service inputs
2021-12-16T11:02:13Z D! [agent] Stopping service inputs
2021-12-16T11:02:13Z D! [agent] Input channel closed
2021-12-16T11:02:13Z D! [agent] Processor channel closed
2021-12-16T11:02:13Z D! [agent] Stopped Successfully
root@ebrahim-G751JY:/etc/telegraf# cat telegraf.debug.txt               

root@ebrahim-G751JY:/etc/telegraf#
13

Without real data, it becomes difficult.
Please post a snippet of the real json data and the whole real configuration of the [[inputs.http]] plugin.

14 
[agent]

debug = true

precision = "s"

interval = "4s"

[global_tags]

source="NBTX"

[[inputs.http]]

urls = ["https://api.nobitex.ir/v2/trades/USDTIRT"]

name_override = "USDTIRT"

data_format = "json"

tagexclude = [ "url", "host"]

json_query = "trades"

tag_keys = ["price", "volume","type"]

json_time_key = "time"

json_time_format="unix_ms"

json_timezone="Asia/Tehran"
15

Okay, so we’re getting somewhere.
Two problems catch my eye right away:

  1. You have defined all values from the json payload as tags. But there must be at least one field, otherwise the metric is invalid. Therefore you don’t see any metrics. Remove price and volume and try again.
  2. All values from the json payload are returned by the API as a string. You would probably have to convert this if you want to process these values in a meaningful way. It could be that this does not work with the json parser and you would have to use the newer json parser json_v2.
16

This works:

[global_tags]
  source="NBTX"

[[inputs.http]]
  urls = ["https://api.nobitex.ir/v2/trades/USDTIRT"]
  name_override = "USDTIRT"
  data_format = "json"
  tagexclude = ["url", "host"]
  json_query = "trades"
  tag_keys = ["type"]
  json_string_fields = ["price", "volume"]
  json_time_key = "time"
  json_time_format = "unix_ms"
  json_timezone = "Asia/Tehran"

[[processors.converter]]
  [processors.converter.fields]
    float = ["price", "volume"]

# file output only for debugging
[[outputs.file]]
  files = ["nobitex.out"]
  influx_sort_fields = true
17

And here is a version with the json_v2 parser.
However, configuration of this parser is a bit of a nightmare :grimacing:

[global_tags]
  source="NBTX"

[[inputs.http]]
  urls = ["https://api.nobitex.ir/v2/trades/USDTIRT"]
  tagexclude = ["url", "host"]
  data_format = "json_v2"
  [[inputs.http.json_v2]]
    measurement_name = "USDTIRT"
    [[inputs.http.json_v2.object]]
        path = "trades"
        tags = ["type"]
        timestamp_key = "time"
        timestamp_format = "unix_ms"
        timestamp_timezone = "Asia/Tehran"
        disable_prepend_keys = true
      [inputs.http.json_v2.object.fields]
        price = "float"
        volume = "float"

# file output only for debugging
[[outputs.file]]
  files = ["nobitex2.out"]
  influx_sort_fields = true
1 Like
18

works like a charm :slight_smile:
Thanks a lot

1 Like