Telegraf does not replace a null timestamp with the insertion timestamp

Telegraf
, , ,
1

Hello
Colleagues send sensor values from a lorawan gateway in mqtt topics.

With the help of telegraf, I consume the data on the mqtt topics. I choose to use the time value of the gateway to be able to graph with grafana. I convert it like this:

[[inputs.mqtt_consumer]]
  ...
  ...
  ...
  # parse time
  json_time_format    = "2006-01-02T15:04:05.999999999Z"
  json_time_key       = "rxInfo_0_time"
  json_timezone       = "UTC"

But, it happens that the time value is null in some messages. Theoretically, at this moment, telegraf should write the data with the insertion timestamp … but it is not the case !

Does anyone have an idea of what the problem is?

Thanks
TH

Below is my complete config file:

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################


[[outputs.influxdb]]
  urls                   = ["https://my.address.fr:8086"]
  database               = "sicpa_iot"
  skip_database_creation = true
  username               = "user"
  password               = "password"
  namepass               = ["device_*"]
  tls_cert               = "/home/telegraf/ssl/telegraf_cert.crt"
  tls_key                = "/home/telegraf/ssl/telegraf.key"
  tagexclude             = ["adr","applicationID","applicationName","fCnt","fPort","gatewayID","loRaSNR","device_name","rssi","uplinkID","dr","frequency","insert_time"]


[[outputs.influxdb]]
  urls                   = ["https://my.address.fr:8086"]
  database               = "sicpa_iot_with_meta"
  skip_database_creation = true
  username               = "user"
  password               = "password"
  namepass               = ["device_*"]
  tls_cert               = "/home/telegraf/ssl/telegraf.crt"
  tls_key                = "/home/telegraf/ssl/telegraf.key"



###############################################################################
#                            PROCESSOR PLUGINS                                #
###############################################################################

[[processors.rename]]
  order = 1
  [[processors.rename.replace]]
    field = "object_analogInput_4"
    dest  = "analogInput4"
  [[processors.rename.replace]]
    field = "object_analogInput_8"
    dest  = "analogInput8"
  [[processors.rename.replace]]
    field = "object_barometer_6"
    dest  = "pressure"
  [[processors.rename.replace]]
    field = "object_humiditySensor_7"
    dest  = "humidity"
  [[processors.rename.replace]]
    field = "object_temperatureSensor_2"
    dest  = "temperature"


[[processors.rename]]
  order = 2
  [[processors.rename.replace]]
    tag   = "deviceName"
    dest  = "device_id"
  [[processors.rename.replace]]
    tag   = "devEUI"
    dest  = "device_eui"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_gatewayID"
    dest  = "gatewayID"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_location_altitude"
    dest  = "altitude"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_location_latitude"
    dest  = "latitude"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_location_longitude"
    dest  = "longitude"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_loRaSNR"
    dest  = "loRaSNR"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_name"
    dest  = "device_name"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_rssi"
    dest  = "rssi"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_uplinkID"
    dest  = "uplinkID"
  [[processors.rename.replace]]
    tag   = "txInfo_0_dr"
    dest  = "dr"
  [[processors.rename.replace]]
    tag   = "txInfo_0_frequency"
    dest  = "frequency"
  [[processors.rename.replace]]
    tag   = "rxInfo_0_time"
    dest  = "gateway_time"

[[processors.template]]
  order = 3
  tag = "device"
  template = 'device_{{ .Tag "device_id" }}'

[[processors.converter]]
  order = 4
  [processors.converter.tags]
    measurement = ["device"]

[[processors.date]]
  order       = 5
  tag_key     = "gateway_time"
  date_format = "2006-01-02T15:04:05.000000000Z"
  date_offset = "0s"
  timezone    = "UTC"




###############################################################################
#                            AGGREGATOR PLUGINS                               #
###############################################################################





###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################





###############################################################################
#                            SERVICE INPUT PLUGINS                            #
###############################################################################


[[inputs.mqtt_consumer]]
  servers            = ["ssl://my.address.fr:8883"]
  topics             = ["application/1/device/+/event/up"]
  qos                = 2
  persistent_session = true
  client_id          = "telegraf"
  username           = "user"
  password           = "password"
  data_format        = "json"
  tls_ca             = "/home/telegraf/ssl/telegraf_ca.crt"
  tls_cert           = "/home/telegraf/ssl/telegraf_cert.crt"
  tls_key            = "/home/telegraf/ssl/telegraf.key"
  tag_keys           = [
    "adr",
    "applicationID",
    "applicationName",
    "deviceName",
    "devEUI",
    "fCnt",
    "fPort",
    "rxInfo_0_gatewayID",
    "rxInfo_0_location_altitude",
    "rxInfo_0_location_latitude",
    "rxInfo_0_location_longitude",
    "rxInfo_0_loRaSNR",
    "rxInfo_0_name",
    "rxInfo_0_rssi",
    "rxInfo_0_time",
    "rxInfo_0_uplinkID",
    "txInfo_0_dr",
    "txInfo_0_frequency"
  ]
  fieldpass           = [
    "object_analogInput_4",
    "object_analogInput_8",
    "object_barometer_6",
    "object_humiditySensor_7",
    "object_temperatureSensor_2"
  ]
  # parse time
  json_time_format    = "2006-01-02T15:04:05.999999999Z"
  json_time_key       = "rxInfo_0_time"
  json_timezone       = "UTC"
2

What do you mean by “null” exactly?
What exactly does the json payload look like in this case?

3

when the time is present, it is in the “time” variable of the rx_info section
section, you will find below the different scenarios we encounter

  • with a correct timestamp :
{
    "applicationID":"1",
    "applicationName":"chirp-app",
    "deviceName":"0200000002",
    "devEUI":"60c5a8fffe76ea87",
    "rxInfo":[
        {
            "gatewayID":"60c5a8fffe76154b",
            "uplinkID":"bb50f9f0-d095-4e3e-bafe-0b4296a51ad2",
            "name":"rak7249",
            "time":"2021-07-09T09:49:05.957635Z",
            "rssi":-71,
            "loRaSNR":8,
            "location":{
                "latitude":43.50438,
                "longitude":1.53005,
                "altitude":216
            }
        }
    ],
    "txInfo":{
        "frequency":867500000,
        "dr":0
    },
    "adr":false,
    "fCnt":5391,
    "fPort":8,
    "data":"CAIBeAdoaAZzJ1ECZwD2BAIG+Q==",
    "object":{
        "analogInput":{
            "4":17.85,
            "8":3.76
        },
        "barometer":{
            "6":1006.5
        },
        "humiditySensor":{
            "7":52
        },
        "temperatureSensor":{
            "2":24.6
        }
    }
}
  • Error 1 : a null timestamp
{
    "applicationID":"1",
    "applicationName":"chirp-app",
    "deviceName":"0200000002",
    "devEUI":"60c5a8fffe76ea87",
    "rxInfo":[
        {
            "gatewayID":"dca632fffe0ca891",
            "uplinkID":"fcdc7154-fd25-440e-89c0-58f14fb0ebb5",
            "name":"RAK7244",
            "time":"null",
            "rssi":-95,
            "loRaSNR":8.8,
            "location":{
                "latitude":43.52878,
                "longitude":1.49932,
                "altitude":157
            }
        }
    ],
    "txInfo":{
        "frequency":867100000,
        "dr":0
    },
    "adr":false,
    "fCnt":2001,
    "fPort":8,
    "data":"CAIBjwdokwZzJ30CZ//rBAIgtg==",
    "object":{
        "analogInput":{
            "4":83.74,
            "8":3.99
        },
        "barometer":{
            "6":1010.9
        },
        "humiditySensor":{
            "7":73.5
        },
        "temperatureSensor":{
            "2":-2.1
        }
    }
}
  • Error 2 : a missing timestamp
{
    "applicationID":"1",
    "applicationName":"chirp-app",
    "deviceName":"ers-sound",
    "devEUI":"a81758fffe04ba31",
    "rxInfo":[
        {
            "gatewayID":"dca632fffe0ca891",
            "uplinkID":"4f31e476-5098-467f-8b0b-544c08592c3d",
            "name":"RAK7244",
            "rssi":-42,
            "loRaSNR":9,
            "location":{
                "latitude":43.52878,
                "longitude":1.49932,
                "altitude":157
            }
        }
    ],
    "txInfo":{
        "frequency":867900000,
        "dr":0
    },
    "adr":true,
    "fCnt":5713,
    "fPort":5,
    "data":"AQDiAhwEAOcFBwcOKRVKIw==",
    "object":{
        "humidity":28,
        "light":231,
        "motion":7,
        "soundAvg":35,
        "soundPeak":74,
        "temperature":22.6,
        "vdd":3625
    }
}
4

hello
does anyone have an idea how to solve this problem?

5

As far as I know, a “null” or missing timestamp should always lead to an error in the input plugin parser.

Error in plugin: parsing time "null" as "2006-01-02T15:04:05.999999999Z": cannot parse "null" as "2006"
Error in plugin: JSON time key could not be found
  1. As far as I know, the only option would be to remove the timestamp entirely from the json config. Also the tag rxInfo_0_time would have to be removed.
    But then you always get the timestamp from Telegraf at the time of the message ingress.
  2. There is another problem with the config. The fieldpass config does not match the structure of the payload of Error 2 : a missing timestamp below object. The structure is different from the other two previous examples.

This configuration works in principle, but may not meet your requirements:

  tag_keys = [
    "adr",
    "applicationID",
    "applicationName",
    "deviceName",
    "devEUI",
    "fCnt",
    "fPort",
    "rxInfo_0_gatewayID",
    "rxInfo_0_location_altitude",
    "rxInfo_0_location_latitude",
    "rxInfo_0_location_longitude",
    "rxInfo_0_loRaSNR",
    "rxInfo_0_name",
    "rxInfo_0_rssi",
    # "rxInfo_0_time",
    "rxInfo_0_uplinkID",
    "txInfo_0_dr",
    "txInfo_0_frequency"
  ]
  # fieldpass = [
  #   "object_analogInput_4",
  #   "object_analogInput_8",
  #   "object_barometer_6",
  #   "object_humiditySensor_7",
  #   "object_temperatureSensor_2"
  # ]
  # parse time
  # json_time_format    = "2006-01-02T15:04:05.999999999Z"
  # json_time_key       = "rxInfo_0_time"
  # json_timezone       = "UTC"

In addition, I always advocate solving problems at the source and ensuring that senders send correct data. :wink: