Hello everyone, I am trying to implement a small datalogger using Influxdb v2.7, Telegraf v1.26, and Grafana v9.4.
The data is in Json format and it is fetched from the mosquitto mqtt server using Telegraf .
I am trying to store data with custom timestamp given in the json object .
I have specified the data_format = “json” , json_time_key = “time”, json_time_format = “2006-01-02 15:04:05” in Telegraf configuration file. Data is not shown in Influxdb UI on browser. when i comment these above lines and use server time it shows the data. i have made sure that after every time changes were made in config file the Telegraf service was restarted.
The Json data which Telegraf recieves from MQTT server is as given below:-
{
“TNS0001” :{
“dev_id” : “S0001”,
“time”: “2023-01-02 15:04:05”,
“Temp1”: 25.31,
“Temp2”: 25.32,
“Temp3”: 25.33,
“Temp4”: 25.34,
“Temp5”: 25.35,
“Temp6”: 25.36,
“Temp7”: 25.37
}
}
I need to store all the above parameters in separate tags. I have attached telegraf config file. please let me know how this issue can be resolved. Thank you
Is that a typo or does your time really exist in the year 20,023? 
json_time_key = “time”
Note your key time is nested under an object. In the example it is TNS0001. As such saying the field is time is not actually valid.
I have attached telegraf config file.
I don’t see an attachment, can you copy it in a comment inside tripple backticks (e.g. ```)
Hi, Thanks for the Quick reply. Yes it was a typo mistake I have made it right.
Here i have mentioned nested json data, but I have also tried to sent simple json data as given below
{
"time": "2006-01-02 15:04:05",
"data":123
}
this data was not logged in the database which I checked on Influxdb UI .
when I intentionally entered the wrong month in above json data, it showed following error:
Error in plugin: parsing time “2006-55-02 15:04:05”: month out of range
and no error when I sent data in correct format.
From this, I believe Telegraf is processing data and timestamp but perhaps not passed to DB.
Also, when I commented following 2 lines in the Telegraf configuration, it worked and I could see data being logged to Influxdb.
json_time_key = "time"
json_time_format = "2006-01-02 15:04:05"
I have made sure that the time format mentioned in Telegraf and sent data is matching.
Please guide.
Thank You in advance.
# Configuration for telegraf agent
[agent]
## Default data collection interval for all inputs
interval = "10s"
## Rounds collection interval to 'interval'
## ie, if interval="10s" then always collect on :00, :10, :20, etc.
round_interval = true
## Telegraf will send metrics to outputs in batches of at most
## metric_batch_size metrics.
## This controls the size of writes that Telegraf sends to output plugins.
metric_batch_size = 1000
## Maximum number of unwritten metrics per output. Increasing this value
## allows for longer periods of output downtime without dropping metrics at the
## cost of higher maximum memory usage.
metric_buffer_limit = 10000
## Collection jitter is used to jitter the collection by a random amount.
## Each plugin will sleep for a random time within jitter before collecting.
## This can be used to avoid many plugins querying things like sysfs at the
## same time, which can have a measurable effect on the system.
collection_jitter = "0s"
## Default flushing interval for all outputs. Maximum flush_interval will be
## flush_interval + flush_jitter
flush_interval = "10s"
## Jitter the flush interval by a random amount. This is primarily to avoid
## large write spikes for users running a large number of telegraf instances.
## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
flush_jitter = "0s"
## By default or when set to "0s", precision will be set to the same
## timestamp order as the collection interval, with the maximum being 1s.
## ie, when interval = "10s", precision will be "1s"
## when interval = "250ms", precision will be "1ms"
## Precision will NOT be used for service inputs. It is up to each individual
## service input to set the timestamp at the appropriate precision.
## Valid time units are "ns", "us" (or "µs"), "ms", "s".
precision = ""
## Log at debug level.
# debug = false
## Log only error level messages.
# quiet = false
## Log target controls the destination for logs and can be one of "file",
## "stderr" or, on Windows, "eventlog". When set to "file", the output file
## is determined by the "logfile" setting.
# logtarget = "file"
## Name of the file to be logged to when using the "file" logtarget. If set to
## the empty string then logs are written to stderr.
# logfile = ""
## The logfile will be rotated after the time interval specified. When set
## to 0 no time based rotation is performed. Logs are rotated only when
## written to, if there is no log activity rotation may be delayed.
# logfile_rotation_interval = "0d"
## The logfile will be rotated when it becomes larger than the specified
## size. When set to 0 no size based rotation is performed.
# logfile_rotation_max_size = "0MB"
## Maximum number of rotated archives to keep, any older logs are deleted.
## If set to -1, no archives are removed.
# logfile_rotation_max_archives = 5
## Pick a timezone to use when logging or type 'local' for local time.
## Example: America/Chicago
# log_with_timezone = ""
## Override default hostname, if empty use os.Hostname()
hostname = ""
## If set to true, do no set the "host" tag in the telegraf agent.
omit_hostname = false
[[outputs.influxdb_v2]]
## The URLs of the InfluxDB cluster nodes.
##
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
## ex: urls = ["https://us-west-2-1.aws.cloud2.influxdata.com"]
urls = ["http://192.168.1.111:8086"]
## Token for authentication.
token = "token"
## Organization is the name of the organization you wish to write to; must exist.
organization = "test"
## Destination bucket to write into.
bucket = "Datalogger1"
## The value of this tag will be used to determine the bucket. If this
## tag is not set the 'bucket' option is used as the default.
# bucket_tag = ""
## If true, the bucket tag will not be added to the metric.
# exclude_bucket_tag = false
## Timeout for HTTP messages.
# timeout = "5s"
## Additional HTTP headers
# http_headers = {"X-Special-Header" = "Special-Value"}
## HTTP Proxy override, if unset values the standard proxy environment
## variables are consulted to determine which proxy, if any, should be used.
# http_proxy = "http://corporate.proxy:3128"
## HTTP User-Agent
# user_agent = "telegraf"
## Content-Encoding for write request body, can be set to "gzip" to
## compress body or "identity" to apply no encoding.
# content_encoding = "gzip"
## Enable or disable uint support for writing uints influxdb 2.0.
# influx_uint_support = false
## Optional TLS Config for use on HTTP connections.
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
# Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
## Broker URLs for the MQTT server or cluster. To connect to multiple
## clusters or standalone servers, use a separate plugin instance.
## example: servers = ["tcp://localhost:1883"]
## servers = ["ssl://localhost:1883"]
## servers = ["ws://localhost:1883"]
servers = ["mqtt://192.168.1.111:1883"]
## Topics that will be subscribed to.
topics = [
"telegraf/host01/cpu",
"telegraf/+/mem",
"test/Sensor/#",
]
## The message topic will be stored in a tag specified by this value. If set
## to the empty string no topic tag will be created.
# topic_tag = "topic"
## QoS policy for messages
## 0 = at most once
## 1 = at least once
## 2 = exactly once
##
## When using a QoS of 1 or 2, you should enable persistent_session to allow
## resuming unacknowledged messages.
qos = 0
## Connection timeout for initial connection in seconds
connection_timeout = "60s"
## Maximum messages to read from the broker that have not been written by an
## output. For best throughput set based on the number of metrics within
## each message and the size of the output's metric_batch_size.
##
## For example, if each message from the queue contains 10 metrics and the
## output metric_batch_size is 1000, setting this to 100 will ensure that a
## full batch is collected and the write is triggered immediately without
## waiting until the next flush_interval.
# max_undelivered_messages = 1000
## Persistent session disables clearing of the client session on connection.
## In order for this option to work you must also set client_id to identify
## the client. To receive messages that arrived while the client is offline,
## also set the qos option to 1 or 2 and don't forget to also set the QoS when
## publishing.
# persistent_session = false
## If unset, a random client ID will be generated.
# client_id = ""
## Username and password to connect MQTT server.
username = "user1"
password = "password123"
## Optional TLS Config
# tls_ca = "/etc/telegraf/ca.pem"
# tls_cert = "/etc/telegraf/cert.pem"
# tls_key = "/etc/telegraf/key.pem"
## Use TLS but skip chain & host verification
# insecure_skip_verify = false
## Data format to consume.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
data_format = "json"
## Time key is the key containing the time that should be used to create the
## metric.
json_time_key = "time"
json_time_format = "2006-01-02 15:04:05"
## Enable extracting tag values from MQTT topics
## _ denotes an ignored entry in the topic path
# [[inputs.mqtt_consumer.topic_parsing]]
# topic = ""
# measurement = ""
# tags =[""]
# fields = ""
## Value supported is int, float, unit
# [[inputs.mqtt_consumer.topic.types]]
# key = type
My suggestion then is to put the JSON data you are using in a file. Use the file input + json parser and get that working as you expect. If you come up with a reproducer provide 1) the config 2) the example JSON 3) your expected metric you want
Thanks for the reply,
Can you please explain why we can’t see output with this simple configuration?
Also seems I am new to this platform, I could not understand your solution.
Can you please elaborate your points and share some reference links for the same?
Thanks in Advance.
