How to avoid data loss in the influxdb

InfluxDB 2
, , ,
1

I am pushing data from NATS to InfluxDB using Telegraf. The data is being generated continuously, which results in multiple entries having the same timestamp. My timestamps are in nanoseconds, but in InfluxDB, the data is being aggregated, causing a loss of data points in my Grafana dashboard.

Can any one suggest how to avoid data loss.

example data.

requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test time=1728981942717004500,count=1,errorCount=0,sentBytes=1467,timeStamp=1728981940732,receivedBytes=59275,latency=0,processingTime=0,connectTime=0,responseTime=3151 1728981942717004544
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test latency=0,receivedBytes=59275,time=1728981942717004500,errorCount=0,responseTime=1276,count=1,processingTime=0,sentBytes=1467,timeStamp=1728981942619,connectTime=0 1728981942717004544

Below is my telegraf configuration

[[inputs.nats_consumer]]
servers = [“nats://localhost:4222”]
subjects = [“nats.subject.requestsraw”]
data_format = “xpath_json”
xpath_native_types = true
fieldexclude = [“subjects”, “runId”, “testName”, “nodeName”, “ObjectType”, “errorMessage”, “errorResponseBody”, “requestName”, “responseCode”, “result”, “samplerType”]

[[inputs.nats_consumer.xpath]]
metric_name = “‘requestsRaw’”
metric_selection = “/"
timestamp = “time”
timestamp_format = “unix_ns”
field_selection = "”
[inputs.nats_consumer.xpath.tags]
runId = “string(/runId)”
testName = “string(/testName)”
nodeName = “string(/nodeName)”
ObjectType = “string(/ObjectType)”
errorMessage = “string(/errorMessage)”
errorResponseBody = “string(/errorResponseBody)”
requestName = “string(/requestName)”
responseCode = “string(/responseCode)”
result = “string(/result)”
samplerType = “string(/samplerType)”

2

Hello @Thrinatha_Reddy,
I notice a timestamp in your data that is different::
timeStamp=1728981940732
timeStamp=1728981942619

Could you use those timestamps as the timestamp instead?

Can you add a tag to your line protocol to seperate the fields so you don’t overwrite them?

I would look into doing one of those two first.

alternatively you could also jitter timestamps with a processor plugin like execd if you have wiggle room and can add a couple nanoseconds here are there.