Telegraf does not write all data to influxdb (skips some lines)

Telegraf
1

Hi,

I’m uploading data from multiline lineprotocol file to InfluxDB 2.0.5 server using telegraf 1.18.3. Somehow only part of data is successfully written to Influxdb. Reported 15 written metrics but metric “2021:05:17-00:00:01~13” cannot be found. Can somebody comment why? Example is below

BD-data,action=drop,fwrule=60001,initf=eth6,dstport=60753 tme=“2021:05:17-00:00:01~1”
BD-data,action=log,fwrule=62050,initf=eth6,dstport=43389 tme=“2021:05:17-00:00:01~2”
BD-data,action=drop,fwrule=60001,initf=eth6,dstport=57358 tme=“2021:05:17-00:00:01~3”
BD-data,action=drop,fwrule=60001,initf=eth6,dstport=51564 tme=“2021:05:17-00:00:01~4”
BD-data,action=drop,fwrule=60002,initf=eth10.18,outitf=eth6,dstport=53 tme=“2021:05:17-00:00:01~5”
BD-data,action=drop,fwrule=60001,initf=eth6,dstport=52723 tme=“2021:05:17-00:00:01~6”
BD-data,action=drop,fwrule=60001,initf=eth6,dstport=59579 tme=“2021:05:17-00:00:01~7”
BD-data,action=drop,fwrule=60001,initf=eth7,dstport=50355 tme=“2021:05:17-00:00:01~8”
BD-data,action=drop,fwrule=60001,initf=eth7,dstport=64904 tme=“2021:05:17-00:00:01~9”
BD-data,action=accept,fwrule=272,initf=eth10.89,outitf=eth10.70,dstport=10051 tme=“2021:05:17-00:00:01~10”
BD-data,action=drop,fwrule=60002,initf=eth10.18,outitf=eth7,dstport=53 tme=“2021:05:17-00:00:01~11”
BD-data,action=drop,fwrule=60002,initf=eth10.36,outitf=eth10.40,dstport=7680 tme=“2021:05:17-00:00:01~12”
BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.51,dstport=10050 tme=“2021:05:17-00:00:01~13”
BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.89,dstport=10050 tme=“2021:05:17-00:00:01~14”
BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.51,dstport=10050 tme=“2021:05:17-00:00:01~15”

c:\Program Files\InfluxData\telegraf\telegraf-1.18.3>telegraf --input-filter file --output-filter influxdb_v2 --config “C:\Program Files\InfluxData\telegraf\telegraf-1.18.3\telegraf.conf” --once
2021-05-29T09:14:50Z I! Starting Telegraf 1.18.3
2021-05-29T09:14:50Z D! [agent] Initializing plugins
2021-05-29T09:14:50Z D! [agent] Connecting outputs
2021-05-29T09:14:50Z D! [agent] Attempting connection to [outputs.influxdb_v2]
2021-05-29T09:14:50Z D! [agent] Successfully connected to outputs.influxdb_v2
2021-05-29T09:14:50Z D! [agent] Starting service inputs
2021-05-29T09:14:50Z D! [agent] Stopping service inputs
2021-05-29T09:14:50Z D! [agent] Input channel closed
2021-05-29T09:14:50Z I! [agent] Hang on, flushing any cached metrics before shutdown
2021-05-29T09:14:50Z D! [outputs.influxdb_v2] Wrote batch of 15 metrics in 98.0983ms
2021-05-29T09:14:50Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2021-05-29T09:14:50Z I! [agent] Stopping running outputs
2021-05-29T09:14:50Z D! [agent] Stopped Successfully

#group,false,false,false,false,false,false,false,false
#datatype,string,long,string,string,string,string,string,string
#default,mean,,,,,,,
,result,table,_value,action,dstport,fwrule,initf,outitf
,,0,2021:05:17-00:00:01~1,drop,60753,60001,eth6,
,,0,2021:05:17-00:00:01~10,accept,10051,272,eth10.89,eth10.70
,,0,2021:05:17-00:00:01~11,drop,53,60002,eth10.18,eth7
,,0,2021:05:17-00:00:01~12,drop,7680,60002,eth10.36,eth10.40
,,0,2021:05:17-00:00:01~14,accept,10050,256,eth10.70,eth10.89
,,0,2021:05:17-00:00:01~15,accept,10050,256,eth10.70,eth10.51
,,0,2021:05:17-00:00:01~2,log,43389,62050,eth6,
,,0,2021:05:17-00:00:01~3,drop,57358,60001,eth6,
,,0,2021:05:17-00:00:01~4,drop,51564,60001,eth6,
,,0,2021:05:17-00:00:01~5,drop,53,60002,eth10.18,eth6
,,0,2021:05:17-00:00:01~6,drop,52723,60001,eth6,
,,0,2021:05:17-00:00:01~7,drop,59579,60001,eth6,
,,0,2021:05:17-00:00:01~8,drop,50355,60001,eth7,
,,0,2021:05:17-00:00:01~9,drop,64904,60001,eth7,
2

If you pay attention you’ll notice that lines 13 and 15 share the same key, therefore the point is considered as one (the latest wins).
Time is also part of the key, but since it’s not specified in your file Telegraf will generate it, and in this case the timestamp is also the same (or almost the same, but the difference is too small to make a difference)

BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.51,dstport=10050 tme=“2021:05:17-00:00:01~13”
BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.51,dstport=10050 tme=“2021:05:17-00:00:01~15”

the key (measurement + tag set + field ke + time) is:
BD-data,action=accept,fwrule=256,initf=eth10.70,outitf=eth10.51,dstport=10050

1 Like
3

Thanks Giovanni. You are absolutely right