Telegraf Tail plugin as a replacement for Filebeat

Telegraf
,
1

Hello,

I am currently using the tail plugin from Telegraf to push InfluxDB line protocol data directly to InfluxDB. Have not fully scaled up yet but on a smaller test system I noticed there are occasionally a few measurements being recorded that seem to be incorrectly formatted/parsed. It looks like it took a partial cut of line protocol formatted line and used that as the new line, for example:

Real line -
ClientRequest,host=cassandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead field1=42,field2=100,field3=400 1644353580033000000

However I will notice something like the following as a series in InfluxDB -
sandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead

So the measurement recorded in the DB is sandraHost01 with the tags and fields that followed, and the preceding bit (ClientRequest,host=cas) is lost.

Anyone know why this might be happening? My tail plugin config is this -

[[inputs.tail]]
    files = ["/path/to/logs/telegraf-metrics*"]
    data_format = "influx"
    path_tag = ""

And my agent config is this -

# Configuration for telegraf agent
[agent]
    interval = "60s"
    debug = false
    hostname = "hostname"
    round_interval = true
    flush_interval = "60s"
    flush_jitter = "0s"
    collection_jitter = "0s"
    metric_batch_size = 1000
    metric_buffer_limit = 15000
    quiet = false
    logtarget = "file"
    logfile = "/opt/telegraf.log"
    logfile_rotation_interval = "24h"
    logfile_rotation_max_size = "10MB"
    logfile_rotation_max_archives = 5
    omit_hostname = false
2

Have you looked through the telegraf-metrics* files for any matches to that odd line? It would be good to check to see if any \n characters are mid-line as well.

A great way to triage something like this is to send your data to a file/stdout using [[outputs.file]]. It would help narrow down if the data is coming from the input or if the output is having issues.

3

I lied! Sorry about that, I had actually looked through and not found the malformed lines before so I assumed it was being chopped up during the transport by the tail plugin. I guess my logs must have aged out and therefore I didn’t find the offending lines. I appreciate the debugging tips though, will note that.