InfluxDB line protocol ingesting as partial lines and messing up measurements

Hello,

I am currently using the tail plugin from Telegraf to push InfluxDB line protocol data directly to InfluxDB. Have not fully scaled up yet but on a smaller test system I noticed there are occasionally a few measurements being recorded that seem to be incorrectly formatted/parsed. It looks like it took a partial cut of line protocol formatted line and used that as the new line, for example:

Real line -
ClientRequest,host=cassandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead field1=42,field2=100,field3=400 1644353580033000000

However I will notice something like the following as a series in InfluxDB -
sandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead

So the measurement recorded in the DB is sandraHost01 with the tags and fields that followed, and the preceding bit ( ClientRequest,host=cas ) is lost.

To debug this so far, I have added a file output config so I can track all outbound measurements from the telegraf plugins and none of them are broken up this way, meaning this must be an issue with the ingest/write in InfluxDB 1.8.

Anyone know why this might be happening?

The only Idea I have is that there might be a hidden char, a new-line or something similar that breaks your line protocol

so instead of being:

#Expected
ClientRequest,host=cassandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead field1=42,field2=100,field3=400 1644353580033000000

#Actual
ClientRequest,host=
cassandraHost01,tagOne=valueOne,type=cassandra,name=ConditionNotMet,scope=CASRead field1=42,field2=100,field3=400 1644353580033000000

I’d look inside the original data to find the specific “broken rows” because it makes no sense that the very same rows is fine for the output.file but not for InflxDB.
I suggest you use a proper text editor for your search otherwise some hidden chars might pass unnoticed

Appreciate the feedback!

I’ve actually exhausted every which way of trying to find why this happens, it is not consistent and happens for 1 data point every once in a while. I tried to detect all kinds of special or weirdly encoded or hidden characters or newlines but didn’t find a single one. My other thought process was maybe it happened during a file rollover but sometimes they’re back to back for specific measurements then resolved for days.

At this point I’m not 100% sure if it’s an issue from the telegraf tail plugin or InfluxDB writes. I am currently trying to test the “poll” watch_method on telegraf instead of inotify and see if that makes a difference. Surprised with how frequently I see the issue, this hasn’t been noted by anyone else yet.