Telegraf don't generate measurement

sai_bug · March 30, 2021, 10:04am

Hi folks,

I have an issue to sending data in influxdb, with telegraf plugin .
i saw the metrics on telegraf log but the measurement not created in influxdb !!

Here my telegraf config :

[[inputs.tail]]
  files = ["/tmp/all_log.log"]
  from_beginning = false
  grok_patterns = ['Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}" StateInd="%{NUMBER:StateInd:int}" ErrCode="%{NUMBER:ErrCode:int}" ErrMess="%{DATA:ErrMess:string}" AccessProvider="%{DATA:AccessProvider:string}" Login="%{DATA:Login:string}" Nspace="%{DATA:Nspace}"','Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}" StateInd="%{NUMBER:StateInd:int}" ErrCode="%{NUMBER:ErrCode:int}" ErrMess="%{DATA:ErrMess:string}" AccessProvider="%{DATA:AccessProvider:string}" Login="%{DATA:Login:string}" Nspace="%{DATA:Nspace:string}" Address="%{DATA:Address:string}" IpType="%{NUMBER:IpType:int}" CtrlOption="%{NUMBER:CtrlOption:int}"']
  data_format = "grok"
  grok_timezone = "Local"
  name_override = "all_log"

Actual behavior :
No indication that plugin tail is sending data to influxdb

Expected behavior :
running show measurements should print tail measurement …

Thanks for your help

Franky1 · March 30, 2021, 10:48am

Can you provide a few sample lines of your log file in markdown format?

```log
put the log lines samples here
```

And can you also provide some sample lines of the Telegraf output?
Add this to your telegraf.conf file:

[[outputs.file]]  # only for debugging
  namepass = ["all_log"]
  files = ["all_log.out"]
  influx_sort_fields = true

sai_bug · March 30, 2021, 12:16pm

here log lines from app :

Timestamp="2021-03-30 13:57:00.379" TraceId="SSP16663eb4254dc71234f171e869e13" ProbeName="probe" LogType="KPI_PROBE" MessType="Info_Main" ProcTime="7.332801818847656" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user1" Nspace="1"
Timestamp="2021-03-30 13:57:00.887" TraceId="SSP4963d4c144f046b009bf68d0601e5" ProbeName="probe" LogType="KPI_PROBE" MessType="Meters" ProcTime="12.027263641357422" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user2" Nspace="1"
Timestamp="2021-03-30 13:57:01.399" TraceId="SSP55e177bcbbd59a0fab824819eb599" ProbeName=probe" LogType="KPI_PROBE" MessType="to_Login" ProcTime="13.0767822265625" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user3" Nspace="1

and log in outputs.file, it’s very strange, i’m unable to get metrics for tail plugin here !!

diskio,host=server1,type=sso,name=dm-5     io_time=7342i,iops_in_progress=0i,merged_reads=0i,merged_writes=0i,read_bytes=178930688i,read_time=3551i,reads=1506i,weighted_io_time=19246i,write_bytes=49000448i,write_time=15695i,writes=7660i 1617105640000000000
diskio,host=server2,type=sso,name=dm-6 io_time=4829i,iops_in_progress=0i,merged_reads=0i,merged_writes=0i,read_bytes=177628160i,read_time=3673i,reads=1453i,weighted_io_time=7939i,write_bytes=978944i,write_time=4265i,writes=239i 1617105640000000000

Franky1 · March 30, 2021, 12:23pm

I assume your grok pattern does not match, therefore you have no output data at all from the tail plugin.

sai_bug · March 30, 2021, 12:32pm

humm, but when i run telegraf --test , i saw the plugin output
all_sso,MessType=Info_Main,host=server1,type=sso,path=/tmp/all_1.log AccessProvider="wfr",ErrCode=0i,ErrMess="OK Result found",LogType="KPI_PROBE",Login="user1",Nspace="1",ProbeName="ics_probe",ProcTime=9.063005447387695,StateInd=0i,TraceId="SSP1c2a0405e1e824f2ef28a062840a8" 1617107095867000000

Franky1 · March 30, 2021, 12:50pm

Hmm, ok, than the grok pattern seems to work

Does the log file actually get new lines as well?
Do these new lines look the same?
The configuration under the [agent] section in the config file also influences when and how often the tail plugin is called.

sai_bug · March 30, 2021, 12:59pm

yes the logh file get new lines , the log is send continuously .
it’s not same lines, but all sections are defined on grok_patterns option .
here main agent config :

interval = “10s”
debug = false
hostname = “server1”
round_interval = true
flush_interval = “10s”
flush_jitter = “0s”

Franky1 · March 30, 2021, 1:46pm

It looks like only one line is parsed with the --test option.
So you only see if the grok pattern works for that line.

I would change the following:

Take a log file with many lines already present.
Change the [[inputs.tail]] config to:

from_beginning = true

Start Telegraf normally, without the --test option.
Compare if all lines are really captured by the grok pattern.

sai_bug · March 30, 2021, 2:05pm

yes it works with changing option from_beginning to true, and all lines are captured.
But telegraf don’t generate any measurement !!
For me, from the moment when it generates metrics, it must send to influxdb to create the table .
you agree ?

Franky1 · March 30, 2021, 2:17pm

I don’t understand this statement - it contradicts itself.
How can you tell that all lines have been “captured”?
How can you tell that no “measurement” was created?

sai_bug · March 30, 2021, 2:26pm

i meant, when the option is changed to true, all lines are captured by grok pattern. i have the same count betwen app log and telegraf debug log.
And in influxdb i didn’t show the measurement
as mentioned above , i use name_override option

Note that this behavior is only in production, also staging works fine (the same telegraf conf)

Franky1 · March 30, 2021, 2:32pm

Then I think we’re looking in completely the wrong place.

That would have been important information…

sai_bug · March 30, 2021, 2:49pm

I’m going crazy ,
the same versions of telegraf & influxdb on both platforms.
i thought is was retention policy !, but it’s the same …

Franky1 · March 30, 2021, 3:45pm

If you enable the debug = true option for the Telegraf agent that is not working, does it spit out any suspicious messages?
Does anything arrive in the “production” InfluxDB from this Telegraf agent at all?
Do the TOKENs have write permission to the bucket?
Does the precision match?

sai_bug · March 30, 2021, 3:42pm

solved with adding debug mode, telegraf is teling that it don’t has right access to log file !! so i have just modified this .
it’s strange, why i didn’t see this in telegraf before !!

thanks for your help man