Telegraf don't generate measurement

Telegraf
,
#1

Hi folks,

I have an issue to sending data in influxdb, with telegraf plugin .
i saw the metrics on telegraf log but the measurement not created in influxdb !!

Here my telegraf config :

[[inputs.tail]]
  files = ["/tmp/all_log.log"]
  from_beginning = false
  grok_patterns = ['Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}" StateInd="%{NUMBER:StateInd:int}" ErrCode="%{NUMBER:ErrCode:int}" ErrMess="%{DATA:ErrMess:string}" AccessProvider="%{DATA:AccessProvider:string}" Login="%{DATA:Login:string}" Nspace="%{DATA:Nspace}"','Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}" StateInd="%{NUMBER:StateInd:int}" ErrCode="%{NUMBER:ErrCode:int}" ErrMess="%{DATA:ErrMess:string}" AccessProvider="%{DATA:AccessProvider:string}" Login="%{DATA:Login:string}" Nspace="%{DATA:Nspace:string}" Address="%{DATA:Address:string}" IpType="%{NUMBER:IpType:int}" CtrlOption="%{NUMBER:CtrlOption:int}"']
  data_format = "grok"
  grok_timezone = "Local"
  name_override = "all_log"

Actual behavior :
No indication that plugin tail is sending data to influxdb

Expected behavior :
running show measurements should print tail measurement …

Thanks for your help

#2

Can you provide a few sample lines of your log file in markdown format?

```log
put the log lines samples here
```

And can you also provide some sample lines of the Telegraf output?
Add this to your telegraf.conf file:

[[outputs.file]]  # only for debugging
  namepass = ["all_log"]
  files = ["all_log.out"]
  influx_sort_fields = true
#3

here log lines from app :

Timestamp="2021-03-30 13:57:00.379" TraceId="SSP16663eb4254dc71234f171e869e13" ProbeName="probe" LogType="KPI_PROBE" MessType="Info_Main" ProcTime="7.332801818847656" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user1" Nspace="1"
Timestamp="2021-03-30 13:57:00.887" TraceId="SSP4963d4c144f046b009bf68d0601e5" ProbeName="probe" LogType="KPI_PROBE" MessType="Meters" ProcTime="12.027263641357422" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user2" Nspace="1"
Timestamp="2021-03-30 13:57:01.399" TraceId="SSP55e177bcbbd59a0fab824819eb599" ProbeName=probe" LogType="KPI_PROBE" MessType="to_Login" ProcTime="13.0767822265625" StateInd="0" ErrCode="0" ErrMess="OK Result found" AccessProvider="wfr" Login="user3" Nspace="1

and log in outputs.file, it’s very strange, i’m unable to get metrics for tail plugin here !!

diskio,host=server1,type=sso,name=dm-5     io_time=7342i,iops_in_progress=0i,merged_reads=0i,merged_writes=0i,read_bytes=178930688i,read_time=3551i,reads=1506i,weighted_io_time=19246i,write_bytes=49000448i,write_time=15695i,writes=7660i 1617105640000000000
diskio,host=server2,type=sso,name=dm-6 io_time=4829i,iops_in_progress=0i,merged_reads=0i,merged_writes=0i,read_bytes=177628160i,read_time=3673i,reads=1453i,weighted_io_time=7939i,write_bytes=978944i,write_time=4265i,writes=239i 1617105640000000000
#4

I assume your grok pattern does not match, therefore you have no output data at all from the tail plugin.

#5

humm, but when i run telegraf --test , i saw the plugin output :frowning:
all_sso,MessType=Info_Main,host=server1,type=sso,path=/tmp/all_1.log AccessProvider="wfr",ErrCode=0i,ErrMess="OK Result found",LogType="KPI_PROBE",Login="user1",Nspace="1",ProbeName="ics_probe",ProcTime=9.063005447387695,StateInd=0i,TraceId="SSP1c2a0405e1e824f2ef28a062840a8" 1617107095867000000

#6

Hmm, ok, than the grok pattern seems to work :thinking:

  • Does the log file actually get new lines as well?
  • Do these new lines look the same?
  • The configuration under the [agent] section in the config file also influences when and how often the tail plugin is called.
#7

  • yes the logh file get new lines , the log is send continuously .

  • it’s not same lines, but all sections are defined on grok_patterns option .

  • here main agent config :

    interval = “10s”
    debug = false
    hostname = “server1”
    round_interval = true
    flush_interval = “10s”
    flush_jitter = “0s”

#8

It looks like only one line is parsed with the --test option.
So you only see if the grok pattern works for that line.

I would change the following:

  1. Take a log file with many lines already present.
  2. Change the [[inputs.tail]] config to:
from_beginning = true
  1. Start Telegraf normally, without the --test option.
  2. Compare if all lines are really captured by the grok pattern.
#9

yes it works with changing option from_beginning to true, and all lines are captured.
But telegraf don’t generate any measurement !!
For me, from the moment when it generates metrics, it must send to influxdb to create the table .
you agree ?

#10

I don’t understand this statement - it contradicts itself.
How can you tell that all lines have been “captured”?
How can you tell that no “measurement” was created?

#11

i meant, when the option is changed to true, all lines are captured by grok pattern. i have the same count betwen app log and telegraf debug log.
And in influxdb i didn’t show the measurement
as mentioned above , i use name_override option

Note that this behavior is only in production, also staging works fine (the same telegraf conf) :confused:

#12

Then I think we’re looking in completely the wrong place.

That would have been important information… :smirk:

#13

I’m going crazy ,
the same versions of telegraf & influxdb on both platforms.
i thought is was retention policy !, but it’s the same …

#14
  • If you enable the debug = true option for the Telegraf agent that is not working, does it spit out any suspicious messages?
  • Does anything arrive in the “production” InfluxDB from this Telegraf agent at all?
  • Do the TOKENs have write permission to the bucket?
  • Does the precision match?
#15

solved with adding debug mode, telegraf is teling that it don’t has right access to log file !! so i have just modified this .
it’s strange, why i didn’t see this in telegraf before !!

thanks for your help man :slight_smile: