No data points influxdb [grok pattern]

sai_bug · March 4, 2021, 11:29am

Hi team
i have set telegraf conf to get metrics for python app, see below the telegraf conf . It works fine .

[[inputs.tail]]
  files = ["probesso.log"]
  from_beginning = false
  #[inputs.logparser.grok]
  grok_patterns =  ["%{PROBESSO_LOG}"]
  grok_custom_patterns = '''
  PROBESSO_LOG Timestamp=\"%{TIMESTAMP_ISO8601:time}\" TraceId=\"%{DATA:Trace_Id}\" ProbeName=\"%{DATA:Probe_Name}\" LogType=\"%{DATA:LogType:tag}\" MessType=\"%{DATA:Message_type:tag}\" ProcTime=\"%{NUMBER:Proc_time}\"
''' 
  name_override = "sso_queries_log"
  data_format = "grok"

here the grafana query

From influxdb no rows returnd when i set the time condition
SELECT count("Message_type") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "Message_type" = 'Meters_to_Login') AND time >= now() -3h GROUP BY time(1h) fill(null) ORDER BY time DESC LIMIT 20

Any suggestions ?

thanks

Franky1 · March 4, 2021, 12:19pm

How do you know that?

I would first check whether the plugin in Telegraf is working properly, whether the output is formatted properly and whether Telegraf is sending data to InfluxDB at all.

I would add the outputs.file plugin to your telegraf.conf to check this. This output plugin allows you to send the output of Telegraf to a file or to the console. This way you can check the function of the plug-ins in Telegraf quite easily.

sai_bug · March 4, 2021, 1:23pm

@Franky1 thanks for the reply !!!

here the output of file

sso_queries_log,LogType=KPI_PROBE_SSO,Message_type=Info_Main,host=host.sso, time="2021-03-04 14:16:50.768",Trace_Id="SSP657f2a4e1c5cbc662fa2f83fac85f",Probe_Name="ics_probe",Proc_time="9.58395004272461" 1614863810778576995

anf when i run telegraf --test , i have the same result !!

sai_bug · March 4, 2021, 2:09pm

@Franky1 any feedback ?

thanks a lot

Franky1 · March 4, 2021, 2:16pm

I am not a support person from influxdata - so some patience please!

I am not familiar with the grok input format, but it looks to me like there is something wrong with the grok pattern.

I find the extra quotes suspicious.
time is read as string
ProcTime is read as string

So what does a line from the original log file look like?

Please compare again your grok pattern with the documentation:

sai_bug · March 4, 2021, 2:21pm

Sorry for the spam :’

here original log line,

Timestamp="2021-03-04 15:17:42.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"

Franky1 · March 4, 2021, 2:44pm

As I wrote before, I am not familiar with the ngrok format, but what I notice is:

the timestamp in nanoseconds does not match the time string in the log file
the time string in the logfile is not read as timestamp, but as field
instead the ingress time is used as timestamp
the timezone setting in the config is missing

Franky1 · March 4, 2021, 9:53pm

I have fiddled a bit with the ngrok format and I think I have found a solution that works.

A snippet with 3 lines of fake input data according to your logfile:

Timestamp="2021-03-04 21:17:42.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"
Timestamp="2021-03-04 21:17:43.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce85" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"
Timestamp="2021-03-04 21:17:44.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce86" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"

The configuration snippet of Telegraf:

[[inputs.tail]]
  files = ["probesso.log"]
  from_beginning = false
  grok_patterns =  ['Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}"']
  data_format = "grok"
  grok_timezone = "Local"
  name_override = "sso_queries_log"

[[outputs.file]] # only for debugging
  files = ["probesso.out"]
  tagexclude = ["path"]
  influx_sort_fields = true

The snippet with the 3 output lines of Telegraf in influx line protocol format:

sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" 1614889062919000000
sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce85" 1614889063919000000
sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce86" 1614889064919000000

I would say that looks good

sai_bug · March 5, 2021, 10:27am

Thanks for the reply
Yes it works
but from grafana i have no data points
example query
SELECT count("MessType") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "MessType" =~ /^$MessType$/) AND $timeFilter GROUP BY time($__interval) fill(null)

any suggestions

Franky1 · March 5, 2021, 10:46am

I would first check InfluxDB itself to see if the data points are there!?

Unfortunately I can’t help with the query, I don’t know InfluxQL.

sai_bug · March 5, 2021, 10:55am

query from influxdb
"SELECT count(MessType) FROM sso_queries_log WHERE (host = 'host.sso' AND MessType =~ /^Info_Main$/) AND time >= now() - 1h GROUP BY time(1m) fill(null)

no row returned

MessType is specified as tag, so i put variable on in grafana config

Franky1 · March 5, 2021, 11:19am

As I wrote before - please leave Grafana and queries out of it - are the metrics even in InfluxDB?

sai_bug · March 5, 2021, 11:23am

Yes indeed

time                AccessProvider Address ErrCode ErrCode_1 ErrMess                       IpType LogType       Login               MessType  MessType_1 Nspace ProbeName ProcTime           StateInd Timestamp               TraceId                          host           metrics_type path
----                -------------- ------- ------- --------- -------                       ------ -------       -----               --------  ---------- ------ --------- --------           -------- ---------               -------                          ----           --------- ----
1614880677772331391 wfr                    95                KO ICS Legacy technical error        KPI_PROBE_SSO USER022 Info_Main            1      ics_probe 11.002540588378906 3        2021-03-04 18:57:57.760 SSP6d280b6914e3acd0916ec460bbf37 host.sso sso       probesso.log

Franky1 · March 5, 2021, 11:36am

Then I would say your query just doesn’t return any hits?
Proceed systematically: Simplify the query until something is found. For example, increase the time window? Then increase the complexity of your query again?

sai_bug · March 5, 2021, 11:44am

with other query
SELECT count("MessType") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "MessType" = 'Info_Main') AND time >= now() - 24h GROUP BY time(2m) fill(previous)

with this time range i have data points

but i set from today, no data returned

Franky1 · March 5, 2021, 11:50am

Then I would say something is wrong somewhere with the timestamp. Either on the machine where Telegraf is running. Or the timezone does not match between Telegraf and Grafana?

sai_bug · March 5, 2021, 12:06pm

according to no data points is displaying on graph · Issue #10204 · grafana/grafana · GitHub and
No data Points on grafana dashboard - #8 by jezekjan - InfluxDB - Grafana Labs Community Forums
it seems like time issue

sai_bug · March 10, 2021, 4:17pm

solved with defined the two patterns on same grok_patterns

ramyam07 · February 19, 2022, 5:17am

Please suggest grok pattern to filter only 500 error code logs

Topic		Replies	Views
Telegraf - inputs.logparser problems - Grok no match Telegraf influxdb , telegraf	7	5382	April 26, 2018
Got big trouble in grok patterns Telegraf telegraf	10	2584	December 6, 2018
Telegraf tail-plugin with grok: no data written to influxdb Telegraf influxdb , grok , tail	4	794	November 17, 2022
Telegraf don't generate measurement Telegraf influxdb , telegraf	14	2258	March 30, 2021
Possibility to store the data into influxDB through telegraf based on condition like if timestamp and level is same we need to restrict storing into InfluxDB Telegraf telegraf	3	482	July 8, 2019

No data points influxdb [grok pattern]

Related topics