No data points influxdb [grok pattern]

Hi team
i have set telegraf conf to get metrics for python app, see below the telegraf conf . It works fine .

[[inputs.tail]]
  files = ["probesso.log"]
  from_beginning = false
  #[inputs.logparser.grok]
  grok_patterns =  ["%{PROBESSO_LOG}"]
  grok_custom_patterns = '''
  PROBESSO_LOG Timestamp=\"%{TIMESTAMP_ISO8601:time}\" TraceId=\"%{DATA:Trace_Id}\" ProbeName=\"%{DATA:Probe_Name}\" LogType=\"%{DATA:LogType:tag}\" MessType=\"%{DATA:Message_type:tag}\" ProcTime=\"%{NUMBER:Proc_time}\"
''' 
  name_override = "sso_queries_log"
  data_format = "grok"

here the grafana query

From influxdb no rows returnd when i set the time condition :frowning:
SELECT count("Message_type") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "Message_type" = 'Meters_to_Login') AND time >= now() -3h GROUP BY time(1h) fill(null) ORDER BY time DESC LIMIT 20

Any suggestions ?

thanks

How do you know that?

I would first check whether the plugin in Telegraf is working properly, whether the output is formatted properly and whether Telegraf is sending data to InfluxDB at all.

I would add the outputs.file plugin to your telegraf.conf to check this. This output plugin allows you to send the output of Telegraf to a file or to the console. This way you can check the function of the plug-ins in Telegraf quite easily.

@Franky1 thanks for the reply !!!

here the output of file

sso_queries_log,LogType=KPI_PROBE_SSO,Message_type=Info_Main,host=host.sso, time="2021-03-04 14:16:50.768",Trace_Id="SSP657f2a4e1c5cbc662fa2f83fac85f",Probe_Name="ics_probe",Proc_time="9.58395004272461" 1614863810778576995

anf when i run telegraf --test , i have the same result !!

@Franky1 any feedback ?

thanks a lot

I am not a support person from influxdata - so some patience please! :angry:

I am not familiar with the grok input format, but it looks to me like there is something wrong with the grok pattern.

  • I find the extra quotes suspicious.
  • time is read as string
  • ProcTime is read as string

So what does a line from the original log file look like?

Please compare again your grok pattern with the documentation:

https://docs.influxdata.com/telegraf/v1.17/data_formats/input/grok/

Sorry for the spam :’

here original log line,

Timestamp="2021-03-04 15:17:42.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"

As I wrote before, I am not familiar with the ngrok format, but what I notice is:

  • the timestamp in nanoseconds does not match the time string in the log file
  • the time string in the logfile is not read as timestamp, but as field
  • instead the ingress time is used as timestamp
  • the timezone setting in the config is missing

I have fiddled a bit with the ngrok format and I think I have found a solution that works.

A snippet with 3 lines of fake input data according to your logfile:

Timestamp="2021-03-04 21:17:42.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"
Timestamp="2021-03-04 21:17:43.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce85" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"
Timestamp="2021-03-04 21:17:44.919" TraceId="SSP474d3f90447a256f6d5ebaaa2ce86" ProbeName="infra_probe" LogType="KPI_INFRA_SSO" MessType="Meters_to_Login" ProcTime="9.627580642700195" StateInd="3" ErrCode="95" ErrMess="KO technical error" AccessProvider="UN" Login="10600752"

The configuration snippet of Telegraf:

[[inputs.tail]]
  files = ["probesso.log"]
  from_beginning = false
  grok_patterns =  ['Timestamp="%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05.000"}" TraceId="%{DATA:TraceId:string}" ProbeName="%{DATA:ProbeName:string}" LogType="%{DATA:LogType:string}" MessType="%{DATA:MessType:tag}" ProcTime="%{NUMBER:ProcTime:float}"']
  data_format = "grok"
  grok_timezone = "Local"
  name_override = "sso_queries_log"

[[outputs.file]] # only for debugging
  files = ["probesso.out"]
  tagexclude = ["path"]
  influx_sort_fields = true

The snippet with the 3 output lines of Telegraf in influx line protocol format:

sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce84" 1614889062919000000
sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce85" 1614889063919000000
sso_queries_log,MessType=Meters_to_Login,host=localprobe LogType="KPI_INFRA_SSO",ProbeName="infra_probe",ProcTime=9.627580642700195,TraceId="SSP474d3f90447a256f6d5ebaaa2ce86" 1614889064919000000

I would say that looks good :wink:

Thanks for the reply :slight_smile:
Yes it works
but from grafana i have no data points
example query
SELECT count("MessType") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "MessType" =~ /^$MessType$/) AND $timeFilter GROUP BY time($__interval) fill(null)

any suggestions

I would first check InfluxDB itself to see if the data points are there!?

Unfortunately I can’t help with the query, I don’t know InfluxQL.

query from influxdb
"SELECT count(MessType) FROM sso_queries_log WHERE (host = 'host.sso' AND MessType =~ /^Info_Main$/) AND time >= now() - 1h GROUP BY time(1m) fill(null)

no row returned

MessType is specified as tag, so i put variable on in grafana config

As I wrote before - please leave Grafana and queries out of it - are the metrics even in InfluxDB?

Yes indeed

time                AccessProvider Address ErrCode ErrCode_1 ErrMess                       IpType LogType       Login               MessType  MessType_1 Nspace ProbeName ProcTime           StateInd Timestamp               TraceId                          host           metrics_type path
----                -------------- ------- ------- --------- -------                       ------ -------       -----               --------  ---------- ------ --------- --------           -------- ---------               -------                          ----           --------- ----
1614880677772331391 wfr                    95                KO ICS Legacy technical error        KPI_PROBE_SSO USER022 Info_Main            1      ics_probe 11.002540588378906 3        2021-03-04 18:57:57.760 SSP6d280b6914e3acd0916ec460bbf37 host.sso sso       probesso.log

Then I would say your query just doesn’t return any hits?
Proceed systematically: Simplify the query until something is found. For example, increase the time window? Then increase the complexity of your query again?

with other query
SELECT count("MessType") FROM "sso_queries_log" WHERE ("host" = 'host.sso' AND "MessType" = 'Info_Main') AND time >= now() - 24h GROUP BY time(2m) fill(previous)

with this time range i have data points

but i set from today, no data returned :confused:

Then I would say something is wrong somewhere with the timestamp. Either on the machine where Telegraf is running. Or the timezone does not match between Telegraf and Grafana?

according to no data points is displaying on graph · Issue #10204 · grafana/grafana · GitHub and
No data Points on grafana dashboard - #8 by jezekjan - InfluxDB - Grafana Labs Community Forums
it seems like time issue

solved with defined the two patterns on same grok_patterns

Please suggest grok pattern to filter only 500 error code logs