Telegraf Histogram - Can buckets be more granular at different fields level

, ,
1

Hey All,

We have implemented a histogram using telegraf aggregator and we got the buckets. Here is what we have so far for configuration and metrics output.

But we are wondering if it is possible to get our buckets to display based on different URI Strings? So different buckets for different unique URL string.

If this is not possible, how can we filter out inputs into the aggregator based on a certain fields value? Ex. Only aggregate values with CSURI=“google.com”.

Configuration:

[[inputs.logparser]]
    files = [ "H:\\abc\\logs\\metrics_*.log" ]
    from_beginning = false
    name_override = "responseTime_metric"

## Parse logstash-style "grok" patterns:
  [inputs.logparser.grok]
    ## This is a list of patterns to check the given log file(s) for.
    ## Note that adding patterns here increases processing time. The most
    ## efficient configuration is to have one pattern per logparser.
    ## Other common built-in patterns are:
    ##   %{COMMON_LOG_FORMAT}   (plain apache & nginx access logs)
    ##   %{COMBINED_LOG_FORMAT} (access logs + referrer & agent)
    patterns = ['%{URIPATH:uripathStr} %{NOTSPACE:CSURIQuery:drop} %{NUMBER:SPort:drop} %{NOTSPACE:CSUsername:drop} %{IPORHOST:CIP:drop} %{NOTSPACE:CSVersion:drop} %{NOTSPACE:CSUserAgent:drop} %{NOTSPACE:CSCookie:drop} %{NOTSPACE:CSReferer:drop} %{NUMBER:TotalTime:int} ']

Metrics Output:

responseTime_metric_TotalTime{uripathStr="/serverhelp.htm",host="abcdef",servergroup="TEST"} 52

responseTime_metric_TotalTime{host="WSQPWS078",le="+Inf",servergroup="TEST"} 7767
responseTime_metric_TotalTime{host="WSQPWS078",le="100",servergroup="TEST"} 7617
responseTime_metric_TotalTime{host="WSQPWS078",le="200",servergroup="TEST"} 7687
responseTime_metric_TotalTime{host="WSQPWS078",le="500",servergroup="TEST"} 7717
responseTime_metric_TotalTime{host="WSQPWS078",le="1000",servergroup="TEST"} 7757

@daniel

2

It should work this way if the URIPATH is a tag instead of a string field. In your grok pattern make this change:

- %{URIPATH:uripathStr}
+ %{URIPATH:uripathStr:tag}
3

@daniel : Thanks a lot Daniel. That worked. This tool is great.

Now we have an individual bucket for every single URI. Is there a way to use regex and just filter only specific URLs.

For example URL starting with a word hello and ending with asp.

Lets say we have the below 2 URLs but we only want the bucket for first one.
/hello/abc/test.asp
/goodbye/abc/test.asp

Really appreciate your help. Thanks in advance.

4

Hey @daniel : i think we figured it out. We used a custom_pattern for grok parser using regex something like below.

patterns = [’%{URL:uriEndpoint:tag} ‘]
custom_patterns = ‘’’
URL (/hello.*?asp)
‘’’

Can you confirm if this is the right way of doing it and any downsides of doing this?

5

Yes this will work well, any lines not matched by the pattern will be skipped. This is probaby the way to go so long as you don’t need the full log data saved.

The other option would be to match everything but remove unwanted matches or have them bypass the aggregator using the metric filtering options. However, then you are restricted to less powerful glob patterns instead of regular expressions.

6

Nice. Thanks for confirming. Really appreciate.