Hi, I’m trying to collect Apache logs and store them in InfluxDB. If I use the default patter for combined log (patterns = ["%{COMBINED_LOG_FORMAT}"]), the user-agent information is stored properly. In the actual logs, this is a “quoted string”, and Telegraf properly stuffs the data into InfluxDB without the double quotes on either end.
However, I’ve decided to make the user-agent data a tag in my Influx DB, and so my pattern instead looks like this: %{NOTSPACE:application:tag} %{COMMON_LOG_FORMAT} %{QS:referrer} %{QS:agent:tag}
The piece at the beginning is additional info in our apache logs (virtual host and port information). Per the Telegraf documentation, all COMBINED_LOG_FORMAT is, is this: %{COMMON_LOG_FORMAT} %{QS:referrer} %{QS:agent}
So I merely took that string and just set agent to be a tag… and now, when the data is stored in InfluxDB, I’m getting “escaped double quotes” on either side of the data… so instead of ELB-HealthChecker/1.0, I’m now getting \"ELB-HealthChecker/1.0\".
Per the COMMON_LOG_FORMAT, the response code (resp_code) is also set as a tag, by default. These codes, however, are not saved with the additional quotes. So I can’t believe that it’s the “tag” difference causing this… must be something about %QS?
Any help would be greatly appreciated. I would really like to save the user-agent data as a tag, so I can perform group by statements and such… but I really don’t want to have to deal with the crazy “escaped double quotes” as part of queries and such. Not to mention, I can’t even really get a query to work by specifying a particular user-agent.
apache log example: example.company.com:80 10.22.28.12 - - [03/Aug/2017:19:52:42 +0000] "GET /v2/user_question_answers/?filter%5Bquiz_id%5D=4 HTTP/1.1" 200 813 "https://staging.company.com/app/quiz/4?taking=true" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
The Telegraf configuration is as follows:
# # Stream and parse log file(s).
[[inputs.logparser]]
# ## Log files to parse.
# ## These accept standard unix glob matching rules, but with the addition of
# ## ** as a "super asterisk". ie:
# ## /var/log/**.log -> recursively find all .log files in /var/log
# ## /var/log/*/*.log -> find all .log files with a parent dir in /var/log
# ## /var/log/apache.log -> only tail the apache log file
files = ["/var/log/apache2/*_access.log"]
# ## Read files that currently exist from the beginning. Files that are created
# ## while telegraf is running (and that match the "files" globs) will always
# ## be read from the beginning.
# from_beginning = false
#
# ## Parse logstash-style "grok" patterns:
# ## Telegraf built-in parsing patterns: https://goo.gl/dkay10
[inputs.logparser.grok]
# ## This is a list of patterns to check the given log file(s) for.
# ## Note that adding patterns here increases processing time. The most
# ## efficient configuration is to have one pattern per logparser.
# ## Other common built-in patterns are:
# ## %{COMMON_LOG_FORMAT} (plain apache & nginx access logs)
# ## %{COMBINED_LOG_FORMAT} (access logs + referrer & agent)
patterns = ["%{CUSTOM_ACCESS_LOG}"]
# ## Name of the outputted measurement name.
measurement = "apache_access_log"
# ## Full path(s) to custom pattern files.
# custom_pattern_files = []
# ## Custom patterns can also be defined here. Put one pattern per line.
custom_patterns = '''
CUSTOM_ACCESS_LOG %{NOTSPACE:application:tag} %{COMMON_LOG_FORMAT} %{QS:referrer} %{QS:agent:tag}
'''
I have figured out a workaround which is to use this custom_pattern instead: CUSTOM_ACCESS_LOG %{NOTSPACE:application:tag} %{COMMON_LOG_FORMAT} %{QS:referrer} "%{DATA:agent:tag}"
I just don’t understand why the other way doesn’t work… without :tag it works fine… but add the :tag and it acts weird…
Based on a little searching, the QS pattern is meant to return the string with quotes. The reason you don’t see it when creating a field is that we are trimming outer quotes from fields, but not from tags, as you can see here. I’m not sure that this is the right thing to do as it does make it impossible to have a field with outer quotes, but it is definitely convenient 99% of the time.
You may not want to store the agent string as a tag, there are so many variants of it you might end up with very high series cardinality.
I opened up an issue in the Telegraf issue tracker #3092. I’m going to investigate a workaround before deciding how this should be addressed.
Thanks!!! Yeah, I kind of went back and forth on whether to make the user-agent data a tag or not… but I was thinking about the different data that might be interesting to see, and being able to group by user-agent seemed interesting… Also, not having it a tag also makes it hard to visualize by user-agent in Grafana or something too (again, because of not being able to group by if the data point is not a tag)…
I’ll keep an eye on the series cardinality… but overall, I’m only really at ~1200 right now… If it seems to get out of hand, I’ll re-evaluate. Unless you really discourage setting something like user-agent as a tag for some reason… thanks for the reply!!!
