Hello everyone,
I am a newbie at Telegraf, as a matter of fact this is the first time I am trying to use it. Please be gentle…
Background: I have a Blue Iris server which holds several cameras. Each camera is configured to send a MQTT payload whenever an alert is triggered. The payload format is the same for each camera, generic payload format is listed below:
Camera=&CAM,Alert_Type=&TYPE,AI_Finding=&MEMO,Server_Name=&SERVER,Timestamp=&ALERT_TIME
Here’s a few examples of how that looks like, as a MQTT payload:
Camera=Garaj,Alert_Type=,AI_Finding=,Server_Name=Blue Iris,Timestamp=2024-08-13T22:23:32.542Z
Camera=Gate,Alert_Type=Group,AI_Finding=car:61%,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:44.502Z
Camera=Roof,Alert_Type=Group,AI_Finding=,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:42.073Z
Camera=Street4K,Alert_Type=Motion_A,AI_Finding=car:86%,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:42.072Z
As such:
$CAM = Camera Name
&TYPE = Alert Type (‘Motion_A’, ‘Group’, ‘Motion_AB’, ‘Group, Motion_A’, etc.) - this also can be an empty string.
&MEMO = The alert’s AI findings, e.g., person:81%, car:92% - this part could also be an empty string if camera was triggered by a group trigger, but it didn’t recognize motion or object.
&SERVER = this is the server name. Blue Iris uses a variable to populate it. I don’t intend on having more than one server, but I would like this to be parsed anyway, to make it applicable to various situations.
“at” = standard string, never changes
&ALERT_TIME = The alert record’s creation time in ISO8601 format.
Note: I have control over the payload formatting, therefore if there is a better way to present these strings to Telegraf for processing, please let me know.
Scope:
I would like to parse these payloads (values represented by strings) via Telegraf in such a way that they make sense in InfluxDB. Ultimately, they would be aggregated in a Grafana dashboard. Both InfluxDB and Grafana are installed and working on my main server. I had tried MQTT as a data source in Grafana, but the data is obviously transient and that doesn’t help.
Status:
After reading documentation and politely asking for help from LLM chatbots, I ended up with the following configuration file:
[agent]
## Default data collection interval for all inputs
interval = "10s"
## Log at debug level.
debug = true
## Log only error level messages.
quiet = false
[[processors.regex]]
namepass = ["mqtt_consumer"]
[[processors.regex.fields]]
key = "_value"
pattern = "Camera=(?P<camera>[^,]+),Alert_Type=(?P<alert_type>[^,]*),AI_Finding=(?P<ai_finding>[^,]*),Server_Name=(?P<server>[^,]+),Timestamp=(?P<alert_time>[^,]+)"
result_key = "parsed_fields"
[[processors.regex]]
namepass = ["mqtt_consumer"]
[[processors.regex.fields]]
key = "parsed_fields"
pattern = "Alert_Type=(?P<alert_type>[^,]*),AI_Finding=(?P<ai_finding>[^,]*)"
result_key = "fields"
[[processors.regex]]
namepass = ["mqtt_consumer"]
[[processors.regex.fields]]
key = "fields"
pattern = "Alert_Type=(?P<alert_type>[a-zA-Z]*),AI_Finding=(?P<object_type>[a-zA-Z]*)(?::(?P<confidence>\\d+))?%?"
result_key = "final_fields"
[[processors.converter]]
[processors.converter.fields]
string = ["camera", "alert_type", "object_type", "server", "alert_time"]
float = ["confidence"]
[processors.converter.tags]
tagpass = ["alert_type", "confidence"]
[[inputs.mqtt_consumer]]
client_id = "telegraf"
username = "__redacted__"
password = "__redacted__"
servers = ["tcp://192.168.2.11:1883"]
topics = ["BlueIris/telegraf/#"]
qos = 0
connection_timeout = "30s"
persistent_session = false
## Data format to use for messages
data_format = "value"
data_type = "string"
[[processors.parser]]
parse_fields = ["_value"]
data_format = "value"
[[outputs.influxdb_v2]]
urls = ["http://192.168.2.10:8086"]
token = "_redacted__"
organization = "Home"
bucket = "MQTT_DEV"
[[outputs.file]] # Log output to file for debugging
files = ["stdout", "/tmp/telegraf_parsed_output.log"]
data_format = "json"
InfluxDB is populated, but with the raw content of the MQTT payload, no regex processing occurs.
Checking the output log from Telegraf processing yields the same result.
# tail -4 telegraf_parsed_output.log
{"fields":{"value":"Camera=Garaj,Alert_Type=,AI_Finding=,Server_Name=Blue Iris,Timestamp=2024-08-13T22:23:32.542Z"},"name":"mqtt_consumer","tags":{"host":"Tower","topic":"BlueIris/telegraf/Garage"},"timestamp":1723587812}
{"fields":{"value":"Camera=Roof,Alert_Type=Group,AI_Finding=,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:42.073Z"},"name":"mqtt_consumer","tags":{"host":"Tower","topic":"BlueIris/telegraf/Roof"},"timestamp":1723587881}
{"fields":{"value":"Camera=Street4K,Alert_Type=Motion_A,AI_Finding=car:86%,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:42.072Z"},"name":"mqtt_consumer","tags":{"host":"Tower","topic":"BlueIris/telegraf/Street4K"},"timestamp":1723587884}
{"fields":{"value":"Camera=Gate,Alert_Type=Group,AI_Finding=car:61%,Server_Name=Blue Iris,Timestamp=2024-08-13T22:24:44.502Z"},"name":"mqtt_consumer","tags":{"host":"Tower","topic":"BlueIris/telegraf/Gate"},"timestamp":1723587886}
I know something’s missing in my regex, most likely robust behavior for possible empty strings, but I am not able to fix it.
Any help would be much appreciated!
