Suppress data based on configuration table/file?

1

I need to suppress some of the data coming into InfluxDB. Let’s say every point has a Source IP and OID, and I need to discard certain IP/OID combinations that users configured in a file or a DB table. How can I do this?

The best I can think of is periodically generating TICKscript that would include all the “bad” combinations. Something like this:

stream
	|from()
		.measurement('snmp_trap')
	|where(lambda: 
		"source" != 'source1' AND "oid" != 'oid1' AND
		"source" != 'source2' AND "oid" != 'oid2' AND
		"source" != 'source3' AND "oid" != 'oid3' AND
		"source" != 'source4' AND "oid" != 'oid4' AND
		"source" != 'source5' AND "oid" != 'oid5' AND
		"source" != 'source6' AND "oid" != 'oid6' AND
		"source" != 'source7' AND "oid" != 'oid7' AND
		"source" != 'source8' AND "oid" != 'oid8' AND
		"source" != 'source9' AND "oid" != 'oid9' AND
		"source" != 'source10' AND "oid" != 'oid10' AND
		"source" != 'source11' AND "oid" != 'oid11' AND
		"source" != 'source12' AND "oid" != 'oid12' AND
		"source" != 'source13' AND "oid" != 'oid13' AND
		"source" != 'source14' AND "oid" != 'oid14' AND
		"source" != 'source15' AND "oid" != 'oid15' AND
		"source" != 'source16' AND "oid" != 'oid16' AND
		"source" != 'source17' AND "oid" != 'oid17' AND
		"source" != 'source18' AND "oid" != 'oid18' AND
		"source" != 'source19' AND "oid" != 'oid19' AND
		"source" != 'source20' AND "oid" != 'oid20' AND
		"source" != 'source21' AND "oid" != 'oid21' AND
		"source" != 'source22' AND "oid" != 'oid22' AND
		"source" != 'source23' AND "oid" != 'oid23' AND
		"source" != 'source24' AND "oid" != 'oid24' AND
		"source" != 'source25' AND "oid" != 'oid25' AND
		"source" != 'source26' AND "oid" != 'oid26' AND
		"source" != 'source27' AND "oid" != 'oid27' AND
		"source" != 'source28' AND "oid" != 'oid28' AND
		"source" != 'source28' AND "oid" != 'oid29' AND
		"source" != 'source30' AND "oid" != 'oid30' AND
		"source" != 'source31' AND "oid" != 'oid31' AND
		"source" != 'source32' AND "oid" != 'oid32' AND
		"source" != 'source33' AND "oid" != 'oid33' AND
		"source" != 'source34' AND "oid" != 'oid34' AND
		"source" != 'source35' AND "oid" != 'oid35' AND
		"source" != 'source36' AND "oid" != 'oid36' AND
		"source" != 'source37' AND "oid" != 'oid37' AND
		"source" != 'source38' AND "oid" != 'oid38' AND
		"source" != 'source39' AND "oid" != 'oid39' AND
		"source" != 'source40' AND "oid" != 'oid40'
	)
	|alert()
		.info(lambda: TRUE)
		.log('/tmp/filtered.log')

I suspect this is quite inefficient, and also I’m not sure if there is any limit on the lambda’s length?..

Any better ideas (in TICKscript, flux, or anything)?

Thanks!

2

Hello @nekrassov,
How are you writing data to InfluxDB? I would highly recommend checking out the telegraf execd processor plugin to check for these combinations and discard those points that match that criteria before writing to influxdb. You might consider using the telegraf execd processor plugin in combination with metric filtering. telegraf/CONFIGURATION.md at master · influxdata/telegraf · GitHub

Does that help? Or is there some other solution you’re looking for?

3

Thanks, we’ll use the execd plugin for this.
I was not sure if I’m missing some built-in capabilities to do suppression without an external process.

The data is received by one Telegraf instance (SNMP Trap in/Kafka out), then the second Telegraf instance reads from Kafka and inserts into InfluxDB.

1 Like
4

Hello @nekrassov,
The built in suppression would be the metric filtering. But your metric filtering seems advanced. Please let me know if you need any help using the execd processor plugin.