Telegraf writing in multi databases with no permissions

German_Roldan · April 15, 2022, 4:14pm

Hi. I have 3 databases, each of them have its own retention policy:

> show databases
name: databases
name
----
_internal
COLLECTOR-1
COLLECTOR-3
COLLECTOR-6

Also, every database has one specified user to write on it.

> show grants for DATAIN
database    privilege
--------    ---------
COLLECTOR-1 WRITE

> show grants for DATAIN3
database    privilege
--------    ---------
COLLECTOR-3 WRITE

> show grants for DATAIN6
database    privilege
--------    ---------
COLLECTOR-6 WRITE

I commented all the [[output.influxdb]] in /etc/telegraf/telegraf.conf because I wanted that every metric collector in /etc/telegraf/telegraf.d/ has its own database output. For example:

[[outputs.influxdb]]
	urls = ["http://localhost:8086"]
	database = "COLLECTOR-1"
	database_tag = "COLLECTOR-1"
	exclude_database_tag = false
#	skip_database_creation = true
# Retention policy commented to avoid errors in influxd logs.
#	retention_policy = "30-days-retention"
	timeout = "0s"
#	# HTTP Basic Auth
	username = "DATAIN"
	password = "LALALALA"

This is working fine, the Telegraf doesn’t report errors.The problem is that all the metrics are writing in the 3 databases at the same time.

I think it’s a Telegraf issue because I have Windows servers sending metrics to COLLECTOR-1 database and those metrics are not storing in COLLECTOR-3 or -6, which is fine, it’s the way should be.

I was changing the users permissions and every test resulted as expected. The problem is when the service Telegraf starts. I don’t know what else to check. Any idea?

reimda · April 18, 2022, 9:40pm

Hi, which versions of InfluxDB and Telegraf are you using?

Are you sure there are no more [[outputs.influxdb]] sections in /etc/telegraf/telegraf.conf or in any of the files under /etc/telegraf/telegraf.d?

German_Roldan · April 19, 2022, 1:25am

Hi, Reimda. Thank you for your interest. I have Telegraf 1.22.1 (git: HEAD fc8301ae) and InfluxDB shell version: 1.8.10.

About your question, I’ve just checked one more time -just in case- the telegraf.conf and it’s 100% commented. All the configuration is migrated to the /telegraf.d/*.conf monitors scripts with the next structure (and the scripts are not bigs):

1st the [agent] and all its parameters (please, see my main description);
2nd the [[outputs.influxdb]] with one and only database;
3rd the [[inputs.snmp]], with the agents, the SNMP community, etc;
4th I add tags in [inputs.snmp.tags];
Finally the collectors scripts with the tables names, the OID, etc.

Thinking out loud, all the databases have in common the port 8086. Could be that a reason? Maybe I could investigate to open new ports for differents databases.

Giovanni_Luisotto · April 19, 2022, 2:06pm

The port is not the issue, as all the DBs are on the same InflxDB instance.

I think you are missing a filter…
From how I understand it, you have something like

[[outputs.influxdb]]
	urls = ["http://localhost:8086"]
	database = "COLLECTOR-1"
	{.....}

[[outputs.influxdb]]
	urls = ["http://localhost:8086"]
	database = "COLLECTOR-2"
	{.....}

[[outputs.influxdb]]
	urls = ["http://localhost:8086"]
	database = "COLLECTOR-3"
	{.....}

but there is no filtering in those outputs (database_tag is not a filter), meaning all the metrics are going through all the outputs…
In order to route/filter the data properly, you must have an attribute of some sort to distinguish the different sources.
How are you doing it? can you give some more detail about it?

German_Roldan · April 19, 2022, 2:40pm

Hi, Giovanni. Thank you for your interest in this issue.

I think I’m not doing any filter, only pointed to the database I want to storage the metrics. I’m sorry, I’m beginner with Telegraf and Influx, do you think this documentation could be helpful?

Configuring Telegraf | Telegraf 1.21 Documentation (influxdata.com)

Also, I think this issue is related but I don’t understand how to use namepass, if that’s the filter.

Filtered by telegraf according to 3 different influx databases data and sent according to database and data - Telegraf - InfluxData Community Forums

Giovanni_Luisotto · April 19, 2022, 2:57pm

yes, namepass is a filter, to see which and how to use filter I use the GitHub docs, but for this sections, it seems they’ve cloned it as-is therefore it should be fine.

About how to use it, it depends… namepass filters on measurement names, but if you gather the very same data from different tenants/servers/whatever the measurement name will be the same (see it as the table name), therefore I think tagpass is more common in this kind of scenario

To give you some more guidance I’d need to know your input and output configuration, and maybe some sample data, as:

the attributes you filter on must exist (in the data points)
if no such attribute exists you can add it via Telegraf (by adding static attributes in the configuration)
you can finally apply the filter

German_Roldan · April 19, 2022, 5:05pm

Thanks for your time, Giovanni. I will read the Github documentation at the night, when home.

I didn’t use namepass so I have to read before but I would like to answer that the databases have the same structure: same measurements, tags, names; the diferrences are the retention policies I’ve applied to each of them.

For example, this script.conf collect metrics to store in the database COLLECTOR-6, with a retention policy of 180 days.

[agent]
        interval = "10s"
        round_interval = true
        metric_batch_size = 1000
        metric_buffer_limit = 10000
        collection_jitter = "0s"
        flush_interval = "10s"
        flush_jitter = "0s"
        precision = ""
        quiet = true
        logtarget = "file"
        logfile = "/var/log/telegraf/telegraf.log"
        logfile_rotation_max_size = "500MB"
        hostname = ""
        omit_hostname = true

[[outputs.influxdb]]
        urls = ["http://localhost:8086"]
        database = "COLLECTOR-6"
        database_tag = "COLLECTOR-6"
        exclude_database_tag = false
#       skip_database_creation = true
#       retention_policy = "180-days-retention"
        timeout = "0s"
#       # HTTP Basic Auth
        username = "DATAIN6"
        password = "STRONGPASSWORD"

[[inputs.ping]]
   name_suffix = "_RPV"
   urls = ["192.168.30.251"]
   [inputs.ping.tags]
   name = "DIS RPV GW"
   geohash = "nnnnnn"
   dealer = "Captive"
   site = "Office"
   isp = "Claro"

[[inputs.ping]]
   name_suffix = "_RPV"
   urls = ["192.168.30.252"]
   [inputs.ping.tags]
   name = "RPV Main"
   geohash = "nnnnnn"
   dealer = "Captive"
   site = "Office"
   isp = "Claro"
   reference = "12345678"

In theory, the user DATAIN6 only has writing permissions over COLLECTOR-6 but Telegraf still writes in COLLECTOR-1 and -3. If I test the .conf I get this:

> ping_RPV,dealer=Captive,geohash=nnnnnn,isp=Claro,name=RPV\ Main,reference=12345678,site=DIS,url=192.168.30.252 average_response_ms=1.12,maximum_response_ms=1.12,minimum_response_ms=1.12,packets_received=1i,packets_transmitted=1i,percent_packet_loss=0,result_code=0i,standard_deviation_ms=0,ttl=252i 1650387595000000000
> ping_RPV,dealer=Captive,geohash=nnnnnn,isp=Claro,name=RPV\ GW,site=DIS,url=192.168.30.251 average_response_ms=1.394,maximum_response_ms=1.394,minimum_response_ms=1.394,packets_received=1i,packets_transmitted=1i,percent_packet_loss=0,result_code=0i,standard_deviation_ms=0,ttl=252i 1650387595000000000

I will reply at night. Thanks!

Giovanni_Luisotto · April 20, 2022, 9:53am

The configuration you posted will write only on the DB “COLLECTOR-6” as that’s the only output … if executed alone.
But since you are using a configuration folder, all the files will be concatenated in one big conf file, therefore all the inputs will write to all the outputs in this merged conf file, that’s why you need filtering (or routing).

In order to provide a filter/routing attribute I’d use static tags (that you are already using), which work like this:

##emit measurements with two additional tags: tag1=foo and tag2=bar
[[inputs.cpu]]
  percpu = false
  totalcpu = true
  [inputs.cpu.tags]
    tag1 = "foo"
    tag2 = "bar"

Since you are already using it, you will just need to add an additional key, ie:

[[inputs.ping]]
   name_suffix = "_RPV"
   urls = ["192.168.30.252"]
   [inputs.ping.tags]
     name = "RPV Main"
     geohash = "nnnnnn"
     dealer = "Captive"
     site = "Office"
     isp = "Claro"
     reference = "12345678"
     destinationDB = "COLLECTOR-1" ## routing tag

you can then use this to route the points, I’m not sure what suits you better, but I find routing simpler to manage and it will look like this:

[[outputs.influxdb]]
	urls = ["http://localhost:8086"]
	database = "COLLECTOR-1" ## This is the default DB name
	database_tag = "destinationDB " ##Tag to use for the DB name, if missing use default
	exclude_database_tag = true ##drop the routing tag

About the design itself, If all the DBs share the same data, except for the RPs… Why don’t have a single DB with all the RPs you need? (there might be lots of valid reasons, but I don’t know your use case).
You can also use routing for the retention policies (with retention_policy_tag), see the InfluxDB Output docs

German_Roldan · April 22, 2022, 3:43am

Hi, Giovanni. I read all your comments and I really appreciate the time you spent on them. Also I read the documentation but…

Sadly, and please don’t get mad, it seems I didn’t understand well because it’s not working but I will keep trying until it’s done.

For example, I don’t know if the tag should be in [inputs.snmp.tags] or [inputs.snmp.field.tags] or [inputs.snmp.table.tags] o could be in the [global_tags]. I was mixing the options but no lucky yet.

I’m still learning, maybe I could use a single database with 3 three retention policies. I will read about that. Please keep this post open until I find the answer to share with everyone who needs.

I’ll be back tomorrow at night. Thank you and best regards.

Giovanni_Luisotto · April 22, 2022, 8:38am

no worries.

About adding static tags, it works like this, the .tag “node” is one of the many generic nodes available for input plugins, meaning it can be added to any input.

from a syntax point of view it’s added as a child node of the input plugin like this:

[[whatever.input]]
  {...}
  [whatever.input.tags]
    MyTagKey = 'MyTagValue'

## Some real examples
[[inputs.cpu]]
  {...}
  [inputs.cpu.tags]
    tag1 = "foo"

[[inputs.snmp]]
  {...}
  [inputs.snmp.tags]
    tag1 = "foo"

If you use global_tags, the key-value pair will be added to all the gathered metrics, but since you want different values for different inputs it won’t be useful for routing…

Topic		Replies	Views
401 Unauthorised with Telegraf and InfluxDB - database creation failed - even with All Access API tokens? Telegraf influxdb , telegraf	2	10043	March 28, 2022
The telegraf Unauthorized to InfluxDB InfluxDB 2 telegraf	3	27338	December 30, 2023
Telegraf writes to two InfluxDB databases same data Telegraf telegraf	8	257	October 30, 2023
Run data retention query via telegraf influxdb , telegraf , query	3	573	February 2, 2021
Influxdb 2.1, influxql, problem to access database InfluxDB 2	3	681	December 28, 2021

Telegraf writing in multi databases with no permissions

Related topics