Specify Non-default retention policy in line protocol for telegraf

Hello all,

Currently, for speed reasons, I writing to a telegraf unix socket to get data into influx. I am writing to different measurements in a database that will ideally have different retention policies. I have found on the influx FAQ how to specify non-default retention policies through CLI:

INSERT INTO [<database>.]<retention_policy> <line_protocol>

However, lines written to the socket ONLY contain <line_protocol> portion of the above string. So, while I know it is possible to write to this socket and specify different measurements (as the measurement info is in <line_protocol>), is it possible to also specify different retention policies?

Hello @generic_coder,
I think I might be missing something obvious about your question…
but yes you can specify different retention policies in the <retention_policy> portion of the CLI command.
https://docs.influxdata.com/influxdb/v1.8/troubleshooting/frequently-asked-questions/#how-do-i-write-to-a-non-default-retention-policy-with-the-influxdb-cli

Hello @Anaisdg, thank you for your response. Yes I see that this can be done through the CLI when inserting points into influx, however, I am inserting points to influx through telegraf unix socket. Because the database is specified through a configuration file for telegraf and I only write line protocol to this unix socket, only the bold portion below is ever written to the file:

INSERT INTO [.]<retention_policy> <line_protocol>

Thus, I am not sure how to specify retention policy when writing to this telegraf unix socket.

Hello @generic_coder ,
Unfortunately you can’t assign retention policies to data with telegraf. You have to use the CLI to do that. But this is the path that users take. They write line protocol with telegraf and then apply different retention policies with specific measurements for example.

Can you provide some more context around what you’re trying to achieve/ultimate goal? There might be another way to help.

The only reason I am using telegraf is because of the unix socket. This appears to be the only way to use a unix socket to insert data into influx (that is, via telegraf). I am doing this because the unix socket seems to be the fastest way to insert data in influx. I am writing to various measurements, say A and B, within database X. X is configured in the telegraf configuration file. A and B are specified by the line protocol I write to the telegraf unix socket file. However, I would like A to use the default retention policy of X (happens to be one day), and B to use a retention policy of 7 days (non-default). I am not sure if this can be done through the telegraf unix socket file, but I guess what you’re saying is that it’s not.

Hello @generic_coder,
Here is documentation on the metric filtering that is available to you within telegraf.

I don’t think that’s possible but I’ll check with the telegraf team. Thanks for your patience.

Hi @generic_coder, I took a quick look in the telegraf source code and I found that telegraf can specify a retention policy only when it writes to influxdb over http. This is because the retention policy is provided as part of the query string in the url.

(the query string param is called ‘rp’)
https://docs.influxdata.com/influxdb/v1.8/tools/api/#query-string-parameters-2

(here’s the telegraf code that adds ‘rp’ to the url if it was set in telegraf.conf)

When connecting over a unix socket, telegraf doesn’t tell influxdb the retention policy. I’m not sure if there is a way to provide a retention policy to influxdb when using a unix socket, but I’m more familiar with telegraf than influxdb and I might have missed it.

That said, I’d be surprised if writing to influxdb over unix socket performs much better than over http connecting to localhost. If your performance requirements allow you to use http to localhost, you may be able to do what you’re asking for. You would need to configure telegraf to have two influxdb outputs, both writing to the same influxdb instance, but one with the default retention policy and another with the special (7 day) policy. Then using metric filtering (linked in Anais’s previous comment) you would drop the 7 day metrics from the default output and pass only the 7 day metrics to the 7 day output.

You can pass or drop metrics based on the metric name or field or tag name or even a tag value. It sounds like your metrics are named A and B so you would filter by metric name using namepass and namedrop.

What I have described so far is the connection from telegraf to influxdb. You could still use a unix socket to send from your program to telegraf.

1 Like