Specify Non-default retention policy in line protocol for telegraf

generic_coder · January 18, 2021, 5:18pm

Hello all,

Currently, for speed reasons, I writing to a telegraf unix socket to get data into influx. I am writing to different measurements in a database that will ideally have different retention policies. I have found on the influx FAQ how to specify non-default retention policies through CLI:

INSERT INTO [<database>.]<retention_policy> <line_protocol>

However, lines written to the socket ONLY contain <line_protocol> portion of the above string. So, while I know it is possible to write to this socket and specify different measurements (as the measurement info is in <line_protocol>), is it possible to also specify different retention policies?

Anaisdg · January 20, 2021, 6:05pm

Hello @generic_coder,
I think I might be missing something obvious about your question…
but yes you can specify different retention policies in the <retention_policy> portion of the CLI command.

generic_coder · January 20, 2021, 7:12pm

Hello @Anaisdg, thank you for your response. Yes I see that this can be done through the CLI when inserting points into influx, however, I am inserting points to influx through telegraf unix socket. Because the database is specified through a configuration file for telegraf and I only write line protocol to this unix socket, only the bold portion below is ever written to the file:

INSERT INTO [.]<retention_policy> <line_protocol>

Thus, I am not sure how to specify retention policy when writing to this telegraf unix socket.

Anaisdg · January 20, 2021, 7:26pm

Hello @generic_coder ,
Unfortunately you can’t assign retention policies to data with telegraf. You have to use the CLI to do that. But this is the path that users take. They write line protocol with telegraf and then apply different retention policies with specific measurements for example.

Can you provide some more context around what you’re trying to achieve/ultimate goal? There might be another way to help.

generic_coder · January 20, 2021, 7:31pm

The only reason I am using telegraf is because of the unix socket. This appears to be the only way to use a unix socket to insert data into influx (that is, via telegraf). I am doing this because the unix socket seems to be the fastest way to insert data in influx. I am writing to various measurements, say A and B, within database X. X is configured in the telegraf configuration file. A and B are specified by the line protocol I write to the telegraf unix socket file. However, I would like A to use the default retention policy of X (happens to be one day), and B to use a retention policy of 7 days (non-default). I am not sure if this can be done through the telegraf unix socket file, but I guess what you’re saying is that it’s not.

Anaisdg · January 21, 2021, 7:26pm

Hello @generic_coder,
Here is documentation on the metric filtering that is available to you within telegraf.

github.com

influxdata/telegraf/blob/master/docs/CONFIGURATION.md#metric-filtering

<!-- markdownlint-disable MD024 -->

# Configuration

Telegraf's configuration file is written using [TOML][] and is composed of
three sections: [global tags][], [agent][] settings, and [plugins][].

View the default [telegraf.conf][] config file with all available plugins.

## Generating a Configuration File

A default config file can be generated by telegraf:

```sh
telegraf config > telegraf.conf
```

To generate a file with specific inputs and outputs, you can use the
--input-filter and --output-filter flags:

This file has been truncated. show original

I don’t think that’s possible but I’ll check with the telegraf team. Thanks for your patience.

reimda · January 25, 2021, 3:57pm

Hi @generic_coder, I took a quick look in the telegraf source code and I found that telegraf can specify a retention policy only when it writes to influxdb over http. This is because the retention policy is provided as part of the query string in the url.

(the query string param is called ‘rp’)

(here’s the telegraf code that adds ‘rp’ to the url if it was set in telegraf.conf)

github.com

influxdata/telegraf/blob/master/plugins/outputs/influxdb/http.go#L474


      
          		return nil, fmt.Errorf("failed creating new request: %s", err.Error())
          	}
          
          
	req.Header.Set("Content-Type", "text/plain; charset=utf-8")
          	c.addHeaders(req)
          
          
	if c.config.ContentEncoding == "gzip" {
          		req.Header.Set("Content-Encoding", "gzip")
          	}
          
          
	return req, nil
          }
          
          
// requestBodyReader warp io.Reader from influx.NewReader to io.ReadCloser, which is usefully to fast close the write
          // side of the connection in case of error
          func (c *httpClient) requestBodyReader(metrics []telegraf.Metric) (io.ReadCloser, error) {
          	reader := influx.NewReader(metrics, c.config.Serializer)
          
          
	if c.config.ContentEncoding == "gzip" {
          		rc, err := internal.CompressWithGzip(reader)
          		if err != nil {

When connecting over a unix socket, telegraf doesn’t tell influxdb the retention policy. I’m not sure if there is a way to provide a retention policy to influxdb when using a unix socket, but I’m more familiar with telegraf than influxdb and I might have missed it.

That said, I’d be surprised if writing to influxdb over unix socket performs much better than over http connecting to localhost. If your performance requirements allow you to use http to localhost, you may be able to do what you’re asking for. You would need to configure telegraf to have two influxdb outputs, both writing to the same influxdb instance, but one with the default retention policy and another with the special (7 day) policy. Then using metric filtering (linked in Anais’s previous comment) you would drop the 7 day metrics from the default output and pass only the 7 day metrics to the 7 day output.

You can pass or drop metrics based on the metric name or field or tag name or even a tag value. It sounds like your metrics are named A and B so you would filter by metric name using namepass and namedrop.

What I have described so far is the connection from telegraf to influxdb. You could still use a unix socket to send from your program to telegraf.

Topic		Replies	Views
How to override retention policy in telegraf.conf? Store	5	13380	March 14, 2018
Ingesting data to different retention policies by tag	1	1116	March 23, 2019
Run data retention query via telegraf influxdb , telegraf , query	3	505	February 2, 2021
Telegraf writing in multi databases with no permissions Telegraf influxdb , telegraf	9	1016	April 22, 2022
What is deference between default and autopen on telegraf	2	2239	March 27, 2017

Specify Non-default retention policy in line protocol for telegraf

Related Topics