Line Protocol Compression

Store
1

Is There compression supported for HTTP API Writing (Line Protocol)?
I didn’t find any documentation for it, although there appear to be some Pull Requests for InfluxDB on github.

If it is supported, what compression protocols are supported and what headers are required?
(It is currently bottleneck, when uploading data as line protocol to the cloud, especially backfilling the same series with multiple tags.)

Thank you very much.

2

gzip compression is supported. If you are using Telegraf, you can enable gzip via the config file:

  ## HTTP Content-Encoding for write request body, can be set to "gzip" to
  ## compress body or "identity" to apply no encoding.
  content_encoding = "gzip"

If you are using a client library, you will have to check the documentation for that library to find out how to enable gzip.

3

Thanks, Noah.

Yeah I figured that GoLang’s HTTP Library natively supports GZip compression
(using example at support gzip for http request to improve performance by jiafu1115 · Pull Request #241 · influxdata/influxdb-java · GitHub)

I think it would still be worth mentioning in the documentation,
as not many people are familiar with Go HTTP features,
and other wise there is an impression that InfluxDB doesn’t support compression.

.NET Standard Library supports compression as well and this works amazingly!
If somebody is interested, here is code snipet:

using (var request = new HttpRequestMessage(HttpMethod.Post, _uploadAddress))
{
	request.Headers.Add("Keep-Alive", "true");

	byte[] bytes = Encoding.UTF8.GetBytes(sb.ToString());
	using (var memoryStream = new MemoryStream())
	{
		// Compressing bytes into MemoryStream's buffer.
		using (var gZipStream = new GZipStream(memoryStream, _config.CompressionLevel, true))
		{
			gZipStream.Write(bytes, 0, bytes.Length);
		}
		memoryStream.Position = 0;

		using (request.Content = new StreamContent(memoryStream))
		{
			request.Content.Headers.Add("Content-Encoding", "gzip");
			request.Content.Headers.ContentLength = memoryStream.Length;
			using (HttpResponseMessage response = await _client.SendAsync(request, _token))
			{
				await ThroughContentsOnError(response);
			}
		}
	}
}
4

Our docs are open-source! Would you mind creating an issue on GitHub?

Glad the compression is working. If you’re writing all of your points using the .Net Standard Library, though, we recommend batching up points in order to ease HTTP/S overhead on InfluxDB. Telegraf takes care of that for you, as should the client libraries.

Have you tried any of the .NET libraries for InfluxDB?