Http.post the whole table

Fluxlang
,
1

Hi there,

I’m looking at this link: Convert results to JSON and send them to a URL | InfluxDB OSS 2.0 Documentation, where a code excerpt is as follow:

import "http"
import "json"

from(bucket: "noaa")
  |> filter(fn: (r) => r._measurement == "average_temperature")
  |> mean()
  |> map(fn: (r) => ({ r with
    jsonStr: string(v: json.encode(v: {"location":r.location,"mean":r._value}))}))
  |> map(fn: (r) => ({r with
    status_code: http.post(
      url: "http://somehost.com/",
      headers: {x:"a", y:"b"},
      data: bytes(v: r.jsonStr)
    )
  }))

This sends a http post for each of the records in the table.

My question: is there a way to send the whole table in 1 http.post request?

Thanks

2

Usually, that’s an approach that doesn’t scale very well (not just in influxdb, but many many programming languages) . At some point, it’s just too much data to handle, either on the sender end, the transport in the middle, the receiver end, or the receiver application behind the endpoint.

Is there a specific problem you’re seeing with the way it’s working currently? What’s the problem you need to solve?

3

Hi there,

For me I’d think sending a HTTP request for every rows would be less scalable.

Do you prefer 100 request, 1kb each or 1 request of 100kb?

4

Yes that’s the idea, batching up the requests to a reasonable size. That’s where you’ll need to test for your use case.

Influxdb docs recommend a batch size of around 5000 for writes, so a useful starting point . Your use case may perform better slightly above this number or below.

5

This isn’t relevant, I need to “batch” out with that http packages, can you point me to the doc that helps doing this?

6

We have the same needs here.

We have Flux queries that return a small (10-100) number of rows and we would like to send them as one HTTP POST request.

A working solution would be greatly appreciated.

7

Currently, findColumn() can extract an array of values.

It would be nice to have findColumns() to extract and array of records to be given to json.encode() (for instance).

8

This is very late but if anyone else ends up here the same way I did, I found something that works roughly in what I think OP was looking for:

import "http"
import "json"
import "strings"

from(bucket: "noaa")
  |> filter(fn: (r) => r._measurement == "average_temperature")
  |> mean()
  |> map(fn: (r) => ({ r with
    jsonStr: string(v: json.encode(v: {"location":r.location,"mean":r._value}))}))
  |> group()
  |> reduce(
        fn: (r, accumulator) =>
            ({sum: strings.joinStr(arr: [accumulator["sum"], r["jsonStr"]], v: ",")}),
        identity: {sum: ""},
    )
  |> map(
        fn: (r) =>
            ({
                jsonStr:
                    strings.joinStr(
                        v: "",
                        arr: ["{table:[", strings.trimLeft(v: r["sum"], cutset: ","), "]}"],
                    ),
            }),
    )
  |> map(fn: (r) => ({r with
    status_code: http.post(
      url: "http://somehost.com/",
      headers: {x:"a", y:"b"},
      data: bytes(v: r["jsonStr"])
    )
  }))

Hopefully the code translated well back to the generic example. Sorry if there’s a typo in there but you can figure out what I did. I reduced all the individual json strings and wrapped it all in an array named table.