Transform data from one bucket to another

,
1

I have a large dataset that I want to transform and save. But it ends up using all the memory and crashes. Is there an optimal way to do something like this in flux ?

from(bucket: "bucket01")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "measurement01")
  |> filter(fn: (r) => r["_field"] == "field01")
  |> map(fn: (r) => ({ r with _value: r._value*1.89}))
  |> to(bucket: "bucket02", org: "org01")
2

Hello @tintin,
You could split it up into smaller time chunks and do that.
How large is your dataset?

3

Two months of 1Hz data on AWS r5.xlarge instance, being transferred in 1 day chunks. I ended up exporting it to files/line protocol, replaced bucket name and importing it.

1 Like
4

Hi tintin,
How did you import the line protocol file?
I tried using the influx write command but it was rejected on the following error: “Error: failed to write data: 401 Unauthorized: unauthorized access”.
Any suggestions?