Tag edits have been a tricky topic for years with InfluxDB. We need a way to reprocess a sizable number of tags in an InfluxCloud 2 bucket.
A tool like Infix would probably be able to help us if our datastore was local. Of course, we don’t have that access for InfluxCloud 2.
Ultimately, we intend to build a data pipeline that collects our raw writes in parallel to InfluxCloud writes so we can later reprocess / replay them to rebuild a bucket. Alas, this doesn’t exist yet.
So, our question is… Given available tools, what would be our easiest option to read all records from a given bucket, process tag value rewrite rules, and then write those corrected records to a new bucket?
Is this perhaps a Python script that queries batches of records in incrementing time chunks that then executes the tag value edits and writes the records back to a new bucket?
Is this something we could do with a Flux-based InfluxDB Task?
Is there an existing tool good for this sort of data workflow that can be configured to work with InfluxDB?
What gotchas and angles do we need to be thinking about that we may be missing? We’re certainly aware that we’ll need to specially handle the latest records from ongoing writes.
Thank you for any ideas.
