Migration task failure from Influx Cloud to OSS v2

InfluxDB 2
, ,
1

error calling function “findRecord” count: aggregate found duplicate table with key

I’m attempting to migrate to 2.7.5 OSS influx from cloud. While investigating a migration task failure I attempted a raw export:

from(bucket: "xyz_updates")
  |> range(start: -3h, stop: now())

And thought this would produce a csv with headers written only once, as it did for another bucket in that Influx Cloud instance. But the following headers are repeated for each data line:

#group,false,false,true,true,....true,true,true,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,unsignedLong,string,string,...
#default,_result,,,,,,,,,,,,,,,,,,,,,
,result,table,_start,_stop,_time,_value,_field,_measurement,tags...

Background: Tried setting up a migration task for this bucket based on Migrate data from InfluxDB Cloud to InfluxDB OSS | InfluxDB OSS v2 Documentation (since there isn’t a migration guide for cloud v3 to OSS (v2) so I’m just following above), and got the following task error logs:

2024-02-16 12:20:57
error exhausting result iterator: error calling function "metadata" @99:1-99:11: error calling function "findRecord" @69:32-69:69: runtime error @61:12-61:19: count: aggregate found duplicate table with key: {_field=thatField,_measurement=thatMeasurement,_start=2024-02-13T00:00:00.000000000Z,_stop=2024-02-13T01:00:00.000000000Z,...}

I’m not sure how to go about solving for this situation. I’ve read about schema restrictions that point towards “Do not use duplicate names for tags and fields” but in that case the write itself fails. In my case, there’s a “duplicate table with some key”. How is that possible within a measurement if fields and tags are represented as columns and I have a unique measurement name? Would be great if you could point out a root cause for me to investigate here or just clarify my understanding of this situation!

PS: Hi @pauldix ! If this error can be dealt with via an OSS v3 release (which I hear is just around the corner), I can see that you’ve already replied to InfluxDB 3.0 release timeline - #8 by austinredacted But could we do something about this error or do we just have to wait? Thanks!

1 Like
2

@Arco_Redacted Can you provide the full migration task you tried to run? Technically, it should work, however, I’m wondering if the /api/v2/query endpoint’s behavior in InfluxDB Cloud Serverless is different than InfluxDB v2.

3

@scott Hey I literally used the one at Migrate data from InfluxDB Cloud to InfluxDB OSS | InfluxDB OSS v2 Documentation with just migration config changes for my setup - bucket name, times…

4

@scott Hey just following up, any clues ^?

5

I’m not 100% sure what’s happening. You may try resetting the migration and running it again. To reset the migration, delete and recreate the migration bucket in your local instance. This is where the task stores and reads migration metadata.

Once the new migration bucket is created, try starting the task again.

6

I just ran this again on a separate OSS instance from scratch and I still got

error exhausting result iterator: error calling function "metadata" @99:1-99:11: error calling function "findRecord" @69:32-69:69: runtime error @61:12-61:19: count: aggregate found duplicate table with key:...

same as in my description. Think it’s just the v3 cloud serverless ↔ v2 OSS compatibility and I guess we really need the OSS to have a v3 compatible release for this to work?

7

Hello @Arco_Redacted
Yah I wouldn’t have hypothesized that that migration guide would work for v3 cloud because as scott mentioned the query endpoint is different in v3. I think the best way to migrate data would be to use something like quix:

Those guides are for migrating from v2 to v3. However I you could easily build your own stream and query from v3 and write to v2.

Otherwise if you want to perform a bulk migration, you might be able to just do that with the python v3 client library.