Failed Docker Migration from Influx 1.8 to 2.7 without information

InfluxDB 2
,
1

Hi everybody!
I’m trying to upgrade my influx image from 1.8 to 2.7 using the Docker upgrade option, but process stops everytime.
I don’t see any resource saturation like CPU or RAM and the only information that I see are the following logs:

{"level":"info","ts":1708447564.151024,"caller":"upgrade/upgrade.go:401","msg":"Starting InfluxDB 1.x upgrade"}

{"level":"info","ts":1708447564.1510768,"caller":"upgrade/upgrade.go:404","msg":"Upgrading config file","file":"/etc/influxdb/influxdb.conf"}

{"level":"info","ts":1708447564.151148,"caller":"upgrade/upgrade.go:408","msg":"Config file upgraded.","1.x config":"/etc/influxdb/influxdb.conf","2.x config":"/etc/influxdb2/config.toml"}

{"level":"info","ts":1708447564.1511612,"caller":"upgrade/upgrade.go:418","msg":"Upgrade source paths","meta":"/var/lib/influxdb/volume/meta","data":"/var/lib/influxdb/volume/data"}

{"level":"info","ts":1708447564.151177,"caller":"upgrade/upgrade.go:419","msg":"Upgrade target paths","bolt":"/var/lib/influxdb2/influxd.bolt","engine":"/var/lib/influxdb2/engine"}

{"level":"info","ts":1708447564.1599119,"caller":"bolt/bbolt.go:83","msg":"Resources opened","service":"bolt","path":"/var/lib/influxdb2/influxd.bolt"}

{"level":"info","ts":1708447564.1613262,"caller":"migration/migration.go:175","msg":"Bringing up metadata migrations","service":"migrations","migration_count":20}

{"level":"info","ts":1708447564.396699,"caller":"upgrade/setup.go:73","msg":"CLI config has been stored.","path":"/etc/influxdb2/influx-configs"}

{"level":"info","ts":1708447564.3967407,"caller":"upgrade/database.go:202","msg":"Checking available disk space"}

{"level":"info","ts":1708447564.397583,"caller":"upgrade/database.go:223","msg":"Computed disk space","free":"63 GB","required":"45 GB"}

{"level":"info","ts":1708447564.3976183,"caller":"upgrade/database.go:51","msg":"Upgrading databases"}

{"level":"info","ts":1708447564.4023683,"caller":"upgrade/database.go:101","msg":"Creating mapping","database":"default","retention policy":"defaul","orgID":"example","bucketID":"example"}

2024-02-20T16:47:33. warn cleaning bolt and engine files to prevent conflicts on retry {"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt", "engine_path": "/var/lib/influxdb2/engine"}

It seems that the issue is related to the volume size since the same procedure on a database of 10GB was successful.
Any ideas?

2

I have performed the migration on debug mode with INFLUXD_LOG_LEVEL environment variable, but I don’t see any critical log.

{"level":"info","ts":1708452939.8188407,"caller":"upgrade/upgrade.go:401","msg":"Starting InfluxDB 1.x upgrade"}
{"level":"info","ts":1708452939.818899,"caller":"upgrade/upgrade.go:404","msg":"Upgrading config file","file":"/etc/influxdb/influxdb.conf"}
{"level":"info","ts":1708452939.8189814,"caller":"upgrade/upgrade.go:408","msg":"Config file upgraded.","1.x config":"/etc/influxdb/influxdb.conf","2.x config":"/etc/influxdb2/config.toml"}
{"level":"info","ts":1708452939.8189976,"caller":"upgrade/upgrade.go:418","msg":"Upgrade source paths","meta":"/var/lib/influxdb/volume/meta","data":"/var/lib/influxdb/volume/data"}
{"level":"info","ts":1708452939.819011,"caller":"upgrade/upgrade.go:419","msg":"Upgrade target paths","bolt":"/var/lib/influxdb2/influxd.bolt","engine":"/var/lib/influxdb2/engine"}
{"level":"info","ts":1708452939.8291152,"caller":"bolt/bbolt.go:83","msg":"Resources opened","service":"bolt","path":"/var/lib/influxdb2/influxd.bolt"}
{"level":"info","ts":1708452939.8304136,"caller":"migration/migration.go:175","msg":"Bringing up metadata migrations","service":"migrations","migration_count":20}
{"level":"debug","ts":1708452939.8304374,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"initial migration","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.8861432,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"initial migration","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.8861682,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"userresourcemappingsbyuserindexv1\"","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.8897743,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"userresourcemappingsbyuserindexv1\"","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.8897972,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"migrate task owner id","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.8920665,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"migrate task owner id","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.8920867,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create DBRP buckets","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.8986511,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create DBRP buckets","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.8986921,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create pkger stacks buckets","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9080582,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create pkger stacks buckets","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9080744,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"delete sessionsv1 bucket","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.914051,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"delete sessionsv1 bucket","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9140654,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create TSM metadata buckets","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.918674,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create TSM metadata buckets","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.918691,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create Legacy authorization buckets","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9245334,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create Legacy authorization buckets","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9245477,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create legacy auth password bucket","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9286199,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"Create legacy auth password bucket","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.928639,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"telegrafbyorgindexv1\"","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.933135,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"telegrafbyorgindexv1\"","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.933152,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"populate dashboards owner id","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9365034,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"populate dashboards owner id","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9365447,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"dbrpbyorgv1\"","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9414568,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"dbrpbyorgv1\"","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.941475,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"repair DBRP owner and bucket IDs","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9443226,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"repair DBRP owner and bucket IDs","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9443405,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"dbrpbyorgv1\"","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.948634,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add index \"dbrpbyorgv1\"","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.948658,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"record shard group durations in bucket metadata","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9511344,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"record shard group durations in bucket metadata","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9511576,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add annotations and notebooks resource types to operator token","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9548705,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add annotations and notebooks resource types to operator token","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.954886,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add annotations and notebooks resource types to all-access tokens","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.958519,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add annotations and notebooks resource types to all-access tokens","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.958532,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"repair missing shard group durations","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.961096,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"repair missing shard group durations","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9611104,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add remotes and replications resource types to operator and all-access tokens","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9656513,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"add remotes and replications resource types to operator and all-access tokens","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.965666,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create remotes and replications metrics buckets","target_state":"up","migration_event":"started"}
{"level":"debug","ts":1708452939.9713132,"caller":"migration/migration.go:294","msg":"Executing metadata migration","service":"migrations","migration_name":"create remotes and replications metrics buckets","target_state":"up","migration_event":"completed"}
{"level":"debug","ts":1708452939.9777346,"caller":"tenant/middleware_user_logging.go:36","msg":"user create","store":"new","took":0.001771084}
{"level":"debug","ts":1708452940.0416768,"caller":"tenant/middleware_user_logging.go:137","msg":"set password","store":"new","took":0.063904592}
{"level":"debug","ts":1708452940.0433817,"caller":"tenant/middleware_org_logging.go:48","msg":"org find by ID","store":"new","took":0.000021206}
{"level":"debug","ts":1708452940.044672,"caller":"tenant/middleware_bucket_logging.go:35","msg":"bucket create","store":"new","took":0.00131354}
{"level":"debug","ts":1708452940.0461304,"caller":"tenant/middleware_org_logging.go:48","msg":"org find by ID","store":"new","took":0.000024085}
{"level":"debug","ts":1708452940.0477734,"caller":"tenant/middleware_bucket_logging.go:35","msg":"bucket create","store":"new","took":0.001668193}
{"level":"debug","ts":1708452940.050733,"caller":"tenant/middleware_urm_logging.go:35","msg":"urm create","store":"new","took":0.001653261}
{"level":"debug","ts":1708452940.050761,"caller":"tenant/middleware_org_logging.go:35","msg":"org create","store":"new","took":0.009048714}
{"level":"debug","ts":1708452940.052243,"caller":"tenant/middleware_urm_logging.go:35","msg":"urm create","store":"new","took":0.001474157}
{"level":"debug","ts":1708452940.0522773,"caller":"tenant/middleware_org_logging.go:48","msg":"org find by ID","store":"new","took":0.000019279}
{"level":"debug","ts":1708452940.0536025,"caller":"tenant/middleware_bucket_logging.go:35","msg":"bucket create","store":"new","took":0.001345289}
{"level":"debug","ts":1708452940.055023,"caller":"tenant/middleware_user_logging.go:49","msg":"user find by ID","store":"new","took":0.000017114}
{"level":"debug","ts":1708452940.0550513,"caller":"tenant/middleware_org_logging.go:48","msg":"org find by ID","store":"new","took":0.000012126}
{"level":"info","ts":1708452940.0566823,"caller":"upgrade/setup.go:73","msg":"CLI config has been stored.","path":"/etc/influxdb2/influx-configs"}
{"level":"info","ts":1708452940.056709,"caller":"upgrade/database.go:202","msg":"Checking available disk space"}
{"level":"info","ts":1708452940.1959965,"caller":"upgrade/database.go:223","msg":"Computed disk space","free":"84 GB","required":"61 GB"}
{"level":"info","ts":1708452940.196074,"caller":"upgrade/database.go:51","msg":"Upgrading databases"}
{"level":"debug","ts":1708452940.196092,"caller":"upgrade/database.go:61","msg":"Upgrading database","database":"default"}
{"level":"debug","ts":1708452940.1961148,"caller":"upgrade/database.go:78","msg":"Creating bucket","Bucket":"default/default"}
{"level":"debug","ts":1708452940.1961937,"caller":"tenant/middleware_org_logging.go:48","msg":"org find by ID","store":"new","took":0.000059863}
{"level":"debug","ts":1708452940.1992478,"caller":"tenant/middleware_bucket_logging.go:35","msg":"bucket create","store":"new","took":0.003116858}
{"level":"debug","ts":1708452940.1992679,"caller":"upgrade/database.go:86","msg":"Creating database with retention policy","database":"0ad7fe2569a13c50"}
{"level":"info","ts":1708452940.201298,"caller":"upgrade/database.go:101","msg":"Creating mapping","database":"default","retention policy":"default","orgID":"6014d19dad0048ea","bucketID":"0ad7fe2569a13c50"}
{"level":"debug","ts":1708452940.201385,"caller":"tenant/middleware_bucket_logging.go:48","msg":"bucket find by ID","store":"new","took":0.000057113}
{"level":"debug","ts":1708452940.2014256,"caller":"tenant/middleware_bucket_logging.go:45","msg":"failed to find bucket with ID 0c9dee1b6a6ff000","store":"new","error":"bucket not found","took":0.000010195}
{"level":"debug","ts":1708452940.2030492,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1698624000}
{"level":"debug","ts":1708452940.204596,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1699228800}
{"level":"debug","ts":1708452940.205893,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1699833600}
{"level":"debug","ts":1708452940.2072716,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1700438400}
{"level":"debug","ts":1708452940.2088203,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1701043200}
{"level":"debug","ts":1708452940.2102702,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1701648000}
{"level":"debug","ts":1708452940.2117949,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1702252800}
{"level":"debug","ts":1708452940.2131066,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1702857600}
{"level":"debug","ts":1708452940.2146516,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1703462400}
{"level":"debug","ts":1708452940.2162488,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1704067200}
{"level":"debug","ts":1708452940.2176363,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1704672000}
{"level":"debug","ts":1708452940.218943,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1705276800}
{"level":"debug","ts":1708452940.2205255,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1705881600}
{"level":"debug","ts":1708452940.2219942,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1706486400}
{"level":"debug","ts":1708452940.2236526,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1707091200}
{"level":"debug","ts":1708452940.2249715,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1707696000}
{"level":"debug","ts":1708452940.2265859,"caller":"upgrade/database.go:114","msg":"Creating shard group","database":"0ad7fe2569a13c50","retention policy":"autogen","time":1708300800}
{"level":"debug","ts":1708452940.2277763,"caller":"upgrade/database.go:129","msg":"Copying data","source":"/var/lib/influxdb/volume/data/default/default","target":"/var/lib/influxdb2/engine/data/0ad7fe2569a13c50/autogen"}
2024-02-20T18:16:59.	warn	cleaning bolt and engine files to prevent conflicts on retry	{"system": "docker", "bolt_path": "/var/lib/influxdb2/influxd.bolt", "engine_path": "/var/lib/influxdb2/engine"}
3

I didn’t mention it, but I was running the docker image in Kubernetes.
It seems that the root cause was the Liveness Probe timeout, it was too low interrupting the upgrade

1 Like
4

@Aurelio_Marin_Aranza thanks so much for sharing your solution with the community. I’m sure they’ll appreciate it. Out of curiosity what are you doing with InfluxDB?