Upgrade InfluxDB from 1.8.x to 2.0

InfluxDB 2
1

Hi there,
since InfluxDB is production ready, I want to upgrade my InfluxDB from 1.8 to 2.0. The official documentation consists exactly this subject and provides the command ‘influxd upgrade’. But there is no upgrade command! There es only: backup, config, help, restore, run, version.
Where is the failure?
Thanks, Adam

1 Like
2

@adam, you have to install InfluxDB 2 first. upgrade command is supported in influxd from version 2 RC 1.

3

Thanks!
Yes that make sense, I saw Version 2 has other commands … and upgrade.

4

As a follow-up to this (since this title seems generic to the 1.8.x to 2.0 upgrades),

I followed the instructions in Upgrade from InfluxDB 1.x to 2.0 | InfluxDB OSS 2.0 Documentation and used influxd upgrade against my /etc/influxdb/influxdb.conf file. The upgrade went fine, and the logs that printed out showed that my existing databases were converted into buckets: "msg":"Upgrading database ","database":"telegraf"} and then "msg":"Creating bucket ","Bucket":"telegraf/autogen"}. The documentation noted that the 1.x retention policy would be tacked on to the bucket name, which is why the new bucket is named telegraf/autogen since I used the default retention policy in 1.x.

However, I’ve been unable to find the data now that it’s been imported into 2.0. My 1.x /var/lib/influxdb/ wal, meta, and data folders summed to about 4GB, and the 2.0 data storage locations ~/.influxdbv2/meta and ~/.influxdbv2/engine are around the same size, so I know the data was imported.

When logging into the Influx 2.0 UI on port 8086 though, I’m prompted to create a new user/org, and that user doesn’t seem to be able to see the migrated buckets. Furthermore, the user and org I created during influxd upgrade also don’t seem to work for logging into the Web UI.

What’s going on here? Where can I find the data that I’ve migrated from Influx 1.8.3 to Influx 2.0.1?

5

I figured out the issue here, there was remnant data from a previous attempt at installing/upgrading Influx. I also found that running sudo -u influxdb influxd upgrade to run the upgrade as the user that is created for this service can also help set up permissions correctly.

6

I am struggling with this same thing. I am running the ‘influxd upgrade’ command and I am able to see the log complete the migration from the databases to buckets. I can not find any errors from the upgrade that stand out in the logs. Everything appears to migrate correctly. However, when I login to the influxdb portal (:8086) I am prompted with the same create user, pass, org, etc… I do not see my new buckets.

I have tried multiple times to do the upgrade. I am not sure what I am missing for the step to see it completed :confused:

V1.7.9 to V2.0.3

  1. created user account for read that I tested before the upgrade for my dashboards with auth
  2. backed up databases
  3. downloaded influx V2.0.3
  4. removed influx V1.7.9
  5. Installed influx V2.0.3
  6. Verified version
  7. influxd upgrade
  8. read log for migration
  9. service influxd start
  10. Go to portal and create admin user
  11. skip to bucket list, nothing except bucket created in admin prompt.

Any advice would be greatly appreciated.

7

i followed the exact same steps and had the same results.
i went poking around with influx a bit and created a user with read/write access to the bucket based on the logs:
{“level”:“info”,“ts”:1613602189.1908314,“caller”:“upgrade/database.go:99”,“msg”:"Creating bucket ",“Bucket”:“telegraf/autogen”}
{“level”:“info”,“ts”:1613602189.1964552,“caller”:“upgrade/database.go:128”,“msg”:“Creating mapping”,“database”:“telegraf”,“retention policy”:“autogen”,“orgID”:“b19beece76e9a501”,“bucketID”:“d76e92f0b49ba03b”}

root@db:~# influx v1 auth create --username userv1 --password password --read-bucket d76e92f0b49ba03b --write-bucket d76e92f0b49ba03b

but no dice. i can’t log in with this user and still not seeing anything with the admin user in the UI / cli with bucket ls

8

@ajstimer, the admin user is created during the upgrade process. It looks as the server points to another data location than the upgrade. Check influxd logs, please, and make sure it is reading data files from the same location where upgrade created data files.

9

@VlastaHajek Thanks for the response! I can see where the logs are showing the new path going to the …/influxdb2

i.e.
{“level”:“info”,“ts”:1613660493.4088225,“caller”:“upgrade/database.go:156”,“msg”:“Copying data”,“source”:"/var/lib/influxdb/data/metricdata/autogen",“target”:"/root/.influxdbv2/engine/data/8e8f7798fc2e1c71/autogen"}

How can I check where the config is reading from for the location of the new buckets?

I apologize if there is somewhere in the docs where I could be finding this information.

…/influxdbv2/config.toml

bolt-path = “/root/.influxdbv2/influxd.bolt”
engine-path = “/root/.influxdbv2/engine”
storage-series-id-set-cache-size = 100

10

You have to check InfluxDB server logs to see which data path is used: journalctl -u influxdb

11

Thanks for the help! @VlastaHajek

I created another V1.8 instance and added some data and users then upgraded.

I followed the journal logs and copied the data path just to be sure.

then

service influxdb stop
influxd --engine-path=~/.influxdbv2/engine
export INFLUXD_ENGINE_PATH=~/.influxdbv2/engine


root@Influxdb03:~/.influxdbv2/engine/data# service influxdb stop
root@Influxdb03:~/.influxdbv2/engine/data# influxd --engine-path=~/.influxdbv2/engine
2021-02-19T19:23:35.664269Z info Welcome to InfluxDB {“log_id”: “0SQwnyS0000”, “version”: “2.0.3”, “commit”: “fe04d346df”, “build_date”: “2020-12-15T01:00:16Z”}
2021-02-19T19:23:35.665045Z info Resources opened {“log_id”: “0SQwnyS0000”, “service”: “bolt”, “path”: “/root/.influxdbv2/influxd.bolt”}
2021-02-19T19:23:35.669073Z info Checking InfluxDB metadata for prior version. {“log_id”: “0SQwnyS0000”, “bolt_path”: “/root/.influxdbv2/influxd.bolt”}
2021-02-19T19:23:35.669127Z info Using data dir {“log_id”: “0SQwnyS0000”, “service”: “storage-engine”, “path”: “~/.influxdbv2/engine/data”}
2021-02-19T19:23:35.669155Z info Compaction settings {“log_id”: “0SQwnyS0000”, “service”: “storage-engine”, “max_concurrent_compactions”: 1, “throughput_bytes_per_second”: 50331648, “throughput_bytes_per_second_burst”: 50331648}
2021-02-19T19:23:35.669166Z info Open store (start) {“log_id”: “0SQwnyS0000”, “service”: “storage-engine”, “op_name”: “tsdb_open”, “op_event”: “start”}
2021-02-19T19:23:35.669203Z info Open store (end) {“log_id”: “0SQwnyS0000”, “service”: “storage-engine”, “op_name”: “tsdb_open”, “op_event”: “end”, “op_elapsed”: “0.038ms”}
2021-02-19T19:23:35.669227Z info Starting retention policy enforcement service {“log_id”: “0SQwnyS0000”, “service”: “retention”, “check_interval”: “30m”}
2021-02-19T19:23:35.669240Z info Starting precreation service {“log_id”: “0SQwnyS0000”, “service”: “shard-precreation”, “check_interval”: “10m”, “advance_period”: “30m”}
2021-02-19T19:23:35.669279Z info Starting query controller {“log_id”: “0SQwnyS0000”, “service”: “storage-reads”, “concurrency_quota”: 10, “initial_memory_bytes_quota_per_query”: 9223372036854775807, “memory_bytes_quota_per_query”: 9223372036854775807, “max_memory_bytes”: 0, “queue_size”: 10}
2021-02-19T19:23:35.669912Z info Configuring InfluxQL statement executor (zeros indicate unlimited). {“log_id”: “0SQwnyS0000”, “max_select_point”: 0, “max_select_series”: 0, “max_select_buckets”: 0}
2021-02-19T19:23:35.952447Z info Starting {“log_id”: “0SQwnyS0000”, “service”: “telemetry”, “interval”: “8h”}
2021-02-19T19:23:35.952527Z info Listening {“log_id”: “0SQwnyS0000”, “transport”: “http”, “addr”: “:8086”, “port”: 8086}

Then I was able to find the data from V1.8 that had been migrated.

thanks again for the help and hopefully this helps someone :v:

12

Now it’s time to migrate to 2.x but since I’m newcomer in using docker I need your guidance

I running influxdb version 1.8.2 on a Synology NAS with docker support.

image
image980×347 13.4 KB

Where to start

  1. Backup database
  2. Download latest inxluxdb container 2.0/2.0.9
  3. Stop influxdb 1.8.2 container
  4. …and now
  • What script to use to migrate
  • Do I need to use SSH command or can I manage through inbuild docker terminal?
13

@Magnus_P see this section of the docs: docs/influxdb at master · docker-library/docs · GitHub

14

Believe me, I have been there reading through a number of times. Here are output from my upgrade that fails. I want to retain my custom paths so I run this script…

 $ sudo docker run -p 8086:8086 \
   -v influxdb:/docker/influxdb/data \
   -v influxdb.conf:/docker/influxdb/influxdb.conf \
   -v influxdb2:/docker/influxdb2/data \
   -v influxdb2:/docker/influxdb2 \
   -e DOCKER_INFLUXDB_INIT_MODE=upgrade \
   -e DOCKER_INFLUXDB_INIT_USERNAME=my-user \
   -e DOCKER_INFLUXDB_INIT_PASSWORD=my-password \
   -e DOCKER_INFLUXDB_INIT_ORG=my-org \
   -e DOCKER_INFLUXDB_INIT_BUCKET=my-bucket \
   -e DOCKER_INFLUXDB_INIT_UPGRADE_V1_CONFIG=/docker/influxdb/influxdb.conf \
   -e INFLUXD_CONFIG_PATH=/docker/influxdb2/config.toml \
   -e INFLUXD_BOLT_PATH=/docker/influxdb2/influxdb.bolt \
   -e INFLUXD_ENGINE_PATH=/docker/influxdb2/engine \
   influxdb:2.0

And here is the result

time="2021-10-26T18:08:12Z" level=info msg="reading OCSP Response cache file. /root/.cache/snowflake/ocsp_response_cache.json\n" func="gosnowflake.(*defaultLogger).Infof" file="log.go:104"
time="2021-10-26T18:08:12Z" level=error msg="failed to open. Ignored. open /root/.cache/snowflake/ocsp_response_cache.json: no such file or directory\n" func="gosnowflake.(*defaultLogger).Errorf" file="log.go:120"
find: File system loop detected; '/docker/influxdb2/data' is part of the same file system loop as '/docker/influxdb2'.
16

@Magnus_P the problem is these mounts:

   -v influxdb2:/docker/influxdb2/data \
   -v influxdb2:/docker/influxdb2 \

You should only need one or the other

17

hallo, for you who are not sure to run upgrade i suggest you following:

1-install somewhere new influx_2 from scratch
2-get familiar with CLI config and on one of these system set both connections:

$ influx config ls --json
{
“influx_1”: {
“url”: “http://192.168.0.105:8086”,
“token”: “”,
“org”: “ooo”,
“active”: true
},
“influx_2”: {
“url”: “https://192.168.0.106:8086”,
“token”: “”,
“org”: “ooo”
}
}

3 - if config set on influx_2 machine and already familiar with flux, export all your data from influx_1: for instance like

$ influx query -c influx_1 ‘from(bucket: “vcely”) |> range(start: -6h)’ --raw > vcely_bucket_6h.csv

[if not familiar with flux, write same query with influxQL]

4 - verify your data and import it to influx_2 installation.
$influx write -c influx_2 --skip-verify --skipRowOnError --format csv -b test_bucket -f vcely_bucket_6h.csv -t TOKEN

this way, you can test a lot. if you make a mistake just delete whole bucket or just some measurements.

have fun, pavel

18

2 days and still no success. I really wanted to stay on InfluxDB because it’s been working for me so far. But I’ve been trying to upgrade to 2.0 from 1.8 for the last 2 days and still haven’t succeeded. It seems like everything I try fails. This is a very complicated upgrade process. Everything seems to work but when I try to add the DB to Grafana, it fails with unauthorized. I followed all the steps (including adding a non-admin user to inlfux1.8) and it doesn’t get ported to 2.0 after influx upgrade. SOOOO frustrating!!!

19

Still struggle to get the migration to 2.x using Synology NAS with docker. I get the following error code when checking the docker log Error: 1.x meta.db 'meta.db' does not exist

Please anyone??

image
image1171×876 45.8 KB

20

@Magnus_P I have the same problem.
Were you able to solve it?

21

@henkiepenkie - I did not managed to migrate the data during installation. Instead I took a detour by exporting the data from 1.8 to InfluxDB line protocol data format, then after importing in 2.1