How to import old telegraf databases to InfluxDB v2

InfluxDB 2
1

Hi,

I am really struggling with the new influx db v2 and I am also very frustrated about the handling at all. Kapacitor gone, Chronograf gone, Grafana only Beta compatibility.

I´ve used influx v1 for a year via telegraf and imported data by a custom telegraf endpoint. These data seem to be available after the upgrade at /var/lib/… at debian 10, but I can´t find them in any of my buckets - also there is no bucket, which seems to be similar to my old database structure.
I´ve used influx upgrade and influxd upgrade on command line and all seems running well: there is an org, a token, an admin user and services are running, grafana can connect and find three buckets:
_monitoring
_tasks
logs

But: how can I import my old telegraf database to a bucket? I can´t find any hint or command how I can bring back my old data (bases). Has anyone a hint?

Thanks in advance!

2

Hi @benhartwich, influx upgrade should be converting your existing databases in-place. At first glance I can think of 2 reasons why it might not be happening.

First, influxd upgrade might not “see” the existing data files. When you run it, are you passing --config-file / --v1-dir to point at the 1.x files on disk?

Second, post-upgrade influxd might not be reading the upgraded config file, so it doesn’t know to read from the upgraded data files. If you’re upgrading a V1 influxdb.conf, the V2 equivalent will be written into the same directory as config.toml. If you don’t see the file, look in your home directory as well (through 2.0.2 it gets written there if the 1st option hits a permission error; starting in 2.0.3 the behavior will change, and instead the command will provide a CLI option to specify the target path). Once you find the file, you’ll need to point at it using an environment variable when running influxd:

INFLUXD_CONFIG_PATH=/path/to/config.toml influxd

Please let me know if you’re still hitting problems after double-checking these 2 possibilities.

1 Like
3

Hi,

thanks - it´s a debian default package installation with no folder customizations. At /etc/influxdb/ there is this config.toml file, but it seems to be updated / migrated to v2 already.

The series is saved at /var/lib/influxdb/telegraf/autogen/ - it seems to be the default telegraf database from v1. Is there any telegraf migration, which I need to run? The influx installation was upgraded a while ago via apt update it seems. I didn´t recognize it and now I´m trying to get all services up again and running.

4

Telegraf outputs are handled like other databases within InfluxDB, so you shouldn’t need to do anything special to upgrade them other than run influxd upgrade.

The fact that /etc/influxdb/config.toml exists makes me think that the upgrade command did find your existing config. Do you see any data files under ~/.influxdbv2? What happens if you run:

INFLUXD_CONFIG_PATH=/etc/influxdb/config.toml influxd

You might also want to consider downgrading your package back to the 1.8.x line for now. Pushing 2.0 packages under the influxdb name was a UX oversight on our part during the GA process, and we’ve since removed them from the repo. Our current plan is to release under a new influxdb2 name, starting with the 2.0.3 release. Some more context is here.

5

Thanks. I´ve to stop influxdb in order to get this command running. The command is executed, but it seems that it listens only without migrating:

2020-12-03T05:58:29.577704Z     info    Welcome to InfluxDB     {"log_id": "0QqnjTnG000", "version": "2.0.1", "commit": "37cc047133", "build_date": "2020-11-11T03:53:31Z"}
2020-12-03T05:58:29.579661Z     info    Resources opened        {"log_id": "0QqnjTnG000", "service": "bolt", "path": "/root/.influxdbv2/influxd.bolt"}
2020-12-03T05:58:29.585839Z     info    Using data dir  {"log_id": "0QqnjTnG000", "service": "storage-engine", "path": "/root/.influxdbv2/engine/data"}
2020-12-03T05:58:29.585897Z     info    Compaction settings     {"log_id": "0QqnjTnG000", "service": "storage-engine", "max_concurrent_compactions": 3, "throughput_bytes_per_second": 50331648, "throughput_bytes_per_second_burst": 50331648}
2020-12-03T05:58:29.585916Z     info    Open store (start)      {"log_id": "0QqnjTnG000", "service": "storage-engine", "op_name": "tsdb_open", "op_event": "start"}
2020-12-03T05:58:29.585987Z     info    Open store (end)        {"log_id": "0QqnjTnG000", "service": "storage-engine", "op_name": "tsdb_open", "op_event": "end", "op_elapsed": "0.072ms"}
2020-12-03T05:58:29.586022Z     info    Starting retention policy enforcement service   {"log_id": "0QqnjTnG000", "service": "retention", "check_interval": "30m"}
2020-12-03T05:58:29.586047Z     info    Starting precreation service    {"log_id": "0QqnjTnG000", "service": "shard-precreation", "check_interval": "10m", "advance_period": "30m"}
2020-12-03T05:58:29.586097Z     info    Starting query controller       {"log_id": "0QqnjTnG000", "service": "storage-reads", "concurrency_quota": 10, "initial_memory_bytes_quota_per_query": 9223372036854775807, "memory_bytes_quota_per_query": 9223372036854775807, "max_memory_bytes": 0, "queue_size": 10}
2020-12-03T05:58:29.587091Z     info    Configuring InfluxQL statement executor (zeros indicate unlimited).     {"log_id": "0QqnjTnG000", "max_select_point": 0, "max_select_series": 0, "max_select_buckets": 0}
2020-12-03T05:58:29.889994Z     info    Starting        {"log_id": "0QqnjTnG000", "service": "telemetry", "interval": "8h"}
2020-12-03T05:58:29.890277Z     info    Listening       {"log_id": "0QqnjTnG000", "transport": "http", "addr": "127.0.0.1:8086", "port": 8086}
2020-12-03T06:28:29.586534Z     info    Retention policy deletion check (start) {"log_id": "0QqnjTnG000", "service": "retention", "op_name": "retention_delete_check", "op_event": "start"}
2020-12-03T06:28:29.586766Z     info    Retention policy deletion check (end)   {"log_id": "0QqnjTnG000", "service": "retention", "op_name": "retention_delete_check", "op_event": "end", "op_elapsed": "0.420ms"}

Yes, this folder exists for root and has configs, engine and influxd.bolt in it.

6

@benhartwich, could you, please, share the upgrade.log? It should be in the user’s home.

Most often problem is that users run infuxd ugprade as root, but it must be run under the same user as influxdb runs, so under the influxdb user: sudo -u influxdb influxd upgrade

7

Thanks, running the command without root doesn´t work, because he tries to create .influxdbv2 at root everytime. Running INFLUXD_CONFIG_PATH=/etc/influxdb/config.toml influxd upgrade with root succeded, but buckets() doesn´t show me the telegraf database:

 {"level":"info","ts":1606986561.4901881,"caller":"upgrade/setup.go:159","msg":"CLI config has been stored.","path":"/root/.influxdbv2/configs"}
{"level":"info","ts":1606986561.490266,"caller":"upgrade/database.go:35","msg":"Checking space"}
{"level":"info","ts":1606986561.5329006,"caller":"upgrade/database.go:51","msg":"Disk space info","Free space":"516 GB","Requested space":"1.3 GB"}
{"level":"info","ts":1606986561.5330343,"caller":"upgrade/database.go:58","msg":"Upgrading databases"}
{"level":"info","ts":1606986561.5330534,"caller":"upgrade/database.go:65","msg":"Skipping _internal "}
{"level":"info","ts":1606986561.533078,"caller":"upgrade/database.go:70","msg":"Upgrading database ","database":"telegraf"}
{"level":"info","ts":1606986561.5331163,"caller":"upgrade/database.go:89","msg":"Creating bucket ","Bucket":"telegraf/autogen"}
{"level":"info","ts":1606986561.5358748,"caller":"upgrade/database.go:100","msg":"Creating database with retention policy","database":"b3272794bf76171c"}
{"level":"info","ts":1606986561.5366452,"caller":"upgrade/database.go:118","msg":"Creating mapping","database":"telegraf","retention policy":"autogen","orgID":"091a95bbbb8d7c3c","bucketID":"b3272794bf76171c"}
{"level":"info","ts":1606986561.537489,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1571616000}
{"level":"info","ts":1606986561.538131,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1572220800}
{"level":"info","ts":1606986561.539193,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1572825600}
{"level":"info","ts":1606986561.5399375,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1573430400}
{"level":"info","ts":1606986561.5406117,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1574035200}
{"level":"info","ts":1606986561.5412316,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1574640000}
{"level":"info","ts":1606986561.541806,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1575244800}
{"level":"info","ts":1606986561.5426145,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1575849600}
{"level":"info","ts":1606986561.5433614,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1576454400}
{"level":"info","ts":1606986561.5440562,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1577059200}
{"level":"info","ts":1606986561.5446656,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1577664000}
{"level":"info","ts":1606986561.5452921,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1578268800}
{"level":"info","ts":1606986561.54595,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1578873600}
{"level":"info","ts":1606986561.5469573,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1579478400}
{"level":"info","ts":1606986561.547686,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1580083200}
{"level":"info","ts":1606986561.5483437,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1580688000}
{"level":"info","ts":1606986561.5489972,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1581292800}
{"level":"info","ts":1606986561.549604,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1581897600}
{"level":"info","ts":1606986561.5502133,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1582502400}
{"level":"info","ts":1606986561.5510652,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1583107200}
{"level":"info","ts":1606986561.5517354,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1583712000}
{"level":"info","ts":1606986561.5526805,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1584316800}
{"level":"info","ts":1606986561.553833,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1584921600}
{"level":"info","ts":1606986561.5547915,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1585526400}
{"level":"info","ts":1606986561.555506,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1586131200}
{"level":"info","ts":1606986561.556905,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1586736000}
{"level":"info","ts":1606986561.5601249,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1587340800}
{"level":"info","ts":1606986561.560886,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1587945600}
{"level":"info","ts":1606986561.5615547,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1588550400}
{"level":"info","ts":1606986561.562207,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1589155200}
{"level":"info","ts":1606986561.5630903,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1589760000}
{"level":"info","ts":1606986561.563758,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1590364800}
{"level":"info","ts":1606986561.5645032,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1590969600}
{"level":"info","ts":1606986561.565201,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1591574400}
{"level":"info","ts":1606986561.5658882,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1592179200}
{"level":"info","ts":1606986561.5671074,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1592784000}
{"level":"info","ts":1606986561.5679815,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1593388800}
{"level":"info","ts":1606986561.5687187,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1593993600}
{"level":"info","ts":1606986561.5694442,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1594598400}
{"level":"info","ts":1606986561.5702093,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1595203200}
{"level":"info","ts":1606986561.5711987,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1595808000}
{"level":"info","ts":1606986561.5719697,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1596412800}
{"level":"info","ts":1606986561.572701,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1597017600}
{"level":"info","ts":1606986561.5734658,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1597622400}
{"level":"info","ts":1606986561.574233,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1598227200}
{"level":"info","ts":1606986561.5752165,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1598832000}
{"level":"info","ts":1606986561.5759737,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1599436800}
{"level":"info","ts":1606986561.5767853,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1600041600}
{"level":"info","ts":1606986561.5776384,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1600646400}
{"level":"info","ts":1606986561.5786722,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1601251200}
{"level":"info","ts":1606986561.579565,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1601856000}
{"level":"info","ts":1606986561.5802233,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1602460800}
{"level":"info","ts":1606986561.5809176,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1603065600}
{"level":"info","ts":1606986561.5817525,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1603670400}
{"level":"info","ts":1606986561.5826938,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1604275200}
{"level":"info","ts":1606986561.584108,"caller":"upgrade/database.go:131","msg":"Creating shard group","database":"b3272794bf76171c","retention policy":"autogen","time":1604880000}
{"level":"info","ts":1606986561.5849862,"caller":"upgrade/database.go:146","msg":"Copying data","source":"/var/lib/influxdb/data/telegraf/autogen","target":"/root/.influxdbv2/engine/data/b3272794bf76171c/autogen"}
{"level":"info","ts":1606986573.1870513,"caller":"upgrade/database.go:161","msg":"Copying wal","source":"/var/lib/influxdb/wal/telegraf/autogen","target":"/root/.influxdbv2/engine/wal/b3272794bf76171c/autogen"}
{"level":"info","ts":1606986573.3513372,"caller":"upgrade/upgrade.go:431","msg":"Upgrade successfully completed. Start service now"}
8

Upgrade log shows copying data and server log shows the same data dir, so telegraf database should be shown.

How do you check buckets?

If you open a web browser and point it to the server and port 8086, at the login screen use credentials created during the upgrade and in Data->Buckets you don’t see it?

9

I´ve used buckets() within grafana. But I think, I found the issue. After the redone upgrade I´ven´t set the grafana login token again. Unfortunatelly I cannot examine the new root token by command line:

influx auth list
Error: Unauthorized access.

At influxdb.conf authentification is not required: auth-enabled = false
I don´t know, what messed this up…

10

InfluxDB 2 has different concepts and architecture. See the docs for more info.

In short, it offers built-in UI, very similar to Chronograf, but with more functions, available at http://localhost:8086.
InfluxDB 2 is secured by default. When using API, authorization is done using tokens.
UI is authorized by credentials entered during the upgrade.

All mighty token is also generated during the upgrade process.

You can list tokens in the UI->Data->Tokens, where you can also generate a new token per bucket/permission.

Buckets are also shown in the UI->Data->Buckets.

You can check data in the UI->Explore.

11

Exact same problem for us as well: influxdb upgrade states it is successful but data simply do not appear in the new database.
It might be due to the fact that we were running the upgrade command as root, but it should be handled internally by the command to work with the proper instance.
In V1 when I did a backup and restore, I didn’t care about the user I was running the command as, and data were properly stored in the proper folders. Right now it looks like the installation through deb packages is not yet “ready” to be used.
We really like V2 and the feature it has fullfill our needs, so we are looking forward to adopt it as soon as possible.

Thanks,

12

Hello,
Same problem here too ; in the migration log, I can see the buckets that have been created.
After migration the data folders are well created in /root/.influxdbv2/engine/wal/ …:
{“level”:“info”,“ts”:1606734839.3655813,“caller”:“upgrade/database.go:161”,“msg”:“Copying wal”,“source”:“/var/lib/influxdb/wal/nperf/autogen”,“target”:“/root/.influxdbv2/engine/wal/584741edf9e44a3d/autogen”}

However, I can’t see the buckets through the UI.
As I understand, @ benhartwich you’ve tried to upgrade your data using influxdb but it does not help right ?
If anyone has a solution, thank you for sharing, I am getting a bit lost in my migration :frowning:

13

@mirkocomparetti, @Yuann_B, if you run influxd as root and if you log in to the UI using credentials entered during the upgrade, you still don’t see migrated buckets?

14

I didn’t try to run manually the influxd process and check; I installed influx2 from the deb package, so the database was running as a service.
Now we stopped using the influxdb upgrade procedure for that reason and we are focusing on a scripted way to move the data around, reading from one database V1 and writing to the other one V2.

15

@mirkocomparetti, when InfluxDB is installed from deb package it runs as service under the influxdb user.
The easiest way smooth upgrade process is to run upgrade as influxdb user.

Another option is to grant permission of migrated data to influxdb user but then you need to specify infuxd to use different data path with -e and -m options, or just move data, or so.