Influx_inspect export runs forever, consumes all disk space

Using influx_inspect export for the first time, have 2 databases that I want to dump. Thought I could dump all databases by not using the --database options, but it hits _internal/monitor and then builds a dump big enough to fill all available disk space.

Using Influx 1.1.0.

My /var/lib/influxdb is less than 150MB, but after almost crashing my / volume on my development/test VM I mounted a 100GB filesytem under /dump and tried it again, it got to 85GB before I killed it…

[root@vm-centos-72-mls ~]# systemctl stop influxdb.service 
[root@vm-centos-72-mls ~]# influx_inspect export -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal -out /dump/influx.dump
writing out tsm file data for performance_metrics/rp_7d...complete.
writing out wal file data for performance_metrics/rp_7d...complete.
writing out tsm file data for _internal/monitor...^C

And that’s as far as it got, /dump/influx.dump was 85GB and still growing…

@johndburk Can you upgrade to the 1.2.4 release? We have merged some fixes to backup and restore since the 1.1.0 release.

It’s a production server which I’m trying to migrate off of, would like to be able to skip the upgrade if at all possible.

If I manually re-create the users, create empty db’s, then add the retention polices and CQ’s, can I just use the --database arg to dump each DB separately? I have scripts which create the db’s, RP’s, and CQ’s.

1 Like

@johndburk Thats what I would do.

Sometimes the old-fashioned ways are the best.

1 Like

Got my db’s migrated the old-school way, so I’m good to go in that respect.

Then went back to my development host I had running the same version as my original production server (1.1.0) and upgraded it to 1.2.4-1

Same thing: chokes on _internal/monitor, will run and spew to the out file forever. Is there a way for me to file a bug report?

[root@vm-centos-72-mls influxdb]# influx_inspect export -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal -out /dump/influx.dump
writing out tsm file data for _internal/monitor...^C

[root@vm-centos-72-mls influxdb]# influx_inspect export -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal -out /dump/metered_licensing.dump -database metered_licensing
writing out tsm file data for metered_licensing/rp_infinite...complete.
writing out wal file data for metered_licensing/rp_infinite...complete.
writing out tsm file data for metered_licensing/rp_24w...complete.
writing out wal file data for metered_licensing/rp_24w...complete.
writing out tsm file data for metered_licensing/rp_72w...complete.
writing out wal file data for metered_licensing/rp_72w...complete.

[root@vm-centos-72-mls influxdb]# influx_inspect export -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal -out /dump/perf_metrics.dump -database performance_metrics
writing out tsm file data for performance_metrics/rp_30d...complete.
writing out wal file data for performance_metrics/rp_30d...complete.
writing out tsm file data for performance_metrics/rp_7d...complete.
writing out wal file data for performance_metrics/rp_7d...complete.

[jburk@vm-centos-72-mls ~]$ du -sh /dump/*
51G	/dump/influx.dump
2.5M	/dump/metered_licensing.dump
1.6M	/dump/perf_metrics.dump

last few lines of the 50GB dump look like:

tsm1_wal,database=metered_licensing,engine=tsm1,hostname=vm-centos-72-mls,id=3815,path=/var/lib/influxdb/data/metered_licensing/rp_72w/3815,retentionPolicy=rp_72w,walPath=/var/lib/influxdb/wal/metered_licensing/rp_72w/3815 currentSegmentDiskBytes=0i 1497500000000000000
tsm1_wal,database=metered_licensing,engine=tsm1,hostname=vm-centos-72-mls,id=3815,path=/var/lib/influxdb/data/metered_licensing/rp_72w/3815,retentionPolicy=rp_72w,walPath=/var/lib/influxdb/wal/metered_licensing/rp_72w/3815 currentSegmentDiskBytes=0i 1497500010000000000
tsm1_wal,database=metered_licensing,engine=tsm1,hostname=vm-centos-72-mls,id=3815,path=/var/lib/influxdb/data/metered_licensing/rp_72w/3815,retentionPolicy=rp_72w,walPath=/var/lib/influxdb/wal/metered_licensing/rp_72w/3815 currentSegmentDiskBytes=0i 1497500020000000000

@johndburk That would be awesome and much appreciated! Please open an issue on InfluxDB and reference this post.

Github bug report: https://github.com/influxdata/influxdb/issues/8516

Contains a reference back to this thread.

1 Like

After:

  • uninstalling influxdb
  • deleting /var/lib/influxdb
  • re-installing 1.2.4.-1
  • re-running my db creation scripts
  • adding a few series

I’m unable to reproduce this issue. Export completes in under a second, and writes out a 3M uncompressed export.

The database had originally been 1.0, then 1.1.0, then eventually 1.2.4-1, must have been an artifact from 1.0 or upgrading from 1.0 -> 1.1.0.

I’ve updated and closed https://github.com/influxdata/influxdb/issues/8516 as well.

1 Like

@johndburk Thank you for doing the work there! It is very much appreciated. Also always good to have bug reports like this kicking around incase anyone runs into similar.