Database Host migration / replacement

Store
1

We are running influxDB v1.7.1 on 2x Servers which are behind a VIP. We use the relay to keep both DBs in sync. We have a requirement to replace one of the Servers, but are concerned about Data loss.

Is there a best practice for the migration? Anyone done it before and have lessons learnt etc?

2

This will be difficult to do without some sort of buffer.

If you had a buffer set up(AMQP/Kafka etc) you would follow this process

  1. Install Influxdb on the new host
  2. Turn off the queue reader/influx writer agent
  3. Run a sync(rsync in example) to copy data from the old host to the new host
  4. Update the influx output in the telegraf writer agent to the new host
  5. Start Influxdb on the new host
  6. Once finished starting, turn on the Telegraf writer instance

The reason this works is that the metrics get buffered in the queue until you are ready to start writing again. Without that, metrics would get dropped. Without a buffering technique or influx enterprise it will be nearly impossible to not lose data but you could minimize impact by doing something like this

  1. install influxdb on new host
  2. run rsync to copy data while host is alive
  3. stop relay
  4. run rsync to get most recent changes
  5. update relay to point to new host
  6. restart relay

You could potentially lose all metrics between step 3 and 6 depending on the Telegraf buffer sizes set