Incremental backup using rsync or lsyncd

InfluxDB 2
,
1

Hello everyone,

I desperately need to ensure high availability for my current setup.

  • I have two server on two different location.
  • InfluxDB version: V2.7 OSS on bare-metal

I tried to use a cronjob which use the backup/restore command of influx but the problem is it takes 10hours to run since it uses a full backup.

My question is:

> Will it be possible to run rsync or lsyncd directly on the data at ~/.influxdbv2/engine (or any other value of engine-path) instead of running influx backup ?

I want to implement my own incremental backup using something like this to copy only the data from the last 24 hour:

find ~/.influxdbv2/engine \
    -type f \
    -ctime -0 \
    -exec rsync -a {} USER@REMOTE:~/.influxdbv2/engine

Thanks !

2

Hello everyone,

I desperately need to ensure high availability for my current setup.

Please tell us what your definition of “high availability” is.

I have two server on two different location.

Do you have clients connecting to both of those servers?

I want to implement my own incremental backup using something like this to
copy only the data from the last 24 hour:

Well, that certainly isn’t my definition of “high availability”, then.

Please explain to us what you are trying to achieve (not how you are trying to
achieve it):

  • what level of service are you trying to provide to clients (for example, in
    the situation where one of your servers stops working or becomes
    uncontactable)?

  • what potential problem/s are you trying to avoid affecting client service
    (for example, data corruption or loss, remote network latency, high server
    load…)?

If you can give us a good explanation of what you are trying to achieve, we
might be able to suggest ways to achieve it.

Antony.

3

As I said, I want to make my InfluxDB available on the second server in case of a failure on the first server; that’s what I mean by high availability. I’m stuck on the step where I duplicate the data between the two instance.

I want my client to be able to contact to the other instance in case the first one fail.

My instance currently host monitoring data from multiple services.
These data are shown on the Grafana dashboard.
I can’t afford to lose visibility on the monitoring data in my current instance InfluxDB.

I already do regular backup on the InfluxDB.

4

I want to make my InfluxDB available on the second server in case of a
failure on the first server

  • I can’t use the InfluxDB Clustering because it’s only for InfluxDB
    Enterprise
  • I can’t use the open source version (chengshiwen/influxdb-cluster) because
    of we cannot afford to buy another server for the meta node

What, not even a virtual machine?

  • I can’t use influxdb-relay for the same reason

My instance currently host monitoring data from multiple services.
These data are shown on the Grafana dashboard.
I can’t afford to lose visibility on the monitoring data in my current
instance InfluxDB.

I do wonder why you “can’t afford to lose visibility…” and yet also “cannot
affort to buy another server”. However your economic priorities are your own
decision.

I can only say that I suspect, given the above constraints, that InfluxDB may
not be the right solution for you.

I know that InfluxDB is a specifically-focused Time Series Database, and does
that job very well, whereas other more generalised database are not so good at
it, but a combination of MariaDB and Galera replication (both of which are
free, open source applications, and MariaDB can be used as a data source for
Grafana) would do what you want with no licensing costs, albeit less efficiently
for acting as a TSDB.

There may well be other similar alternatives, but MariaDB + Galera is a system
I am familiar with and find very easy to set up and maintain.

If you can’t use InfluxDB Clustering or Relay then I don’t think there’s going
to be a sufficiently low-latency solution to your requirements, using InfluxDB.

Other people here may have other suggestions, of course.

Antony.

5

It’s an enterprise level so it’s not my " economic priorities".
But anyway thanks for the time you took to answer.

6

The next release off OSS v2.X will have faster backups. The work has already been done in V1 Enterprise and V1 OSS (though not released in the latter yet).

1 Like