Data loss using SELECT * INTO

#1

Hi!

I am using the following line in a batch file to generate daily backups from a customer database:

%binPath%influxd.exe backup -portable -database %databaseToCopyCurrentName% -start %startTime% -end %endTime% %pathToBackup%

I merge these daily backups into my local copy of the customer database using the following line:

%binPath%\influx.exe -execute “select * into %databaseName%…:measurement from /.*/ group by *” -database %tempDatabaseName%

This all works fine as long as there is no additional time series generated in the customer database. If there is a new time series, this is not merged into my local copy.

Does anyone have a solution for this problem. Any help would be appreciated…

Regards

Ludger

#2

Hi @LudgerM welcome !

can you share the details of the step between the backup and the select into ?
Do you see the new time series ( i suppose you mean measurements here ) in the
%tempDatabaseName% database ?

#3

Hi @MarcV,

thank you for your response.

To your first question:
On the customer system, which has very restricted network access, the backup files are generated using the backup function of influxd.exe.
The resulting backup files are then transferred to us for analysis purposes.
My task is now to restore these backups into an existing database.
I hope this answers your question properly. Please let me know if this is not the case.

To your second question:
I haven’t checked this yet, since we directly deleted it as part of the restore process.
However, on the customer end there are definitly new measurements present.
So, I would expect them to be present in the %tempDataBaseName% database, right?

#4

Hi @LudgerM ,
It would be interesting to know where exactly the data is lost ,
Is it possible for you to find out ?

#5

@MarcV,

thanks for your help.

in my initial request I forgot to mention that before the SELECT * INTO query is executed, a temporary DB is created from the backup files using the restore function of influxd.exe. This DB is reffered to as %tempDatabaseName%. I hope I did not cause too much confusion.

Basically, I am following the example called “To restore data to database that already exists”, which I found here: https://docs.influxdata.com/influxdb/v1.7/administration/backup_and_restore/

In the meantime, I have been able to look into this tempDatabase and I can confirm that the additional measurements are also present in this databes. Therefore, I assume that the data is lost when the SELECT * INTO query is executed. Does that make any sense to you?

#6

Hi , it is strange … are the new time series generated in the same retention policy on the customer database ?
Is there only one retention policy in each database ?

#7

Hi,

I would think so, although I must admit that I am not 100% sure.
We do not use specific retention policies, therefore all time series should be in the same policy.

retentionPolicies
retentionPolicies.jpg909×1080 58.4 KB

#8

There is a second retention policy , can you see which measurements exist in each retention policy ?
use InGOmv.autogen
show measurements
use InGOmv.tsm_meta_rp
show measurements

do you see the missing one ?