Hello ,
I will try to explain my problem maybe somebody can help me with an idea or a solution .
I have some data into Oracle that i want to be inserted into InfluxDb.
I’m using telegraph so i create files with 10 000 line per file and i let telegraf to treat them (i choose this solution because i had put in place already telegraf that is treating files every minute)
In total i have 100 file with 10 000 line each so in oracle i have around 1 000 000 rows.
I was expected when i will insert all this data into influx and i will have the same number .
But not at all .
If i put all the 100 files in the same time , the number of lines into Influx is around 19000.
If i put file after file and let telegraf to treat the file entirely before to add the new file , every 10 000 lines per files are inserted , i did not test it with all 100 files , just with 10, but after each file treated , the number of lines was increesing well with 10 000 .
So i’m a bit confused , knowing that i will need a big amount of files to be treated in the same time.
Did somebody had the same problem , and find the solution?
Thank you for your help.
Do you have a time column in your data?
Do the data share the same series? (same tag keys and tag values)
a series is the collection of data that share a retention policy, measurement, and tag set
A point is uniquely identified by its series and timestamp
if you do not have a time column in your data influxdb will use the insert datetime.
maybe the points are “not” written because you are updating the same point over and over again.
if you insert the following
time | Tag1 | Tag2 | Field1
200 | aaa | bbb | 10
and then
time | Tag1 | Tag2 | Field1
200 | aaa | bbb | 12
the final result will be
time | Tag1 | Tag2 | Field1
200 | aaa | bbb | 12
this happens because the point with that timestamp and series already exists so its value is updated.
If you load the file one by one the timestamp will be different and the point won’t be overridden
is this your case?
Helle and thank you for your replay.
Yes i have a time in , is the time of insertion of the line .
Here are some of mine data in the files :
Id;Date;TempVal;Unit;TempType;TempS;EvStatus;UnitVal;
1;1578193980982;7.88118e-05;37;0;0;1;4
1;1578193981982;7.88117e-05;37;0;0;1;4
1;1578193982982;7.88116e-05;37;0;0;1;4
1;1578193983982;7.88116e-05;37;0;0;1;4
1;1578193984982;7.88118e-05;37;0;0;1;4
1;1578193985982;7.88116e-05;37;0;0;1;4
I telegraf i configured
csv_tag_columns = [“Id”,“Date”,“TempVal”]
so each line is uniq because of the date that is the occurrence date of the event.
when i’m inserting in influx , i let the column time to be the insertion time of the measurement, so should be unique olso.
a line in influxDb look like that
time EvStatus Id Date TempVal TempType TempS Unit UnitVal path
1579598684286272513 0 1 1578445428029 -1.58536e-08 0 1 1 4 /opt/…_11.dat
1579598684286323516 0 1 1578460237028 -1.58536e-08 0 1 1 4 /opt/…_23.dat
1579598684286323851 0 1 1579079014997 9.67072e-07 0 1 1 4 /opt…*_99.dat
what i think is a problem of telegraf configuration:
if i put all the 100 files with 10 000 line each into the folder where telegraf is listening and
if i configure metric_batch_size = 1000
metric_buffer_limit = 10000
the number of metrics in influx is = 14000
if i put all the 100 files with 10 000 line each into the folder where telegraf is listening and
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 500000
collection_jitter = “0s”
flush_interval = “20s”
flush_jitter = “0s”
the number of metrics in influx is = 502000,
if i put all the 100 files with 10 000 line each into the folder where telegraf is listening and
if i configure interval = “20s”
round_interval = true
metric_batch_size = 1000
Maximum number of unwritten metrics per output.
metric_buffer_limit = 1000000
collection_jitter = “0s”
flush_interval = “20s”
the number of metrics in influx is = 1000000, the the exact nb of lines i have in the file .
So what i’m seeing if that the metric_buffer_limit sould be large enought to treat all the files i put in the same time if not is treating just the number of his size .
It is true ?
Now is happening that i know what is the size of my files , but how i have to handle with the situation when i have to receive every minutes hounded of file with different size , that i don’t know in andvance .
So i don’t think is an overridden problem.
Thanks for your help.
So what I’m seeing if that the metric_buffer_limit should be large enough to treat all the files I put in the same time if not is treating just the number of his size
That’s true since you pass through Telegraf to write the data you must have a big enough buffer to store all the points before they are sent to Influxdb.
If you have no way to estimate the maximum size of the input dataset per given time unit (interval) a solution might be to write the data using a custom script and the write API.
This won’t use telegraf and requires you to write some code, but you will be able to manage the load process in a reliable way, by managing data read and write batching.
(this solution might be more or less complicated also depending on how the file are managed, ie: red and deleted, overridden once a day, written by appending data, etc)
Thank you very much for your answer.
Actually at begin i was thinking to use the solution to write points from a file,but i think i will add more time in process that i happening with telegraf , that is really quickly.
I have a sistem that is giving me evry minute csv files (differents files) with data that are inserted into influxdb.
I just let down this solution because i thought that it will take more time that just let telegraf to treat the files.
But i will give a try to see how much time i’m adding on top doing this process.
I was not expected that telegraf buffer is working like that , i was thinking it will load data with metric_buffer_limit and treat them ,and continuing the treatment until he will finish with all the data it has from beginning, that it will spleet in chunks and treat by metric_buffer_limit, but i guess not :(.
So i’m going exploring the next solution.
Thank you for your help.
Catalina
