Write data to InfluxDB with Java Client library

1

Hi everybody,

I would like to migrate timestamp data (about 500 millions of rows) from PostgreSQL to InfluxDB.
I use ReseltSet to read data from Postgres and java client (influxdb-java) to write points in InfluxDB (row by row). It’s only one measurement (One Table). To write the data to InfluxDB I also use a method influxDB.enableBatch(2000, 100, TimeUnit.MILLISECONDS).

In practice, I see that the InfluxDB write the points very slowly, only 500K in a day (around 30 points in a sec). With this performance, I need the whole year to copy my data.

  1. Is there any possibility to write data into InfluxDB faster?
  2. How many points in a sec. can I write theoretical to InfluxDB?

Thank you in advance!

P.S. I use Apple Mac mini - Core, which ist not real powerful.

2

@matulja Even with that setup you should be seeing faster write performance. Can you post the full hardware stats for the machine you are using for InfluxDB as well as the code you are using to write? The database is easily capable of write speeds in excess of 50k/sec even on very low powered hardware.

Can you also share the schema you are writing as well?

3

Hi @matulja

Could you pls share some code for writing to Influx measurements using Java.

You may email me at kedarsawant85@gmail.com

4

@kedar_sawant There are excellent examples in the documentation for the library,

5

@jackzampolin here is my Hardware Overview :

Model name: Mac mini
Prozessor Name Intel Core i5
Prozessor Speed 2,3 GHz
Total Number of Cores: 2
APPLE SSD TS128C (in Kapazität 121, 33 GB) Festplatte

And my code to write the points is:

 influxDB.disableBatch();
 influxDB.enableBatch(2000, 100, TimeUnit.MICROSECONDS);

  Point.Builder builder = Point.measurement(dataRecordAll.getMeasurements())
          .time(dataRecordAll.getTime(), TimeUnit.MILLISECONDS)
          .tag(dataRecordAll.getTagsData());

  for (Map.Entry<String, BigDecimal> entry1 : dataRecordAll.getFieldsData().entrySet()) {
    builder.addField(entry1.getKey(), entry1.getValue());

  }
  for (Map.Entry<String, BigDecimal> entry2 : dataRecordAll.getFieldsData2().entrySet()) {
    builder.addField(entry2.getKey(), entry2.getValue());
  }

  builder.build();
  Point point = (Point) builder.build();
  influxDB.write(dbName, "autogen" , point);

Write the data is very very slow…:frowning:

6

I find my Problem, I make mistake in my code…

1 Like
7

Hey, can you tell what’s the mistake and the correction you did to that. Thanks

8

Seems he is writing Point by point instead of batch points