Peformance Issue with Java Client Library

[Duck]

Hi, I’ve starting to use InfluxDB v2.0.4 for my Java project. What I read is InfluxDB works with NS(nanoseconds) but what I see from my little benchmark is minimum 100ms for only one data. Is there any key point settings to enable or use? or is Java client that slow?

I’ve tested three times and the results are;

1. 100ms
2. 102ms
3. 65ms

try (WriteApi writeApi = HexInfluxDB.getLib().getClient().getWriteApi()) {
writeApi.writePoint(this.bucket, this.company, new Point(“test_m”).addField(“measure_q”, 100));
}

[Franky1]

I would say one has nothing to do with the other.

Behind every API access of the client library is an HTTPS POST request to the InfluxDB database server - which of course takes its time.

If you need very precise timestamps, you should enter them directly into the Client Library when collecting the data. I don’t know the Java Client Library, but this should definitely be possible.

Another complementary possibility would be to use the Telegraf Agent on the client.
Telegraf can collect individual data points and then post them to the server in one go.

[Duck]

Thanks for the information! I actually don’t have any problem with write latency. I’m just scared when I saw long delay for only one small operation. Is there any problem to write constantly like 1000 write per second?

[Franky1]

Hm, I don’t know your application, but if there are permanently 1000 data points per second, I wouldn’t push them individually with the client library, otherwise you have an enormous protocol overhead. And also with every request the delay caused by the response from the database server.
That could cause problems. I’m just wondering whether the client libraries can also buffer and then send them in bundles.

In this case, I would first collect the data points and, for example, push them every 1 second or every 10 seconds in a separate, slow-running thread.

Another possibility would be that your application pushes the data points to a local Telegraf agent who then handles the buffering and transfer to the InfluxDB.

[Duck]

I’m a it confused. Isn’t InfluxDB’s aim TSDB and be faster? Even it has HTTP requests, it doesn’t make any sense… I just push only one data and it takes 100ms which is too long. I’ve used a lot of databases even TSDB but it really takes too long compare to others.

For Telegraf, is Telegraf basically a data pool? Like we push data in it and it sends to InfluxDB every second.

[Franky1]

I’m not that familiar with the inner details of InfluxDB and the client libraries - perhaps the professionals should comment on that.

But basically you always have a certain roundtrip time for an http request that you can’t undercut.
Where is your InfluxDB running? localhost? Self-hosted? InfluxDB Cloud?
What are the ping times from the client to the InfluxDB?

If the client libraries are programmed asynchronously, this should work even with thousands of requests per second. But even if it works technically, I wouldn’t do it that way. Because of the enormous protocol overhead.

I would either pool the data points in the client application and push them collectively. You can already set the timestamps when creating the data points.

Or I would use a Telegraf agent locally. The application pushes the data points to Telegraf. Telegraf can pool the data and send it collectively. How and how often can be configured.
But Telegraf can do much more than that.

@Duck

or is Java client that slow?

It could also be that you are using the client library inappropriately.
Perhaps there is a bottleneck?
Could it be that you are creating a new instance of the writeApi each time?

[Duck]

@Franky1

Where is your InfluxDB running? localhost? Self-hosted? InfluxDB Cloud?

Localhost, everything inside Docker containers.

@Franky1

I would use a Telegraf agent locally

Telegraf hasn’t any Java client library. If it’s possible, I would not work with YAML and configuration for getting data from JVM(Java Virtual Machine, basically applicaton server.)

@Franky1

Could it be that you are creating a new instance of the writeApi each time?

I’ve though same thing but isn’t it bad practice to use only one client/connection? In MongoDB, I use only one connection. It’s normal to use only one connection for everything but I couldn’t find any documantation about InfluxDB java library. (Even have looked GitHub)

EDIT I’ve tested and use only one writeApi. It works and only takes 1ms. Isn’t it bad to use writeApi for a server that might be run for a long time like 24-48 hours.