Influxdb concurrent access with Python under Docker performance decrease

Telegraf
1

Hello,

My issue is as follows:
I have two docker containers in a stack (docker swarm), one is influxdb 1.5, the other contains N python processes. Each python process reads a message from an AMQP queue, then reads data in influxdb with on single query and finally writes data to influxdb with on single query.

Up until N=3 the read query execution time is around 0.02s (measured as the python client execution time). When I go above N=3, so that more client are requesting data from influxdb, the read query can take up to 0.3s. Thus defeating the purpose of running parallel processes.

I can’t figure out if it is an influxdb issue or a docker network issue or something else.

I would be glad to get some help on this :slight_smile:

2

Hello @Christophe_P,
Welcome and thanks for your question. I have a few questions for you:

  • Why are you using 1.5?
  • What sort of ingestion are you trying to acheive for those 3 processes?
  • What are you using to write to InfluxDB? (if you’re using the python client I highly recommend using the 2.0 client. It’s much more performant and you can perform multiprocessing)