Hello everyone!
We are encountering an issue with tasks in InfluxDB 2.0.
Currently, we have over 1800 active tasks, and for the past two weeks, we’ve noticed delays between when a task is scheduled and when it is actually executed. The tasks themselves are completing successfully without errors, but as shown in the first screenshot, we’re seeing delays of up to 15 minutes—which can grow to hours if left unresolved.
We use a 2-minute offset, so this is not the cause of the issue. Interestingly, if we deactivate and then reactivate the tasks, the delay disappears. However, once we restart InfluxDB, the delays start accumulating again.
In terms of resource usage, InfluxDB is not exceeding any limits (it is deployed in AKS).
We also noticed that the task_executor_delay metric has been steadily increasing over the past two weeks and no longer decreases.
Additionally, we’ve observed that task concurrency seems to be limited to 100 tasks simultaneously.
We’ve already tried increasing the query-concurrency and query-queue-size parameters, but the issue persists.
Does anyone know why this might be happening or if there is a way to increase the number of concurrent tasks? Could be the scheduler getting stuck with that amount of tasks?
Thanks in advance
flux
