INFLUXDB performance


#1

Hello,
I have issue related to influxdb performance. version(1.5)
writes are 3 to 4 thousand per seconds.
Series cardinality is high.
Machine configuration is 8 cores and 32 gb RAM.
retention policy is of 4 weeks and shard duration is 1 day.

when i refresh the grafana( version 4.6) dasboard it takes more than 2 minutes to load the dashboard.
i want to reduce the dashboard loading time from munutes to couple of seconds.
what changes can be made in influx config file to achieve such level of performance.
or any other changes which could help me to come out of this problem.
Please help.


#2

Hi Jassi, did you try to identify all the queries Grafana dashboard makes to influxdb ?
If you execute them all one-by-one - how long it takes ?
How much data they return ?


#3

Hi Anatolijd
Thank you for the reply

The following information is only for one day data
question asked by you : 1.How much data they return ?
query returns 5 millions of points per day.
2. If you execute them all one-by-one - how long it takes ?
graph panel query takes 40 to 50 seconds on grafana as well as on influx and singlestat panels takes 10 to 15 seconds.

As i have only 24 hours data in my measurement. due to slow response i deleted database and gave a fresh start.
in 24 hours i have 10 million points.
series count = 271616
my dashboard has 1 graph panel , 8 singlestat apnel and 7 pie chart. the dasboard takes 60 to 70 seconds to load.


#4

Any help on this issue …PLEASE…


#5

Hey @jassi. Do you have tsi enabled?


#6

Hi katy
thanks for the reply.
TSI is enabled in my configuration.


#7

what is the dashboard refresh time ? have you tried changing that ?


#8

Hi pawan
Thanks for the reply.
we have not set any dashboard refresh time i.e. auto refresh.
time filter is set to “today so far”.
changing “dashboard refresh time” does not help out.
dashboard takes less time to load “last 15 minutes” comparative to “today so far” .


#9

If you’re series cardinality is about 270,000 InfluxDB performance should be fine. I’d recommend trying to run the query in the CLI tool for InfluxDB so we can see whether it’s a Grafana issue or a query issue.


#10

You say that in 24 hours you have 10 million points. This seems less like an influxdb issue and more like a “my browser is choking on the amount of data that Influx is giving me” issue

When you go to the query inspector in your Grafana panel and expand the JSON data, how many points are being rendered?

Here’s an example from one of my dashboards:

xhrStatus:"complete"
request:Object
  method:"GET"
    url:REDACTED
    params:Object
      u:REDACTED
      p:REDACTED
      db:REDACTED
      q:"SELECT sum("ifHCInOctets") FROM "30d"."ifMIB_basics" WHERE ("carrier_name" =~ /^Hurricane 
        Electric$/) AND time >= now() - 3h GROUP BY time(1m) fill(null);SELECT sum("ifHCOutOctets") FROM 
        "30d"."ifMIB_basics" WHERE ("carrier_name" =~ /^Hurricane Electric$/) AND time >= now() - 3h 
        GROUP BY time(1m) fill(null)"
        epoch:"ms"
      data:null
      precision:"ms"
    response:Object
      results:Array[2]
      0:Object
        statement_id:0
        series:Array[1]
      1:Object
        statement_id:1
        series:Array[1]
          0:Object
          name:"ifMIB_basics"
          columns:Array[2]
          values:Array[181]  <---- look here

If that number above is high, then you need to tweak your dashboard/panel so that you’re not returning so many results in the series (you already noted that loading 15 minutes takes less time - this is because the points returned would be in the tens of thousands, rather than the hundreds of thousands or millions), or you need to create RPs/CQs so that the data you want to graph isn’t so vast. 10 million points in a 24 hour period is a huge amount of data. Influx will happily return it all, but that doesn’t mean your browser will like it.


#11

Hi jjensen3d
Thanks for reply.
i will create CQs and hope that issue will be resolved.
the pointed array value in JSON data is not high in my case.
I will share the results after i set CQs and will continue the conversation if issue doesn’t get resolved.


#12

How many values is your query returning in the JSON?