Hello,
I have issue related to influxdb performance. version(1.5)
writes are 3 to 4 thousand per seconds.
Series cardinality is high.
Machine configuration is 8 cores and 32 gb RAM.
retention policy is of 4 weeks and shard duration is 1 day.
when i refresh the grafana( version 4.6) dasboard it takes more than 2 minutes to load the dashboard.
i want to reduce the dashboard loading time from munutes to couple of seconds.
what changes can be made in influx config file to achieve such level of performance.
or any other changes which could help me to come out of this problem.
Please help.
Hi Jassi, did you try to identify all the queries Grafana dashboard makes to influxdb ?
If you execute them all one-by-one - how long it takes ?
How much data they return ?
The following information is only for one day data
question asked by you : 1.How much data they return ?
query returns 5 millions of points per day.
2. If you execute them all one-by-one - how long it takes ?
graph panel query takes 40 to 50 seconds on grafana as well as on influx and singlestat panels takes 10 to 15 seconds.
As i have only 24 hours data in my measurement. due to slow response i deleted database and gave a fresh start.
in 24 hours i have 10 million points.
series count = 271616
my dashboard has 1 graph panel , 8 singlestat apnel and 7 pie chart. the dasboard takes 60 to 70 seconds to load.
Hi pawan
Thanks for the reply.
we have not set any dashboard refresh time i.e. auto refresh.
time filter is set to “today so far”.
changing “dashboard refresh time” does not help out.
dashboard takes less time to load “last 15 minutes” comparative to “today so far” .
If you’re series cardinality is about 270,000 InfluxDB performance should be fine. I’d recommend trying to run the query in the CLI tool for InfluxDB so we can see whether it’s a Grafana issue or a query issue.
You say that in 24 hours you have 10 million points. This seems less like an influxdb issue and more like a “my browser is choking on the amount of data that Influx is giving me” issue
When you go to the query inspector in your Grafana panel and expand the JSON data, how many points are being rendered?
Here’s an example from one of my dashboards:
xhrStatus:"complete"
request:Object
method:"GET"
url:REDACTED
params:Object
u:REDACTED
p:REDACTED
db:REDACTED
q:"SELECT sum("ifHCInOctets") FROM "30d"."ifMIB_basics" WHERE ("carrier_name" =~ /^Hurricane
Electric$/) AND time >= now() - 3h GROUP BY time(1m) fill(null);SELECT sum("ifHCOutOctets") FROM
"30d"."ifMIB_basics" WHERE ("carrier_name" =~ /^Hurricane Electric$/) AND time >= now() - 3h
GROUP BY time(1m) fill(null)"
epoch:"ms"
data:null
precision:"ms"
response:Object
results:Array[2]
0:Object
statement_id:0
series:Array[1]
1:Object
statement_id:1
series:Array[1]
0:Object
name:"ifMIB_basics"
columns:Array[2]
values:Array[181] <---- look here
If that number above is high, then you need to tweak your dashboard/panel so that you’re not returning so many results in the series (you already noted that loading 15 minutes takes less time - this is because the points returned would be in the tens of thousands, rather than the hundreds of thousands or millions), or you need to create RPs/CQs so that the data you want to graph isn’t so vast. 10 million points in a 24 hour period is a huge amount of data. Influx will happily return it all, but that doesn’t mean your browser will like it.
Hi jjensen3d
Thanks for reply.
i will create CQs and hope that issue will be resolved.
the pointed array value in JSON data is not high in my case.
I will share the results after i set CQs and will continue the conversation if issue doesn’t get resolved.
