Grafana is not displaying all filter variable values

[cslm]

Hello,

I have influxdb 2.7.4.
I have a tag (Client_Name) in my bucket. It has 3500 different values.

I try below to create filter variable on Grafana.

import "influxdata/influxdb/schema"
schema.tagValues(
    bucket: "Mbuck",
    tag: "Client_Name"
)

The variable is created. But when i try on grafana dashboard filter variable dropdown. It only shows some of the 3500 Client_Name values. It truncates and show maybe 800 maybe 1000 of variable values.

I also have same data in Influxdb 1.8 version. If i add this variable with influxql with 1.8 version db the variable is created and grafana is showing 3500 of all Client_Names. With below sql.

SHOW TAG VALUES FROM "SEL_VALUES" WITH KEY = "Client_Name"

We are planning to close influxdb 1.8 and continue with influxdb 2.7.4. But why the dashboard is not showing all of 3500 client names in the filter with Flux in influxdb 2.7.4 version?

There is also something strange. If i create variable with below command, it show alittle more client names. 1600-1800 client names. But again not displaying all 3500 client name values in grafana dashborad filter variable dropdown.

from(bucket: "Mbuck")
|> range(start: -30d)
|> filter(fn: (r) => r["_measurement"] == "Meatb")
|> keyValues(keyColumns: ["Client_Name"]) 
|> group() 
|> keep(columns: ["Client_Name"]) 
|> distinct(column: "Client_Name")

Why i cannot see all of 3500 client name values in the filter. If i run flux command in influxdb explorer script area. I can see all 3500 client names there. But cannot see in grafana filter.

[cslm]

The strange thing is.

If i create variable from influx 1.8 with influxql, i can see 3500 client names at variable creation page by preview of values section at bottom of page.

If i create variable from influx v2.7.4, i can only see 1000 client names at variable creation page by preview of values section at bottom of page.

Note : Influxdb 1.8 db and influxdb 2.7.4 db has same test data in databases. We are planning to pass to 2.7.4 version. But couldnt do it because of this limitation yet.

[scott]

@cslm schema.tagValues has a default time range (-30d) to limit the time range queried (this can be a very heavy query). I’m guessing the other tag values exist outside of the default time range.

You can set a custom time range using the start and stop parameters. But again, this can be a very heavy query. Try this and see if it makes a difference:

import "influxdata/influxdb/schema"

schema.tagValues(
    bucket: "Mbuck",
    tag: "Client_Name",
    start: -1y,
    stop: now(),
)

[cslm]

Hello @scott,

Actually totally i have 20 days data total. So 3500 client names are in this 20 days data.
I tried -30d and -1y. Nothing changed. Still filter data is truncated with variables created with flux. Filter variables created with old version influxql doesnt have problem and gives 3500 client names in filter area.

[scott]

Interesting… I wonder if Flux query results are being limited or paginated in some way.

[scott]

@cslm I must have read your initial post too quickly, because I missed this detail:

@cslm

If i run flux command in influxdb explorer script area. I can see all 3500 client names there. But cannot see in grafana filter.

This means something is being limited by Grafana. I think what may be happening is that Grafana is treating each unique tag value as a separate series. When you configure your InfluxDB data source in Grafana, there is a “Max series” setting that defaults to 1000. Try upping that to 4000 and see if it makes a difference.

[cslm]

Hello @scott,

I have already tried that. I made it 50.000. But still not working.

[scott]

Looks like this is a known issue. Grafana limits the number of variable items to 1000: Variable: Return more than 1000 results · Issue #59959 · grafana/grafana · GitHub

[cslm]

Hello @scott,

But how can i search 3500 client names in one dashboard. What kind of filter design mechanism should i create on dashboard to filter and search all 3500 client names?

[scott]

@cslm With the limit Grafana places on variable values, the only other thing I could think of would be to split the client names into groups. I’m not sure if Grafana lets you build dashboard variables that are dependent on other dashboard variables, but if so, you could do something like this:

In Grafana, define a custom variable that selects an alphabetical range and assigns that range to be used in a regular expression string. Let’s call this variable “CustomerGroup”

CustomerGroup

"A-E" : "A-E","F-J" : "F-J","K-O" : "K-O","P-T" : "P-T","U-Z" : "U-Z"

Then, define another variable (let’s call it Customer) that uses the selected value of the CustomerGroup variable (hopefully this is possible) to filter tag values. You can use the regexp.compile() function to compile the selected range string into a regular expression:

Customer

import "influxdata/influxdb/schema"
import "regexp"

customerGroupFilter = regexp.compile(v: "^[${CustomerGroup}]")

schema.tagValues(
    bucket: "Mbuck",
    tag: "Client_Name",
    start: v.timeRangeStart,
    stop: v.timeRangeStop,
)
    |> filter(fn: (r) => r._value =~ customerGroupFilter

I think this should work.

[cslm]

Hello @scott,

I think i found solution. But need help.

Normally with flux i cannot get more than 1000 variable values as dashboard variable filter.

But if i define 8000 variable values by creating custom variable on grafana manually, dashboard filter can search all 8000 values and bring in the selectbox.

So i have to use manuel custom variable in grafana.

Now my problem is updating this custom variable in certain time period. Probably everyday.
How can i update custom grafana variable value dynamically everyday.

A script will run and write new custom values seperated by “,” like this : “Server1,Server2,Server3,…,Server7998,Server7999,Server8000”

I need to fill below area with my scripts dynamically.

[Screenshot from Dashboard/Settings/Variables]

[scott]

@cslm It looks like you tried to include a screenshot, but it’s not showing up for me.

[cslm]

Sorry,
Adding again.

[cslm]

My friend, @scott, this worked as a workaround for me.
I can accept this as a solution.
You saved the day.

There is another opportunity for another solution that would be better for me. I opened another topic for it on Grafana. Can you also look that too.

How to update custom variable in grafana 10

If we cannot solve that one too i will continue with this solution.

[scott]

Does Grafana have an HTTP API for updating a dashboard variable? If so, you could automate this with a Flux task.

[cslm]

Hmmm, let me check this also. Thanks.

[cslm]

It seem like, there is :

Let me try.

payload = {
        "value": new_value
    }
response = requests.post(f"GRAFANA_URL}/variables/byname/{CUSTOM_VARIABLE_NAME}/update", headers=headers, json=payload)

[scott]

What format does Grafana require for the payload? String? JSON?