How can I query all the buckets in an organization using Flux?

InfluxDB 2
, ,
1

Hello everyone

I have a use case that requires me to query all the buckets in an organization, not a specific bucket.

when querying a specific bucket, I use the syntax :

from(bucket: "bourges")
|> range(start: 2024-03-27T10:00:00Z, stop: 2024-03-27T11:00:00Z)

when querying an org, I use the syntax :

from(org: "test")
|> range(start: 2024-03-27T10:00:00Z, stop: 2024-03-27T11:00:00Z)

I get the error message :

{"code": "invalid", "message": "error calling function \"from\" @1:1-1:18: must specify bucket or bucketID"}

when querying a list of buckets, I use the syntax :

from(bucket: "bourges", "tours", "orleans")
|> range(start: 2024-03-27T10:00:00Z, stop: 2024-03-27T11:00:00Z)

I get the error message :

{"code": "invalid", "message": "compilation failed: error @1:6-1:43: string literal key tours must have a value\nnerror @1:6-1:43: string literal key orleans must have a value\nnerror @1:6-1:43: cannot mix implicit and explicit properties"}

when querying a list of buckets with several bucket tag, I use the syntax :

from(bucket: "bourges", bucket: "tours", bucket: "orleans")
|> range(start: 2024-03-27T10:00:00Z, stop: 2024-03-27T11:00:00Z)

I get the error message :

{"code":"invalid","message":"duplicate keyword parameter specified: \"bucket\""}

How do you go about multi-bucket queries?

Thanks for your help

Thierry

2

Hello there’s no built in way to do this.
What you’ll have to do is list your buckets and perform a query for each bucket.
The easiest way is to use a client library. Something like python for example so you can “parameterize” your query.
Lists buckets with requests library:
https://docs.influxdata.com/influxdb/cloud/api/#operation/GetBuckets

Then use the client library to iterate through buckets list and perform multiple queries.

3

Hello
Thanks for your reply @Anaisdg

Here’s how I solved the problem, using multi-processing to optimize queries :

	public List<FluxRecord> getVisitesBySite(String site, String dateDebut, String dateFin)
    {
    	String fluxTemplate  =  "from(bucket: \"{bucket}\")"
                             + "|> range(start: {dateDebut}, stop: {dateFin})";

		// je récupère la liste des buckets du site		
    	List<Bucket> buckets =  this.getConnexion()
    								.getBucketsApi()
    								.findBucketsByOrgName(site);

    	// je crée une liste pour stocker les résultats
        List<FluxRecord> results = new ArrayList<>();

        // je crée un ExecutorService avec un thread pour chaque bucket
        ExecutorService executor = Executors.newFixedThreadPool(buckets.size());

        // je crée une liste pour stocker les tâches futures
        List<Future<List<FluxRecord>>> futures = new ArrayList<>();
        
        // je soumet une tâche pour chaque bucket
        for(Bucket bucket: buckets)
    	{
            String fluxQuery = fluxTemplate.replaceAll(Pattern.quote("{bucket}"), bucket.getName())
                                           .replaceAll(Pattern.quote("{dateDebut}"), dateDebut)
                                           .replaceAll(Pattern.quote("{dateFin}"), dateFin);

            Future<List<FluxRecord>> future  =  executor.submit( () -> 
            									{
            										List<FluxRecord> bucketResults = new ArrayList<>();
            										
            										for (FluxTable table : this.getConnexion().getQueryApi().query(fluxQuery)) 
            											bucketResults.addAll(table.getRecords());
            									
            										return bucketResults;
            									});
            futures.add(future);
    		
    	}
    	
        // j'attends l'achèvement de toutes les tâches et collecte leurs résultats
        try 
        {
            for (Future<List<FluxRecord>> future : futures)
                results.addAll(future.get());
        }
        catch (Exception e) 	{ e.printStackTrace(); }
        finally 				{ executor.shutdown(); }
    	
    	return results;
    }

In the hope that this will be useful to as many people as possible
Have a nice day
Thierry