I have a list of 10,000+ series that I need to be able to get the last value for. Currently I am either doing them one at a time (not the most performant) or I am batching them into groups of 10-100 and querying that way (performance is a little better). The communication between the API and Influx is main thing slowing things down so id like to minimize the amount calls as much as possible.
Ive read that there are performance issues with the contains function so I don’t think that is a great option. Ive also read that I can create multiple streams and union them together however with that comes the overhead of programmatically creating the flux script for very dynamic series/measurements/tag keys/fields. it will also most likely result in batching because of the amount of filters ill need to pull this off in my specific case.
I’m wondering if there have been any improvements in the past month that I might be able to take advantage of here. I’m using the .net InfluxDB API to interface with Influx. Is anyone else doing this or know of a good way of doing this? Is there a way to pass multiple flux scripts at one time instead of only passing one at a time? That would at least eliminate the amount of calls needed from Client to Server which would improve performance quite a bit.
@scott ive found this suggestion you made and am in the process of attempting it. however as ive been working on it ive realized just how much filtering ill still need and I’m starting to think this may not work for my specific case. Ill need at least one filter per series (if not more) and the last time I tried putting that many filters in a flux script it did not work. Are you aware of any improvements to this and or have any other suggestions for me to try? Would love to test some new ideas and see if I can help simplify this area of filtering.