To give some perspective, we are lucky to have some incredibly beefy boxes at our disposal and have been able to handle >1M points/sec on a single instance. All of that comes with a massive caveat which Jack ever so casually mentioned…series cardinality.
Even with the most tailored, amazing, phenomenally large machine in the world if your series are stuck under one measurement consuming your cardinality, you are toast. We encountered this in our own setup very recently and had to reformat our data collection in order to get back to proper ingest capabilities (>1M pps).
Our particular issue was caused by >90% of series under one measurement (you can check this with
influx_inspect report -detailed /path/to/shard/num) which bottlenecked any data coming into or out from the machine. We refactored the collector to send measurements in a much cleaner way (splitting into multiple measurements instead of one ‘umbrella’ measurement) and brought query times back down to sub-second levels quickly.
Outside of pure hardware improvements, it really does help to see how much impact your measurement collection has on the structure of the index altogether. Even with incredibly vertical machines, cardinality is still king.