Increasing query capacity with multiple servers reading from a single data volume?

Store
1

I’d like to do a sanity check on a server setup I have in mind.

The context:
We have an Influx instance running in AWS. This is handling a modest and consistent write volume, of around 10-20 writes/sec. The query volume, on the other hand, is both quite high and quite uneven. The run-rate during periods of moderately heavy use is >50 queries/sec, and sometimes the instance gets hit by >1,000 queries within seconds.

The problem:
Currently the server is running on an EC2 m5.xlarge instance with 32GB RAM. During periods of high query activity, we’ve started experiencing out-of-memory issues.

Solutions that don’t seem quite right:
We could of course keep upgrading to larger and larger EC2 instance sizes, but that doesn’t seem like the right long-term solution - especially since at night, for example, the query load is 1/10th of the daily peaks, so we’d have massively over-provisioned capacity. Switching to a HA setup with Influx Enterprise/InfluxCloud may be an option (if we can afford it), but it also doesn’t seem like the best fit for simply dealing with heavy/variable query loads.

Solution we have in mind:
It seems that Influx (reluctantly?) supports having its data volume be on a network drive. So we could in principle have an AWS Elastic File System that contains the influxdata partition, attached to our main EC2 instance over NFSv4. That in theory opens up the possibility of spinning up additional instances (/containers) to which this same volume is also attached over NFS. Let’s call these “query nodes”. Now, clearly they shouldn’t have write access to the influxdata volume, but perhaps if the query nodes only have read-only mounts, we could still allow them to execute queries against the data?

I appreciate that there are some caveats here, e.g. the query nodes not having access to the latest data, which the main node may be holding in memory and not yet committed to the data volume. That’s probably ok.

But primarily, before embarking on a trial-and-error exercise, I wanted to get an idea of whether such a setup is possible, or whether the idea is fundamentally flawed – e.g. if a read-only mount is simply not an option.

Any other ideas of how to deal with our situation would also be welcome! (“find a way to reduce the query volume” is of course on our radar already :slight_smile:)

2

As a starter (or…non-starter) for ten, mounting the data volume as read-only leads to the Influx server process failing to start. It hangs after the "Open store (start)" log message.

This is when the data volume is mounted from block storage rather than a network volume, but I suspect imposing read-only restrictions on a network mount would lead to a comparable outcome.

I’d be curious to understand what (fundamental or otherwise) considerations underlie this behavior, and whether there’s any way around.

3

Hello, Have you a solution for this probleme ?

4

I have not personally come across a solution since originally posting about it.