InfluxDB 3 Core – 100 Parquet file query limit with Grafana

Systems
1

Hello,

I recently migrated my personal environmental monitoring project from Datacake to InfluxDB 3 Core + Grafana, running on a small DigitalOcean server.

My sensors send environmental measurements approximately once per hour.

Everything works correctly for short Grafana time ranges, but when I increase the time range I receive this error:

“Query would process 432 parquet files, which exceeds the limit of 100.”

The database currently contains only a few days of data, and my objective is to retain and query several years of environmental measurements.

Before changing my installation, I would like to know:

  1. Is increasing the 100-file query limit recommended with InfluxDB 3 Core?
  2. Is there a way to compact or reduce the number of Parquet files in Core?
  3. For long-term historical data with Grafana, would InfluxDB 3 Enterprise with automatic compaction be the better solution?

This is currently a personal, non-commercial environmental monitoring project.

If Enterprise At-Home is recommended, I would also like to know whether it can be used on my personal DigitalOcean server rather than a server physically located at home.

Thank you,
Paul

2

Hey @bropa1 - a few quick answers to your questions, then a recommendation:

  • InfluxDB 3 Core does not come with a compactor.
  • The default query file limit for Parquet files is actually 432 (which equates to a 72-hour time window), not 100. I’m not entirely sure why yours is set to 100.
  • You can absolutely increase this file limit from there, though the more you increase it, the more query performance may degrade (both in terms of latency and risk of OOM errors and crashing).

I would likely recommend Enterprise Home as an alternative, as if you’re using this for non-commercial purposes and don’t have massive volumes of data, the 2 CPU core limitation in Home will be significantly less restrictive than the query file limitation. As long as it’s not a commercial use case, you’re free to install it anywhere you like, including on a DigitalOcean server.

3

Thank you, this is very helpful.

My current DigitalOcean server has 1 vCPU and 1 GB RAM, and currently runs InfluxDB 3 Core and Grafana.

For my relatively low-volume environmental monitoring application, would 1 vCPU / 1 GB RAM be sufficient for InfluxDB 3 Enterprise Home with the compactor enabled, or would you recommend upgrading the server to 2 vCPU / 2 GB RAM or another configuration?

Thank you again.

4

I think your current configuration is plausibly fine, and I wouldn’t pre-emptively upgrade. If you see CPU/memory utilization constantly at 100% or it crashes, then that’s a good indication that it’s time for an upgrade. Good practice to not provision extra (and more expensive) hardware unless you’ve proven that you need to. :slight_smile:

5

I am upgrading from InfluxDB 3 Core to Enterprise 3.11.5 with an At-Home license.
The server starts successfully and logs:
“Email sent … Waiting for verification…”
However, I have not received the verification email, including in spam.
Is there a way to resend the At-Home license verification email?

6

I found another thread reporting a similar licensing issue, where it was confirmed to be a bug on the InfluxData side.
I am running Enterprise 3.11.5 and my server remains at “Email sent … Waiting for verification…”, but no verification email is received.
Could this be the same licensing issue, or is there a workaround for 3.11.5?