How managing "cache-snapshot-memory-size" with (very) different databases

Hi all,

I’m facing an issue with databases containing many… or few data.
My configuration is “wal-dir” = local hard drive, and “dir” = nfs share.
If i understand well, with “cache-snapshot-memory-size” = 25mo, when a database reaching 25mo on wal, flush is starting to write wal-dir data to dir data.
For my biggest database, i see theses flush ~every 5mn
But for small database… they are reaching 2mo after some days… so they are never flushed to nfs…

  • If i modify “cache-snapshot-memory-size” to 2mo… my nfs server will hate me
  • If i keep “cache-snapshot-memory-size” = 25mo and reboot a node… i loose data from small databases and metric readers will hate me

Is there any solution for this type of case ? For example a “cache-snapshot-max-time” is a dream for me :wink:

Thx in advance,
François.

I think you’re looking for cache-snapshot-write-cold-duration; from the config documentation on Settings for the TSM engine:

cache-snapshot-write-cold-duration = "10m"

The time interval at which the engine will snapshot the cache and write it to a new TSM file if the shard hasn’t received writes or deletes.

Environment variable: INFLUXDB_DATA_CACHE_SNAPSHOT_WRITE_COLD_DURATION

Hi,

Thanks for your help, but it’s only for shard with no writes or deletes.

I have made some tests : launch influxd with only a change = cache-snapshot-write-cold-duration = "1m". I create a db, and put some data with influx client.

  • if i do nothing, 1mn after last write, the line “Cache snapshot (start)” happen, and the magic is ok… tsm file here :slight_smile:
  • if i do insert each 10 seconds, cache snapshot never happen :disappointed_relieved:

I have 2 fix in mind for my problem :

  • Put all measurements in same database… but i loose security (i have a user per database for differents uses)
  • Create a new influxd server dedicated to small databases (“cache-snapshot-memory-size” = 1m for example). But it’s take some hardware ressources, and if more writes happens, my nfs server will be sad again :wink:

Hope to have a better solution…

Yes, you’re right, sorry! Given your constraints, I’m not sure if there are better solutions than the ones you described, if you want to stick with open source.

Our enterprise product does include fine-grained authorization, which would let you write data to the same database while still having control over security. If this is for a business use case I’d be happy to put you in touch with someone from our sales team.

Hi,

Thanks for your answer. Going to enterprise version can be a solution, but not in short term (i’m in a research/hospital center…). Fine-grained authorization is a very interesting option :slight_smile:
Just before closing this topic, can you tell me if there is a chance a day to see my dream “cache-snapshot-max-time” configuration ? :wink:

Thanks again anyway :ok_hand: :slight_smile:

Hi,

I’m closing this topic, thx again :slight_smile:
For information, i will go to prometheus + cortex, make more sense for my case.

All the best.