How managing "cache-snapshot-memory-size" with (very) different databases

Store
1

Hi all,

I’m facing an issue with databases containing many… or few data.
My configuration is “wal-dir” = local hard drive, and “dir” = nfs share.
If i understand well, with “cache-snapshot-memory-size” = 25mo, when a database reaching 25mo on wal, flush is starting to write wal-dir data to dir data.
For my biggest database, i see theses flush ~every 5mn
But for small database… they are reaching 2mo after some days… so they are never flushed to nfs…

  • If i modify “cache-snapshot-memory-size” to 2mo… my nfs server will hate me
  • If i keep “cache-snapshot-memory-size” = 25mo and reboot a node… i loose data from small databases and metric readers will hate me

Is there any solution for this type of case ? For example a “cache-snapshot-max-time” is a dream for me :wink:

Thx in advance,
François.

2

I think you’re looking for cache-snapshot-write-cold-duration; from the config documentation on Settings for the TSM engine:

cache-snapshot-write-cold-duration = "10m"

The time interval at which the engine will snapshot the cache and write it to a new TSM file if the shard hasn’t received writes or deletes.

Environment variable: INFLUXDB_DATA_CACHE_SNAPSHOT_WRITE_COLD_DURATION

3

Hi,

Thanks for your help, but it’s only for shard with no writes or deletes.

I have made some tests : launch influxd with only a change = cache-snapshot-write-cold-duration = "1m". I create a db, and put some data with influx client.

  • if i do nothing, 1mn after last write, the line “Cache snapshot (start)” happen, and the magic is ok… tsm file here :slight_smile:
  • if i do insert each 10 seconds, cache snapshot never happen :disappointed_relieved:

I have 2 fix in mind for my problem :

  • Put all measurements in same database… but i loose security (i have a user per database for differents uses)
  • Create a new influxd server dedicated to small databases (“cache-snapshot-memory-size” = 1m for example). But it’s take some hardware ressources, and if more writes happens, my nfs server will be sad again :wink:

Hope to have a better solution…

4

Yes, you’re right, sorry! Given your constraints, I’m not sure if there are better solutions than the ones you described, if you want to stick with open source.

Our enterprise product does include fine-grained authorization, which would let you write data to the same database while still having control over security. If this is for a business use case I’d be happy to put you in touch with someone from our sales team.

5

Hi,

Thanks for your answer. Going to enterprise version can be a solution, but not in short term (i’m in a research/hospital center…). Fine-grained authorization is a very interesting option :slight_smile:
Just before closing this topic, can you tell me if there is a chance a day to see my dream “cache-snapshot-max-time” configuration ? :wink:

Thanks again anyway :ok_hand: :slight_smile:

6

Hi,

I’m closing this topic, thx again :slight_smile:
For information, i will go to prometheus + cortex, make more sense for my case.

All the best.