What options do I have to help reduce IO on an embedded system with SD card?

Store
,
1

Hi. I have a few questions regarding running InfluxDB with an SD card as the primary data store. I originally tried to post these questions to the recent blog post at the following link (a couple of times, in fact) but they never showed up. Maybe they are stuck in moderation, who knows.

Just as a prefix, SD cards or eMMC typically have a limited number of erase/write cycles per erase block. Depending on the type of card, this number could be 100k or 3k even. However, by default, InfluxDB writes to the WAL with every single insertion, which would be a big hit to the card. What are the “knobs” I have to try to reduce IO load on the SD card? I have listed all the ones I know of below plus ones I’m curious about.

Use case: multiple sensors (say, up to 20) generating points with multiple fields (say, up to 4) at fairly high rate (say, up to 10 Hz). Data is written often but queried rarely.

  1. Smartly choose retention policies to downsample and otherwise limit the total amount of data stored on the SD card, giving the SD card’s wear leveling algorithms lots of space to use for wear balancing. In this same vein, storing the data in a way that allows high compression ratio is also useful, such as ints.
  2. Use a high-endurance industrial SD card, for instance from company Swissbit, to give the SD card a long lifetime.
  3. Change influxdb.conf to set wal-fsync-delay from 0s to a higher number. This will limit the frequency that the kernel commits WAL writes to the SD card. The risk is that power loss could cause data loss of anything not yet fsynced. ext4 by default fsyncs every 5 seconds automatically, so this setting should be set with whatever file system options being used in mind.
  4. Change influxdb.conf to set the wal-dir to a location in an in-memory file system. This will use 10 MB of memory per WAL. On power failure, even more data will be lost. There’s already an in-memory cache, so this seems sort of a silly way circumvent the WAL. In this case, the cache-snapshot-mem
    ory-size could be reduced to a small number so that the cache is written to disk as TSM more frequently.
  5. Put the WAL directory on a device different than the SD card; something that can endure many writes, such as NVRAM. The problem here is that the size of each WAL file is 10 MB, and getting large amounts of NVRAM is not cheap and sometimes not possible. How many WAL files will exist at once? Is it possible to reduce their size from 10 MB and also predict the total space needed for the WAL directory?
  6. Choose a file system for the SD card that nicely fits the IO patterns of InfluxDB. Mount it with noatime option to prevent lots of file system metadata updates. What file system is best? ext3, perhaps with journaling disabled?
  7. Anything else?

Thanks very much.

2

How is this working for you? Are there particular “knobs” which are working better than others?

I have similar concerns for rolling out embedded devices to numerous inaccessible locations where I really only need one minute interval data to be stored for around a month while replicating fifteen minute downsampled data to the cloud network permitting. The network being of questionable reliability and potentially even cellular at times.

3

My team’s current plan is to use a FRAM chip to log data to a cache file, then only commit to Influx once the cache fills up. This drastically reduces the IO load on Influx at the expense of delaying data from reaching the database. In some sense, we’re using our own WAL instead of the Influx one, which gives us more control to not only put it in the NVRAM chip, but also manage its size and layout. In practice, we expect the delay to be around a couple of minutes for the most delayed data. Our back of the envelope calculation was that using this scheme with a mid-tier industrial SD card would cause wear out after 275 years of 24/7 operation, but we have yet to test it and that value makes some assumptions about the SD card and how Influx works under the hood.

If you’re not logging to the SD card at high rate, you are probably fine putting the Influx files on the SD card and logging to Influx directly. Depending on the SD card’s wear leveling, size, and transistor tech (SLC, MLC, or TLC), you might be fine.

4

Another solution I am considering is putting the WAL on the FRAM chip but setting the cache-snapshot-memory-size configuration variable to a low value. It needs to be set low enough that the WAL space never exceeds the FRAM space, so it’s a process that requires tuning. More useful would be a WAL-snapshot-size configuration parameter to specify the allowed WAL size directly.

5

Related to 5: If you don’t mind increasing the power usage, size and price, you could add a small laptop harddrive.