@Luv, good questions all around!
First off, regarding CPU usage you should take a look at some of the _internal metrics (which you should enable for collection) to see if garbage collection is occurring or something else is happening at that time.
There are situations where compactions will consume CPU usage when dropping measurements from one level of compression to another. That being said, the spikes you’re seeing are quite drastic and I’d look to see what else is happening on the box during those intervals.
As for the in-mem store, the concept of the “wal” vs “disk” is something to note with influxdb. If properly configured (you might want to paste your config here, masking any sensitive info if necessary), all writes will go into the in-mem index (stored in the write-ahead log ‘wal’) and replicated to disk asynchronously. Your writes are therefore ack’d after completing the wal write and async added to .tsm files on disk.
The compactions from one level of tsm to another will also happen asynchronously and concurrently with on-going reads/writes. The inluxDB team has made major improvements to fork this work off from the incoming metrics to ensure locking is not a problem with heavy write loads.