Hi there
I am running InfluxDB on a single instance 8 core VM with 60GB RAM. The VM is running Ubuntu 16.04 Xenial and InfluxDB 1.3.1.
I am generating a write workload of about 2800/s. Each point in the timeseries is a separate request and has between 50 and 300 fields. There are about 10 measurements in the database.
A couple of hours after the workload started, the db crashed and I see the following error in the logs. Once the compaction started, CPU usage shot up and after a little while, the database shutdown with a sigpanic. I tried restarting the database but the tsm files seem to have gotten corrupted. I could only bring the database back up after deleting the tsm files.
I am using the default influxdb configuration and I am currently not batching the requests and will try to make a change to see if that helps. Aside from this, I was wondering if there is any setting I could tweak to prevent this from happening again or is the non-batch writes that I am doing the main problem ?
Any insight into this would be greatly appreciated
Jul 28 20:40:06 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:06Z beginning full compaction of group 0, 2 TSM files engine=tsm1
Jul 28 20:40:06 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:06Z compacting full group (0) /mnt/influxdb/data/blueshift/autogen/2/000000615-000000005.tsm (#0) engine=tsm1
Jul 28 20:40:06 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:06Z compacting full group (0) /mnt/influxdb/data/blueshift/autogen/2/000000637-000000004.tsm (#1) engine=tsm1
Jul 28 20:40:07 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:07Z compacted level 1 group (0) into /mnt/influxdb/data/blueshift/autogen/2/000000646-000000002.tsm.tmp (#0) engine=tsm1
Jul 28 20:40:07 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:07Z compacted level 1 3 files into 1 files in 12.705450299s engine=tsm1
Jul 28 20:40:29 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:29Z compacted full group (0) into /mnt/influxdb/data/blueshift/autogen/2/000000637-000000005.tsm.tmp (#0) engine=tsm1
Jul 28 20:40:29 ip-172-31-2-161 influxd[21528]: [I] 2017-07-28T20:40:29Z compacted full 2 files into 1 files in 22.466184429s engine=tsm1
Jul 28 21:20:40 ip-172-31-2-161 kernel: [242909.768230] blk_update_request: I/O error, dev nvme0n1, sector 435160504
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: unexpected fault address 0x7fa963c0a092
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: fatal error: fault
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: [signal SIGBUS: bus error code=0x2 addr=0x7fa963c0a092 pc=0x9e2014]
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: goroutine 38 [running]:
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: runtime.throw(0xb82f57, 0x5)
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc4203dfce8 sp=0xc4203dfcc8
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: runtime.sigpanic()
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/usr/local/go/src/runtime/signal_unix.go:287 +0xf4 fp=0xc4203dfd38 sp=0xc4203dfce8
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*indirectIndex).UnmarshalBinary(0xc4203ce280, 0x7fa96272ae26, 0x58d4743, 0x58d474b, 0x0, 0x0)
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:955 +0x1b4 fp=0xc4203dfdb0 sp=0xc4203dfd38
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*mmapAccessor).init(0xc420463000, 0x0, 0x0, 0x0)
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:1053 +0x2d9 fp=0xc4203dfe30 sp=0xc4203dfdb0
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: github.com/influxdata/influxdb/tsdb/engine/tsm1.NewTSMReader(0xc4204fe098, 0x27de2649, 0xef1580, 0x0)
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/reader.go:201 +0x136 fp=0xc4203dfea0 sp=0xc4203dfe30
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).Open.func1(0xc42036e180, 0xc4204f86c0, 0x3, 0xc4204fe098)
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: #011/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:394 +0x63 fp=0xc4203dffc0 sp=0xc4203dfea0
Jul 28 21:20:40 ip-172-31-2-161 influxd[23411]: runtime.goexit()
Thank you
Vinesh
