we had telegraf running for a week. All of a sudden it stopped collecting metrics. The log has lots of lines similar to Error in plugin [inputs.mysql]: took longer to collect than collection interval (1m0s)
We tried to capture the core dump through SIGQUIT. I have attached the core dump here. Can someone help me decipher from the core dump what might be going wrong here. telegraf at this point was collecting OS and mysql metrics. Mysql application was working fine and responsive all the time.
Dec 04 21:00:34 Linux-691 telegraf[29344]: SIGQUIT: quit
Dec 04 21:00:34 Linux-691 telegraf[29344]: PC=0x45d003 m=2 sigcode=0
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 0 [idle]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.futex(0x1d15138, 0x0, 0x7f8570820d88, 0x0, 0x7f8500000000, 0x45cc96, 0x3c, 0x0, 0x7f8570820dd0, 0x410030, …)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sys_linux_amd64.s:426 +0x23
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.futexsleep(0x1d15138, 0x0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/os_linux.go:62 +0xd7
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.notetsleep_internal(0x1d15138, 0xdf8475800, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/lock_futex.go:174 +0xd0
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.notetsleep(0x1d15138, 0xdf8475800, 0x73d9d33dde101)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/lock_futex.go:194 +0x56
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.sysmon()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/proc.go:3805 +0x135
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.mstart1()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/proc.go:1179 +0x11e
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.mstart()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/proc.go:1149 +0x64
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 1 [semacquire, 428 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.runtime_Semacquire(0xc4203988cc)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sema.go:47 +0x34
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.(*WaitGroup).Wait(0xc4203988c0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/sync/waitgroup.go:131 +0x7a
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run(0xc42000e168, 0xc42033e480, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:426 +0x54e
Dec 04 21:00:34 Linux-691 telegraf[29344]: main.reloadLoop(0xc42033e840, 0x1d37f10, 0x0, 0x0, 0x1d37f10, 0x0, 0x0, 0x1d37f10, 0x0, 0x0, …)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:234 +0xa46
Dec 04 21:00:34 Linux-691 telegraf[29344]: main.main()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:411 +0x65b
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 17 [syscall, 428 minutes, locked to thread]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.goexit()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 23 [select, 428 minutes, locked to thread]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.gopark(0x14bb4a8, 0x0, 0x143ae50, 0x6, 0x18, 0x2)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/proc.go:271 +0x13a
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.selectgoImpl(0xc42028bf50, 0x0, 0x18)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/select.go:423 +0x1364
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.selectgo(0xc42028bf50)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/select.go:238 +0x1c
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.ensureSigM.func1()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
Dec 04 21:00:34 Linux-691 telegraf[29344]: runtime.goexit()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 6 [syscall, 428 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: os/signal.signal_recv(0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sigqueue.go:116 +0x104
Dec 04 21:00:34 Linux-691 telegraf[29344]: os/signal.loop()
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by os/signal.init.1
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 18 [select, 428 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.(*router).matchAndDispatch.func1(0xc42033e420, 0xc420130780, 0x1, 0xc420070380)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/router.go:132 +0x58e
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/eclipse/paho%2emqtt%2egolang.(*router).matchAndDispatch
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/router.go:162 +0x5c
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4528 [IO wait]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: net.runtime_pollWait(0x7f8570f91c88, 0x72, 0x4)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/netpoll.go:164 +0x59
Dec 04 21:00:34 Linux-691 telegraf[29344]: net.(*pollDesc).wait(0xc4200f4228, 0x72, 0x1c1e8e0, 0x1c158a0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Dec 04 21:00:34 Linux-691 telegraf[29344]: net.(*pollDesc).waitRead(0xc4200f4228, 0xc4203adc00, 0x400)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Dec 04 21:00:34 Linux-691 telegraf[29344]: net.(*netFD).Read(0xc4200f41c0, 0xc4203adc00, 0x400, 0x400, 0x0, 0x1c1e8e0, 0x1c158a0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/net/fd_unix.go:250 +0x1b7
Dec 04 21:00:34 Linux-691 telegraf[29344]: net.(*conn).Read(0xc42000e038, 0xc4203adc00, 0x400, 0x400, 0x0, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/net/net.go:181 +0x70
Dec 04 21:00:34 Linux-691 telegraf[29344]: crypto/tls.(*block).readFromUntil(0xc420238330, 0x1c1ab20, 0xc42000e038, 0x5, 0xc42000e038, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/crypto/tls/conn.go:488 +0x98
Dec 04 21:00:34 Linux-691 telegraf[29344]: crypto/tls.(*Conn).readRecord(0xc420167880, 0x14bb517, 0xc4201679a0, 0x3)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/crypto/tls/conn.go:590 +0xc4
Dec 04 21:00:34 Linux-691 telegraf[29344]: crypto/tls.(*Conn).Read(0xc420167880, 0xc42025c9a9, 0x1, 0x1, 0x0, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/crypto/tls/conn.go:1134 +0x11d
Dec 04 21:00:34 Linux-691 telegraf[29344]: io.ReadAtLeast(0x1c17760, 0xc420167880, 0xc42025c9a9, 0x1, 0x1, 0x1, 0x1c, 0x80, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/io/io.go:307 +0xa9
Dec 04 21:00:34 Linux-691 telegraf[29344]: io.ReadFull(0x1c17760, 0xc420167880, 0xc42025c9a9, 0x1, 0x1, 0x1, 0x80, 0x1c17760)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/io/io.go:325 +0x58
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho.mqtt.golang/packets.ReadPacket(0x1c17760, 0xc420167880, 0xc420167880, 0x1c17760, 0xc420167880, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/packets/packets.go:105 +0xa9
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.incoming(0xc420070380)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/net.go:86 +0x1ad
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/eclipse/paho%2emqtt%2egolang.(*client).reconnect
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/client.go:352 +0xa09
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4526 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.alllogic(0xc420070380)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/net.go:177 +0x275c
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/eclipse/paho%2emqtt%2egolang.(*client).reconnect
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/client.go:339 +0x889
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4527 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.keepalive(0xc420070380)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/ping.go:33 +0x7aa
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/eclipse/paho%2emqtt%2egolang.(*client).reconnect
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/client.go:349 +0xa58
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5729 [chan send, 237 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*accumulator).AddFields(0xc420356ec0, 0x1438f81, 0x5, 0xc4201cb260, 0xc4201cb1d0, 0x0, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/accumulator.go:53 +0x12e
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/mysql.(*Mysql).gatherGlobalStatuses(0xc4200b8dc0, 0xc4201a06e0, 0xc4201cae40, 0x2d, 0x1c2b1e0, 0xc420356ec0, 0x0, 0xc420269768)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/mysql/mysql.go:867 +0x1834
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/mysql.(*Mysql).gatherServer(0xc4200b8dc0, 0xc4201cae40, 0x2d, 0x1c2b1e0, 0xc420356ec0, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/mysql/mysql.go:584 +0x11d
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/mysql.(*Mysql).Gather.func1(0xc4202cc5e0, 0x1c2b1e0, 0xc420356ec0, 0xc4200b8dc0, 0xc42011a1e1, 0x22)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/mysql/mysql.go:164 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/plugins/inputs/mysql.(*Mysql).Gather
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/mysql/mysql.go:165 +0x1f1
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 24 [select, 428 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: main.reloadLoop.func1(0xc42033e4e0, 0xc42033e480, 0xc42033c150, 0xc42033e840)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:195 +0x24a
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by main.reloadLoop
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:209 +0x6ec
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 25 [chan send, 241 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flusher(0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc42033e7e0, 0x45860c, 0x180001)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:348 +0x412
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func1(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc42033e7e0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:396 +0x7a
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:400 +0x34f
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 26 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420015e80, 0xc420357260, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420015e80, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420015e80, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 27 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420015f40, 0xc4203574e0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420015f40, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420015f40, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 28 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420015fc0, 0xc4200a9c80, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420015fc0, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420015fc0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 29 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420130040, 0xc420296cc0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420130040, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420130040, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 30 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc4201300c0, 0xc4201b44e0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc4201300c0, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc4201300c0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 31 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420130180, 0xc4201279c0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420130180, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420130180, 0xdf8475800)
Dec 04 21:00:34 Linux-691 systemd[1]: telegraf.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 32 [select]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc42033e480, 0xc420130240, 0xc420356ec0, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e168, 0xc42033e480, 0xc420130240, 0xdf8475800, 0xc42033e780)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc4203988c0, 0xc42000e168, 0xc42033e480, 0xc42033e780, 0xc420130240, 0xdf8475800)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:422 +0x7d
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).Run
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:423 +0x505
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5728 [semacquire, 237 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.runtime_Semacquire(0xc4202cc5ec)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sema.go:47 +0x34
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.(*WaitGroup).Wait(0xc4202cc5e0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/sync/waitgroup.go:131 +0x7a
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/mysql.(*Mysql).Gather(0xc4200b8dc0, 0x1c2b1e0, 0xc420356ec0, 0xcc4c8a, 0x11e6be0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/mysql/mysql.go:168 +0x234
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc42033fce0, 0xc420130240, 0xc420356ec0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:153 +0x49
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.gatherWithTimeout
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:154 +0xcc
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4517 [chan receive, 280 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.(*baseToken).Wait(0xc420014c80, 0xc420014800)
Dec 04 21:00:34 Linux-691 systemd[1]: Unit telegraf.service entered failed state.
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/token.go:53 +0xa5
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/outputs/mqtt.(*MQTT).publish(0xc4200e8000, 0xc4200148c0, 0x35, 0xc420351e00, 0x66, 0x80, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/outputs/mqtt/mqtt.go:152 +0x107
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/outputs/mqtt.(*MQTT).Write(0xc4200e8000, 0xc4200e8620, 0xe, 0xe, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/outputs/mqtt/mqtt.go:141 +0x329
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/internal/models.(*RunningOutput).write(0xc4200b8c80, 0xc4200e8620, 0xe, 0xe, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/internal/models/running_output.go:179 +0x10c
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/internal/models.(*RunningOutput).Write(0xc4200b8c80, 0x14bb5d8, 0xc4201b2dc0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/internal/models/running_output.go:161 +0x497
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flush.func1(0xc4201b2dc0, 0xc4200b8c80)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:238 +0x59
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).flush
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:243 +0xa5
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 61 [semacquire, 245 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.runtime_SemacquireMutex(0xc4200b8d1c)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sema.go:62 +0x34
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.(*Mutex).Lock(0xc4200b8d18)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/sync/mutex.go:87 +0x9d
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/internal/models.(*RunningOutput).write(0xc4200b8c80, 0xc420628000, 0x3e8, 0x3e8, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/internal/models/running_output.go:176 +0x77
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/internal/models.(*RunningOutput).AddMetric(0xc4200b8c80, 0x1c34180, 0xc420611200)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/internal/models/running_output.go:119 +0x14c
Dec 04 21:00:34 Linux-691 systemd[1]: telegraf.service failed.
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flusher.func1(0xc420312090, 0xc42033e480, 0xc4203231a0, 0xc42000e168)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:284 +0x1e5
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).flusher
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:292 +0xe5
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5713 [chan send, 237 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*accumulator).AddCounter(0xc4201279c0, 0x143a3e8, 0x6, 0xc4201caba0, 0xc4201cab40, 0x0, 0x0, 0x0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/accumulator.go:75 +0x12e
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/system.(*DiskIOStats).Gather(0xc42033c1c0, 0x1c2b1e0, 0xc4201279c0, 0x1c17d20, 0xc4200123b0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/system/disk.go:158 +0x922
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc42033e000, 0xc420130180, 0xc4201279c0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:153 +0x49
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.gatherWithTimeout
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:154 +0xcc
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 62 [select, 428 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flusher.func2(0xc420312090, 0xc42033e480, 0xc42033e7e0, 0xc42000e168, 0xc4203231a0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:298 +0x2ae
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).flusher
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:315 +0x155
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5730 [chan receive, 237 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: database/sql.(*DB).connectionOpener(0xc4201a06e0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/database/sql/sql.go:837 +0x4a
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by database/sql.Open
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/database/sql/sql.go:582 +0x212
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4516 [semacquire, 280 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.runtime_Semacquire(0xc4201b2dcc)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/runtime/sema.go:47 +0x34
Dec 04 21:00:34 Linux-691 telegraf[29344]: sync.(*WaitGroup).Wait(0xc4201b2dc0)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /usr/local/go/src/sync/waitgroup.go:131 +0x7a
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flush(0xc42000e168)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:246 +0xd9
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*Agent).flusher.func3(0xc420323260, 0xc42000e168, 0xc42033e480)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:332 +0x7f
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.(*Agent).flusher
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:339 +0x2b0
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 4525 [select, 277 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/eclipse/paho%2emqtt%2egolang.outgoing(0xc420070380)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/net.go:114 +0xf49
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/eclipse/paho%2emqtt%2egolang.(*client).reconnect
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/eclipse/paho.mqtt.golang/client.go:338 +0x864
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5735 [chan send, 237 minutes]:
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.(*accumulator).AddGauge(0xc420357260, 0x1436d9a, 0x3, 0xc4201cb320, 0xc4201cb2c0, 0xc4201ae4e0, 0x1, 0x1)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/accumulator.go:64 +0x12e
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/plugins/inputs/system.(*CPUStats).Gather(0xc420126000, 0x1c2b1e0, 0xc420357260, 0xc4202cc680, 0xa)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/plugins/inputs/system/cpu.go:121 +0xfa1
Dec 04 21:00:34 Linux-691 telegraf[29344]: -github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc4203222a0, 0xc420015e80, 0xc420357260)
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:153 +0x49
Dec 04 21:00:34 Linux-691 telegraf[29344]: created by -github.com/influxdata/telegraf/agent.gatherWithTimeout
Dec 04 21:00:34 Linux-691 telegraf[29344]: /home/ubuntu/telegraf-build/src/-github.com/influxdata/telegraf/agent/agent.go:154 +0xcc
Dec 04 21:00:34 Linux-691 telegraf[29344]: goroutine 5734 [chan receive, 237 minutes]:
