Kapacitor won't start, gives "object already exists" error

1

Hello… I’m new to the TICK stack and have been asked to look at a server where kapacitor is not starting up. Another sys admin here stopped telegraf, influxd, and kapacitor on our influx server. In starting those back up, telegraf and influxd lauched fine, but kapacitor does not.

In the /var/log/kapacitor/kapacitor.log, the last several lines are:

[run] 2017/05/30 12:29:12 I! Kapacitor starting, version 1.2.0, branch master, commit 5408057e5a3493d3b5bd38d5d535ea45b587f8ff
[run] 2017/05/30 12:29:12 I! Go version go1.7.4
[srv] 2017/05/30 12:29:12 I! Kapacitor hostname: influx001._myhostname_
[srv] 2017/05/30 12:29:12 I! ClusterID: 8a4df061-c582-411f-bb5a-076c31680c02 ServerID: b74ea789-0bbc-433b-a476-517fb18abd2d
[task_master:main] 2017/05/30 12:29:12 I! opened
[httpd] 2017/05/30 12:29:12 I! Closed HTTP service
[task_master:main] 2017/05/30 12:29:12 I! closed
[run] 2017/05/30 12:29:12 E! open server: open service *task_store.Service: migrating tasks: object already exists

Those lines repeat going back all morning to when kapacitor was shutdown.

The lines following the shutdown are:

[run] 2017/05/30 10:57:08 I! Signal received, initializing clean shutdown...
[run] 2017/05/30 10:57:08 I! Waiting for clean shutdown...
[task_master:main] 2017/05/30 10:57:08 I! Stopped task: puppet_agent_failures
[run] 2017/05/30 10:57:38 I! time limit reached, initializing hard shutdown

So it looks like it was not a graceful shutdown for kapacitor. And now when it starts up, there is some object that already exists that is not allowing the startup.

So what could that object be? How do I clear it out so that kapactior will start correctly?

I have stop/started telegraf, influxd, kapacitor a few times. And have even rebooted the machine entirely. The server is RHEL 7.3. The packages were installed via RPM - kapacitor-1.2.0-1.x86_64, influxdb-1.2.4-1.x86_64, telegraf-1.3.0-1.x86_64.

Appreciate any help and insights, thanks!
-Bob

2

I kept looking at this and figured it out, wanted to post my fix in hopes it helps someone else sometime.

In looking around the file system more, I found the /var/lib/kapacitor/tasks/tasks.db file. I thought this might be related to the task_store referenced in my log file excerpt. The file was also dated a couple weeks ago. Although only having guesses as to what tasks.db file is for, I took a chance and backed it up and then cleared it out. And that allowed kapacitor to start up again:

cp tasks.db tasks.db-bak; cp /dev/null tasks.db
systemctl status kapacitor.service

For me, everything seems to be running fine. If there were saved tasks that needed processing in tasks.db, I didn’t worry about it as I don’t think I need them for my set up.

But was is the tasks.db file for? Can it be safely cleared out like that?

Thanks,
-Bob

1 Like
3

The tasks.db file is from an older version of Kapacitor, during startup Kapacitor was trying to migrate tasks from the old format into the new kapacitor.db file. So you can remove the file and everything will work fine, assuming as you stated that you do not care about any task data still existed in the tasks.db file.

1 Like
4

Yeah, I don’t think I need the old tasks. Thanks for the info and confirmation about the tasks.db file.
-Bob

1 Like