What is my scraper status? And other scraper options

So one of the cool new things is these “Scrapers”. However, after configuration you have no visibility into the status of the scraper. Is it working? Is there an error? Where do I get details into this? So far as I can tell, there is none. It’s also not logged by default (running via Docker).

I would suggest that there be some more information in the webUI as to what the scraper is doing and how frequently. If you know what data to look for (or maybe using a new bucket), you can determine if something is going wrong - but not why.

EDIT: I found that sometimes it is logged. But again, not in the WebUI.

Additionally there should be an option to ignore certificate check failures (aka curl -k) for scraping.

Great feedback…thank you! We’ll log a handful of issues based on this. Much appreciated.

We’ve opened these two GitHub issues for anyone that would like to follow them:

Your feedback prompted discussions on several places we should provide much clearer feedback - thanks!