Architecture: Collectd -> Telegraf -> InfluxDB, hub/spoke or fan-in

1

Hi there, I’m working on a proof of concept for an InfluxDB project where we are sending metrics from our 100+ application servers using collectd and I hope to ingest them into a centralised InfluxDB, visualise with Chronograf and Alert using Kapacitor.

We use 6 Linux servers in our datacenters to allow us to receive logs via SFTP and process them before sending them to other databases.

Is it more efficient to have the application servers sending metrics using collectd to the 6 local Linux servers running Telegraf, unpack the binary stream there and then forward collectd/JSON to the central InfluxDB/Chronograf server (fan-in)?

(100+) Application (collectd) -> (6) Linux Data Collection Server (Telegraf) -> (1) ICK Server

Or should we just send all the 100+ collectd binary streams directly to the central server that would unpack all the collectd streams and run the full TICK stack there (hub and spoke architecture)?

(100+) Application (collectd) -> (1) TICK Server

Thanks!

2

@sgreszcz The fan-in architecture is a more flexible and scalable one. It removes the parsing load from the database and provides batching functionality. Also this removes the need for database restarts if you need to change your collectd ingest configuration. I would definitely recommend going that route.

1 Like
3

Thanks Jack, that reinforces my decision.

I was swaying that way as I’m using the collector servers to manage log ingestion to ElasticStack and can also more easily manage ACL/firewall configurations with static collection servers which allow me to change or work on the upstream DB servers (like you suggested).

1 Like