From what I read, telegraf uses polling to collect data from source, is there a way to use notification-based way? For example, it registers with events from source and in the callback of the event, extract data from event and push them to ES.
The problem with polling is that we may have huge traffic of data when polling. What is the usual practice to handle this huge traffic.
@wanjingxu InfluxDB is designed to handle extremely high ingest volumes. Generally an individual instance of InfluxDB can support 200-500 servers running telegraf. How many servers are you trying to monitor? Also while ES is great for logs it can be non performant for high volume timeseries workloads.
Thanks for replying! We plan to use ES. The sources could be about 300k VMs. If we want to collect topology of the VMs, they won’t change very often, so polling might be overkill. Notification might be a good choice and reduce the traffic.