If Prometheus is deployed in HA mode i.e. two instances scraping the same targets and streaming the collected data (each Prometheus instance stores its identity info as a label) to the same remote-write receiver of Telegraph, does Telegraph have the filtering capability to de-duplicate the collected data based on the Prometheus instance or identifier name?
Or are these metrics de-duplicated at the Influx-DB end?
You can send data to influxdb and if it is identical meaning the same metric name, same tag set, same field set, and same time, then the last entry will win.
What input are you using and are you expecting the URL or the instance where the data was collected from to stay in the data?
We do have the dedup processor as well.
Something like this:
my_app_http_total_requests{code=“200”,instance=“192.168.1.100:8080”,service_name=“example_service”,method=“GET”,endpoints=“/example/endpoint”, replica=”prometheus-0”} 12345 at October 24, 2021, 10:38:20 AM (UTC)
my_app_http_total_requests{code=“404”,instance=“192.168.1.101:8081”,service_name=“example_service”,method=“POST”,endpoints=“/another/endpoint”, replica=”prometheus -0”} 67890 at October 24, 2021, 10:38:20 AM (UTC)
my_app_http_total_requests{code=“200”,instance=“192.168.1.100:8080”,service_name=“example_service”,method=“GET”,endpoints=“/example/endpoint”, replica=”prometheus-1”} 12346 at October 24, 2021, 10:38:22 AM (UTC)
my_app_http_total_requests{code=“404”,instance=“192.168.1.101:8081”,service_name=“example_service”,method=“POST”,endpoints=“/another/endpoint”, replica=”prometheus-1”} 67899 at October 24, 2021, 10:38:22 AM (UTC)
Two Prometheus instances scrape the same scrape targets and stream the collected metrics on remote-write to Telegraph. As two instances can’t scrape exactly at the same time, there would be a mismatch in data and time-stamps.
Do you recommend preserving the uniqueness of data using replica label or do you think it should be avoided or removed?
The data as-is is not duplicate data due to the mismatch timestamp and due to the replica tag. You would need to remove the tag + reset the timestamps, which could get messy.
My personal opinion would be to collect all the data and filter on one of the replicas.
Thanks @jpowers for the prompt response!
I can remove the replica tag but I’m not sure about the timestamps. Because my HA metric collectors collect metrics parallelly I would like to keep them both & expecting the database to de-duplicate the collection. If we rely on one replica, I fear that we may miss out on some data due to the replica’s outages.
For example, AWS Managed Prometheus can de-dupe this based on a replica label. Please find this article on this subject: Send high-availability data to Amazon Managed Service for Prometheus with Prometheus - Amazon Managed Service for Prometheus.
How does Flux recommend users on metrics collection? Should all metrics be collected by a singleton metric collector?
The usage of Flux is a beyond me 
@Jay_Clifford thoughts on dedup with InfluxDB?
Your data itself is not actually duplicate data because it is coming from different sources at different times. In terms of Telegraf sending data to InfluxDB, recall my first comment about what defines duplicate data:
the same metric name, same tag set, same field set, and same time
As such you could use starlark to rip off the timestamp, but this means your data is not actually accurate I probably would not suggest this route.
1 Like
Thanks @jpowers for the confirmation!
But I wonder how Open Telemetry or Prometheus Agent HA deployments integrate with InfluxDB today?!
