Prometheus with Telegraf

Hi,
I am using Telegraf to ship or say expose my metrics on /metrics path for my prometheus operator service monitor to capture application metrics…
But i am facing a problem like since i am parsing my logs using telegraf so for every new timestamp it is creating a new time series…
For example - metrix_x(time=t1), metric_x(time=t2)
So logically it should create a single time series with these timestamps as my base timestamp but i am not sure how to achieve that via telegraf and prometheus.
Need help on this…

Hello @rj23495,
Thanks for your question. Just to be sure, you’re using the prometheus input plugin? Can you please share the output from your /metrics endpoint, the InfluxDB metrics, and your telegraf config?

@Anaisdg, I am using prometheus output plugin and storing my data in prometheus as TSDB.
I also wanted to parse and split path field which is added as label in prometheus output.
my log line looks something like this -
Also theres an issue all the log lines are not shipped or exposed i guess to prometheus due to which the data i am getting doesnt seem reliable.
To avoid the timestamp issue i have dropped that term/field so that i am aggregation is done at the pod level
Should i be using inputs.tail plugin or logparser as there are lot of files to parse…
2019-12-05 05:06:04.499 [634dcaa5-4ed5-491b-a21c-7e16ba882e6b] metric TabularOperations_getTaskTabularData_success_latency int32 4113 milliseconds

telegraf.conf: |+
    [[outputs.prometheus_client]]
      listen = ":9273"
      path = "/metrics"
      expiration_interval = "0"
    [[inputs.logparser]]
      files = ["/var/log/abc/*/*/logs/service.log"]
      from_beginning = false
      [inputs.logparser.grok]
        patterns = ['%{TIMESTAMP_ISO8601} \[%{DATA}\] %{DATA:metric} %{DATA:metricname} %{DATA:datatype} %{NUMBER:metricvalue:float} %{GREEDYDATA:unit}']
        measurement = "test_service_log"
    [[processors.parser]]
      parse_fields = ["metricname"]
      drop_original = true
      merge = "override"