We are using telegraf operator image for capturing the metrics from prometheus endpoint that is exposed by the OCP cluster. metrics are scraped and we have /metrics endpoint containing the metrics we wanted to monitor
telegraf-operator/deploy/dev.yml at master · influxdata/telegraf-operator · GitHub we are using this in our env, renamed as telegraf-operator.yml, but the content is same.
We are getting this error, how can this be fixed,
MutatingWebHookConfiguration is somehow not allowing to perform action on other deployments irrespective of namespace, in case if telegraf is throwing error.
In one case, when telegraf-operator was failing in our test cluster, we faced issues in deleting one of the deployment in other namespace than telegraf-operator, and error looked like this
•
[scb@scbbox haproxy-router-logging-2]$ oc delete po/fluentd-749f9dbf9d-dt72t
Error from server (InternalError): Internal error occurred: failed calling webhook "telegraf.influxdata.com": failed to call webhook: Post "https://telegraf-operator-test.telegraf-operator-test.svc:443/mutate-v1-pod?timeout=10s": service "telegraf-operator-test" not found
I have changed failurePolicy in deployment.yaml from ‘Fail’ to “Ignore’ which seems to have fixed it but I want to understand why telegraf is controlling other deployments as well, since this mutatingwebhookconfiguration is not limited to a namespace
Also there is a second question,
• In the official documentation, only method to fetch metrics are via providing pod-level annotations. But to provide annotation to all deployments is not feasible, so how to configure it to fetch from all existing and new deployments irrespective of annotations they are having?
I understand that when we provide influxdata annotation to a new deployment, telegraf creates a sidecar and then metrics are being sent to output defined in ‘telegraf-operator-classes.yaml’ file
apiVersion: v1
kind: Secret
metadata:
name: telegraf-operator-classes
namespace: telegraf-operator
stringData:
basic: |+
[[outputs.file]]
files = ["stdout"]
[[outputs.kafka]]
brokers = ["abc.com:9101"]
topic = "abctopic"
sasl_username = "abc"
sasl_password = "XXXXXXXX"
sasl_mechanism = "SCRAM-SHA-512"
Can some one help on this?
