[inputs.influxdb] Error in plugin: invalid character '#' looking for beginning of value

Telegraf
,
1

Hi guys, i’m running Telegraf 1.27.1 and InfluxDB v2.7.1 and i have an issue when i try to get influxdb metrics with telegraf.

Here’s my telegraf conf :

[[inputs.influxdb]]
#   ## Works with InfluxDB debug endpoints out of the box,
#   ## but other services can use this format too.
#   ## See the influxdb plugin's README for more details.
#
#   ## Multiple URLs from which to read InfluxDB-formatted JSON
#   ## Default is "http://localhost:8086/debug/vars".
   urls = [
     "http://localhost:8086/metrics"
   ]

#   ## Username and password to send using HTTP Basic Authentication.
#   # username = ""
#   # password = ""
#
#   ## Optional TLS Config
#   # tls_ca = "/etc/telegraf/ca.pem"
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#   ## Use TLS but skip chain & host verification
#   # insecure_skip_verify = false
#
#   ## http request & header timeout
   timeout = "5s"

When i try to run it with telegraf in test mode i get no output :

telegraf --config /etc/telegraf/telegraf.conf --input-filter influxdb --test
2023-07-13T13:10:43Z I! Loading config: /etc/telegraf/telegraf.conf

But when i check the logs i got this :

2023-07-13T13:11:17Z E! [inputs.influxdb] Error in plugin: invalid character '#' looking for beginning of value
2023-07-13T13:11:17Z E! [telegraf] Error running agent: input plugins recorded 1 errors

Has anyone faced a smilar issue ?

Thanks

2

Please check out the plugin readme where it explains how to get metrics from InfluxDB version 2, namely you also need to set metric_version = 1

3

Hi, thx for the quick reply,

I did follow the plugin readmebut it’s unfortunately not working , here’s what my prometheus imput plugin conf looks like :

[[inputs.prometheus]]
#   ## An array of urls to scrape metrics from.
    urls = ["http://localhost:8086/metrics"]
#
#   ## Metric version controls the mapping from Prometheus metrics into Telegraf metrics.
#   ## See "Metric Format Configuration" in plugins/inputs/prometheus/README.md for details.
#   ## Valid options: 1, 2
    metric_version = 1
#
#   ## Url tag name (tag containing scrapped url. optional, default is "url")
#   # url_tag = "url"
#
#   ## Whether the timestamp of the scraped metrics will be ignored.
#   ## If set to true, the gather time will be used.
#   # ignore_timestamp = false
#
#   ## An array of Kubernetes services to scrape metrics from.
#   # kubernetes_services = ["http://my-service-dns.my-namespace:9100/metrics"]
#
#   ## Kubernetes config file to create client from.
#   # kube_config = "/path/to/kubernetes.config"
#
#   ## Scrape Kubernetes pods for the following prometheus annotations:
#   ## - prometheus.io/scrape: Enable scraping for this pod
#   ## - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
#   ##     set this to 'https' & most likely set the tls config.
#   ## - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
#   ## - prometheus.io/port: If port is not 9102 use this annotation
#   # monitor_kubernetes_pods = true
#
#   ## Get the list of pods to scrape with either the scope of
#   ## - cluster: the kubernetes watch api (default, no need to specify)
#   ## - node: the local cadvisor api; for scalability. Note that the config node_ip or the environment variable NODE_IP must be set to the host IP.
#   # pod_scrape_scope = "cluster"
#
#   ## Only for node scrape scope: node IP of the node that telegraf is running on.
#   ## Either this config or the environment variable NODE_IP must be set.
#   # node_ip = "10.180.1.1"
#
#   ## Only for node scrape scope: interval in seconds for how often to get updated pod list for scraping.
#   ## Default is 60 seconds.
#   # pod_scrape_interval = 60
#
#   ## Restricts Kubernetes monitoring to a single namespace
#   ##   ex: monitor_kubernetes_pods_namespace = "default"
#   # monitor_kubernetes_pods_namespace = ""
#   ## The name of the label for the pod that is being scraped.
#   ## Default is 'namespace' but this can conflict with metrics that have the label 'namespace'
#   # pod_namespace_label_name = "namespace"
#   # label selector to target pods which have the label
#   # kubernetes_label_selector = "env=dev,app=nginx"
#   # field selector to target pods
#   # eg. To scrape pods on a specific node
#   # kubernetes_field_selector = "spec.nodeName=$HOSTNAME"
#
#   # cache refresh interval to set the interval for re-sync of pods list.
#   # Default is 60 minutes.
#   # cache_refresh_interval = 60
#
#   ## Scrape Services available in Consul Catalog

Thanks

4

And you need to explain what that means with logs. I can’t help you without knowing what you are seeing :wink:

5

I mean logwise i don’t see much except :

tail /var/log/telegraf/telegraf.log
2023-07-13T13:49:38Z I! Loaded inputs: influxdb
2023-07-13T13:49:38Z I! Loaded aggregators:
2023-07-13T13:49:38Z I! Loaded processors:
2023-07-13T13:49:38Z I! Loaded secretstores:
2023-07-13T13:49:38Z W! Outputs are not used in testing mode!
2023-07-13T13:49:38Z I! Tags enabled: host=dev-monitoring-01
2023-07-13T13:49:38Z I! Found 9 secrets...
2023-07-13T13:49:38Z W! Insufficient lockable memory 64kb when 72kb is required. Please increase the limit for Telegraf in your Operating System!
2023-07-13T13:49:38Z E! [inputs.influxdb] Error in plugin: invalid character '#' looking for beginning of value
2023-07-13T13:49:38Z E! [telegraf] Error running agent: input plugins recorded 1 errors

In journalctl i’ve got :

telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.conf
telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.d/bind.conf
telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.d/elasticsearch.conf
telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.d/graylog.conf
telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.d/rsyslog.conf
telegraf[228556]: 2023-07-13T13:51:19Z I! Loading config: /etc/telegraf/telegraf.d/vsphere.conf
telegraf[228556]: 2023-07-13T13:51:19Z W! DeprecationWarning: Option "force_discover_on_init" of plugin "inputs.vsphere" deprecated since version 1.14.0 and will be removed in 2.0.0: option is ignored

Maybe i’m not searching at the right place i’m not sure haha, sorry :sweat_smile:

6

2023-07-13T13:49:38Z E! [inputs.influxdb] Error in plugin: invalid character ‘#’ looking for beginning of value

You are still using the influxdb input, you should be using the prometheus input. Maybe I didn’t make that as clear as I should have in my first message.

The influxdb input only works with influxdb v1.

Influxdb v2 outputs prometheus based metrics, so you can use the prometheus input plugin to scrape those metrics.

7

Yep that was the issue, i misread the readme thinking i needed to configure both :sweat_smile:

Thank you for your explanations and your time !

1 Like