400 Bad request - http://<URL>:8086/write?db=service_metrics for telegraf in Datadog APM

,
1

Hello,

We got HTTP response status code 400 bad request for “http://:8086/write?db=service_metrics” in Datadog APM for telegraf.

Telegraf agents are sending the data via wavefront proxy. OS version is CentOS Linux release 7.9.2009 (Core). Telegraf version is 1.10.4 and 1.18.0.

Below error is getting on /var/log/messages

E! Error: parsing sample rate, , it must be in format like: @0.1, @0.5, etc. Ignoring sample rate for line: datadog.trace_agent.events.max_eps.sample_rate:1|g|#version:7.44.1
E! Error: splitting '|', Unable to parse metric: datadog.trace_agent.events.max_eps.sample_rate:1|g|#version:7.44.1

/etc/telegraf/telegraf.conf

[global_tags]
  datacenter = "XXX"
  environment = "XXX"

[agent]
  interval = "180s"
  round_interval = true

  metric_batch_size = 1000
  metric_buffer_limit = 10000

  collection_jitter = "0s"
  flush_interval = "30s"
  flush_jitter = "5s"

  precision = ""


  debug = false
  quiet = true
  logfile = ""

  hostname = "XXXXXXX"
  omit_hostname = false


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

[[outputs.wavefront]]
  host = "XXXXXX"
  port = 2878
  metric_separator = "."
  source_override = ["Node"]
  convert_paths = true
  use_regex = false

###############################################################################
#                            PROCESSOR PLUGINS                                #
###############################################################################

[[processors.strings]]
  [[processors.strings.replace]]
    tag = "Node"
    old = "_"
    new = "."


###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  fielddrop = ["usage_guest*"]


[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs", "rootfs"]
  #fielddrop = ["inodes_*"]


[[inputs.diskio]]
  fielddrop = ["weighted_*", "iops_*"]

[[inputs.kernel]]
  fielddrop = ["boot_time"]

[[inputs.mem]]
  # no configuration

[[inputs.processes]]
  # no configuration

[[inputs.swap]]
  # no configuration

[[inputs.system]]
  # no configuration
  fielddrop = ["n_*"]


[[inputs.net]]
  fielddrop = ["icmp*", "ip_*", "tcp*", "udp*"]

[[inputs.system]]
  # no configuration


[[inputs.internal]]
#   ## If true, collect telegraf memory stats.
    collect_memstats = true
    name_prefix = "telegraf."


[[inputs.chrony]]
#   ## If false, set the -n ntpq flag. Can reduce metric gather time.
    dns_lookup = true
    fieldpass = ["last_offset"]

[[inputs.ping]]
  urls = ["XXX"]

/etc/telegraf/telegraf.d/telegraf-application-metrics.conf

[[inputs.influxdb_listener]]
  service_address = ":8086"
  read_timeout = "10s"
  write_timeout = "10s"
  name_prefix = "application."


[[inputs.statsd]]
  protocol = "udp"
  max_tcp_connections = 250
  tcp_keep_alive = false
  service_address = ":8125"
  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true
  metric_separator = "_"
  allowed_pending_messages = 10000
  percentile_limit = 1000
  name_prefix = "application."

/etc/telegraf/telegraf.d/telegraf-system-listener.conf

[[inputs.influxdb_listener]]
  service_address = ":8087"
  read_timeout = "10s"
  write_timeout = "10s"

Please help us to fix the issue.

2

Hello @Mithun_Thomas,
Welcome!
I found the following issue that looks related to your problem:

it appears that these stats examples are using DataDog’s datagram format:

<METRIC_NAME>:<VALUE>|<TYPE>|@<SAMPLE_RATE>|#<TAG_KEY_1>:<TAG_VALUE_1>,<TAG_2>

Therefore, your Telegraf config will need to include the datadog_extensions = true option.

Let me know if that helps : )

1 Like
3

Hi @Anaisdg

In which section of the telegraf config file we need to add datadog_extensions = true option.

We have configured telegraf in below files:
/etc/telegraf/telegraf.d/telegraf-application-metrics.conf
/etc/telegraf/telegraf.d/telegraf-system-listener.conf
/etc/telegraf/telegraf.conf

Which file we need to add that?

In addition we have same configuration in other environments, but the issue is not there. Issue is with a particular environment only.

Thanks