Debugging sending test data to influx via telegraf

Store
,
1

I’m trying to debug why I don’t see some data I’m sending to influxdb.

On myhost.mydomain.com, I’ve installed telegraf (1.3.3). The telegraf config says to listen to udp and to send via https:

[[inputs.socket_listener]]
  service_address = "udp4://:8095"
  data_format = "influx"

[[outputs.influxdb]]
  urls = ["https://influxdb.mydomain.com/"]
  database = "telegraf" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"
  username = "secret"
  password = "more-secret"
  user_agent = "telegraf"

I also say to send (e.g.) CPU info:

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false

Now on myhost.mydomain.com I try to send some test data:

echo 'weather,test=test,tester=jeff humidity=50,location=Nantes,temperature=17 1501160222000000000' | \
nc -u -C  localhost 8095

I’ve confirmed that telegraf is listening for udp on port 8095:

[T] jeff@myhost:telegraf $ sudo netstat -tunapl | grep 8095
udp        0      0 0.0.0.0:8095            0.0.0.0:*                           4975/telegraf   
[T] jeff@myhost:telegraf $

On influxdb.mydomain.com, I can see in the nginx logs that requests are being presented, though I don’t see how to know what data is being transferred:

[T] jeff@loire1:influxdb $ sudo grep x.y.z.w /var/log/nginx/access.log | tail -2
x.y.z.w - telegraf [28/Jul/2017:08:43:29 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 204 0 "-" "telegraf"
x.y.z.w - telegraf [28/Jul/2017:08:43:39 +0000] "POST /write?consistency=any&db=telegraf HTTP/1.1" 204 0 "-" "telegraf"
[T] jeff@loire1:influxdb $

The influxdb log directory (/var/log/influxdb/) is empty. Queries about the weather do not return data about myhost.mydomain.com. Queries about CPU do.

> select * from cpu where host='myhost' limit 5
...stuff...
>

> select * from weather where host='myhost' limit 5
>

So it appears that telegraf is able to send data to influxdb, but when I send the data to telegraf via udp, it’s not making it through.

Any debugging suggestions?

2

I would turn on a file output sending to stdout, then you can easily observe if the weather data is being accepted.

1 Like
3

Excellent suggestion, that is what I was looking for and hadn’t grrokked the significance of.

So that confirms that telegraf is not seeing the data I send. And yet netstat says telegraf is listening:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 *:8095                  *:*                                 11924/telegraf

and using another udp port and nc -l -u, my test line says it is being sent.

If I simplify my metric to send very minimal info, I still don’t see it:

[T] jeff@seine2:telegraf $ echo 'weather humidity=50 1501266203000000000' | \
    nc -u -C  127.0.0.1 8095
[T] jeff@seine2:telegraf $ grep weather /tmp/telegraf.out
1,[T] jeff@seine2:telegraf $

Is my input malformed without telegraf signalling an error? (I see the telegraf-gathered metrics like CPU in the file.)

4

Ah, so the answer is “yes, my input can be malformed without telegraf signalling an error”.

I was missing a comma after “weather”. :frowning:

(To be complete, in my initial test (original question) I sent my UDP packets to localhost rather than 127.0.0.1, and it turned out that while localhost is noted in /etc/hosts, /etc/resolv.conf does not specify file. And we get into subtleties that I only partly master about how name resolution works along different paths.)

5

If Telegraf is dropping the point without an error then it is a bug, and should be reported on the github page.

With your original metric:

echo 'weather,test=test,tester=jeff humidity=50,location=Nantes,temperature=17 1501160222000000000' | nc -u localhost 8094

I get the error:

Error in plugin [inputs.socket_listener]: unable to parse incoming packet:  invalid number

The problem here is that the location field needs to be quoted. Note that I removed the -C option to netcat, since we want unix line endings.

2 Likes