Mesos metrics not showing up in prometheus

,
1

I am using prometheus output plugin to push metrics collected to a prometheus server.
When i test the config locally all the metrics are coming up as expected.

# telegraf --config mesos_inputs.conf --test
* Plugin: inputs.mesos, Collection 1
> mesos,role=slave,host=mesos-slave1.blabla.foo.com,server=localhost slave/gpus_percent=0,slave/executors_running=50,slave/container_launch_errors=545,slave/valid_framework_messages=0,slave/mem_revocable_used=0,containerizer/mesos/provisioner/remove_container_errors=0,containerizer/fetcher/task_fetches_succeeded=3638,slave/mem_total=159744,system/load_1min=5.58,containerizer/mesos/container_destroy_errors=0,system/mem_free_bytes=38771929088,slave/disk_used=2.583,system/mem_total_bytes=169008480256,slave/invalid_status_updates=0,slave/disk_total=506630,slave/disk_percent=0.00000509839527860569,slave/registered=1,containerizer/mesos/provisioner/bind/remove_rootfs_errors=0,slave/gpus_revocable_total=0,containerizer/fetcher/cache_size_total_bytes=2147483648,slave/recovery_errors=0,slave/cpus_used=23.2,slave/tasks_staging=1,slave/cpus_revocable_percent=0,slave/invalid_framework_messages=0,slave/tasks_starting=0,slave/executors_terminating=0,slave/tasks_finished=59,slave/executors_preempted=0,slave/tasks_gone=0,system/load_15min=8.35,slave/frameworks_active=1,slave/mem_used=159584,slave/cpus_revocable_total=0,slave/disk_revocable_total=0,slave/disk_revocable_used=0,slave/gpus_used=0,slave/disk_revocable_percent=0,slave/executors_registering=1,containerizer/fetcher/task_fetches_failed=0,slave/gpus_total=0,slave/tasks_lost=0,slave/executor_directory_max_allowed_age_secs=253071.036932095,slave/tasks_failed=2052,slave/mem_revocable_percent=0,slave/executors_terminated=5280,slave/mem_percent=0.998998397435897,slave/cpus_total=28,system/cpus_total=40,slave/valid_status_updates=6289,slave/gpus_revocable_percent=0,slave/cpus_percent=0.828571428571429,slave/uptime_secs=138506.359238912,system/load_5min=7.5,slave/tasks_running=50,slave/cpus_revocable_used=0,containerizer/mesos/filesystem/containers_new_rootfs=0,slave/tasks_killed=3169,slave/tasks_killing=0,containerizer/fetcher/cache_size_used_bytes=0,slave/gpus_revocable_used=0,slave/mem_revocable_total=0 1511937292000000000

But none of these metrics show up in the prometheus. Other metrics of the host like the system and docker metrics are coming up in prometheus.

What am I missing and do i need to add some translation anywhere?

2

I see the same behaviour for haproxy metrics.

> haproxy,server=/run/haproxy/socket,pxname=website-platform_home-page_10592,svname=slave4_bar_stg_foo_net_10_20_70_129_25336,type=server,host=int2.bar.stg.foo.net check_status="INI",hrsp_5xx=0i,check_fall=4i,wretr=0i,wredis=0i,weight=1i,chkfail=0i,smax=0i,downtime=0i,cli_abrt=0i,iid=1723i,hrsp_other=0i,check_rise=2i,dresp=0i,chkdown=0i,lastchg=7i,pid=1i,rtime=0i,qmax=0i,bout=0i,bck=0i,hrsp_4xx=0i,mode="http",sid=1i,lbtot=0i,hrsp_2xx=0i,check_health=2i,ctime=0i,addr="10.20.70.129:25336",eresp=0i,act=1i,hrsp_1xx=0i,lastsess=-1i,rate=0i,qcur=0i,stot=0i,bin=0i,econ=0i,srv_abrt=0i,qtime=0i,ttime=0i,scur=0i,status="UP 1/4",rate_max=0i,hrsp_3xx=0i 1511959694000000000
> haproxy,server=/run/haproxy/socket,pxname=website-platform_home-page_10592,svname=BACKEND,type=backend,host=int2.bar.stg.foo.net downtime=0i,srv_abrt=0i,rtime=0i,ttime=0i,slim=2000i,dreq=0i,lastchg=7i,hrsp_1xx=0i,hrsp_other=0i,req_tot=0i,ctime=0i,status="UP",lbtot=0i,rate=0i,qmax=0i,hrsp_5xx=0i,weight=1i,bck=0i,comp_in=0i,dresp=0i,eresp=0i,iid=1723i,lastsess=-1i,qcur=0i,smax=0i,bout=0i,wredis=0i,act=1i,chkdown=0i,pid=1i,comp_out=0i,scur=0i,stot=0i,bin=0i,comp_byp=0i,qtime=0i,mode="http",wretr=0i,sid=0i,hrsp_4xx=0i,hrsp_3xx=0i,cli_abrt=0i,comp_rsp=0i,econ=0i,rate_max=0i,hrsp_2xx=0i 1511959694000000000

Any pointers as to why prometheus is not able show the metrics.

3

ah…my mistake…the metrics come with a _ …so mesos/cpus_total comes as mesos_slave_cpus_total

but with the same logic, does not seem to work with haproxy metrics.

4

Using your haproxy metrics I see items like this:

haproxy_weight{addr="",check_status="",host="int2.bar.stg.foo.net",mode="http",pxname="website-platform_home-page_10592",server="/run/haproxy/socket",status="UP",svname="BACKEND",type="backend"} 1
haproxy_weight{addr="10.20.70.129:25336",check_status="INI",host="int2.bar.stg.foo.net",mode="http",pxname="website-platform_home-page_10592",server="/run/haproxy/socket",status="UP 1/4",svname="slave4_bar_stg_foo_net_10_20_70_129_25336",type="server"} 1

Keep in mind that since prometheus has no string values the string fields are not output, for example the status="UP 1/4" field.

5

Thank you Daniel for the info.
I will have to switch to haproxy exporter instead to get the metrics. I was planing to see if I could have one agent pull all the metrics. This seems to be a limitation when using just prometheus as the output plugin.

6

There are still no haproxy* metrics at all?

7

My apologies. This is due to a misconfiguration.
I updated the config and I am able to see the metrics now.