Whats The Best Way To Get Prometheus Data

Store
,
1

Hi,

I’m collecting prometheus data via telegraf prometheus plugin. It creates new series for each row and field (gauge). It makes db cluttered and increase cardinality. As far as I know series has the highest cost between other types. (I think I heard this in an influxdb talk @youtube)

How do you collect prometheus data?

I’m collecting data from jira - prometheus exporter. Here is an example:

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 61506.796875
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.578588827776E9
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
jvm_buffer_pool_used_bytes{pool="direct",} 1.1288304E7
jvm_buffer_pool_used_bytes{pool="mapped",} 1.36569941E8
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
jvm_buffer_pool_capacity_bytes{pool="direct",} 1.1288304E7
jvm_buffer_pool_capacity_bytes{pool="mapped",} 1.36569941E8
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
jvm_buffer_pool_used_buffers{pool="direct",} 83.0
jvm_buffer_pool_used_buffers{pool="mapped",} 311.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 273.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 206.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 314.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 324366.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
2

In Telegraf 1.13, I added a new output format for the prometheus input which addresses some of these issues. Check the plugin documentation for some example output and set metric_version = 2 to enable the new format. I hope this format will work better for you, let me know how it goes.

[[inputs.prometheus]]
  metric_version = 2
  ## ... other options ...
1 Like
3

Now that I’ve updated telegraf and set metric_version = 2. All looks good so far.

I want to ask a question about tags. I’ve nearly 1K different tags for a tag, these are coming from the data I’m collecting via inputs.promethues. I need the data but I don’t need these tags. Are there any simple way to drop tags but keep the data?

4

You can use metric filtering, in particular tagexclude, to remove tags. Just remember that tags also form the identity of the series so don’t remove any tags that you need for uniqueness.

1 Like