Telegraf-Interval issue

Telegraf
25

@Hipska
Please let me know if you are able to see the above mentioned configuration. If not can you please guide me alternate ways to share the details

26

Again, please put it in code blocks so we can read it.

# so that comments actually become comments
[[and blocks look like actual blocks]]
    indentations = "also become clear"
27 
[global_tags]
server = “SBC”
dc = “us-east-1” # will tag all metrics with dc=us-east-1
rack = “1a”
Environment variables can be used as tags, and throughout the config file
#user = “$USER”

Configuration for telegraf agent^M
[agent]^M

Default data collection interval for all inputs^M
interval = “10s”^M

Rounds collection interval to ‘interval’^M
ie, if interval=“10s” then always collect on :00, :10, :20, etc.^M
round_interval = true^M
^M

Telegraf will send metrics to outputs in batches of at most^M
metric_batch_size metrics.^M
This controls the size of writes that Telegraf sends to output plugins.^M
metric_batch_size = 1000^M
^M

Maximum number of unwritten metrics per output. Increasing this value^M
allows for longer periods of output downtime without dropping metrics at the^M
cost of higher maximum memory usage.^M
metric_buffer_limit = 10000^M
^M

Collection jitter is used to jitter the collection by a random amount.^M
Each plugin will sleep for a random time within jitter before collecting.^M
This can be used to avoid many plugins querying things like sysfs at the^M
same time, which can have a measurable effect on the system.^M
collection_jitter = “0s”^M
^M

Collection offset is used to shift the collection by the given amount.^M
This can be be used to avoid many plugins querying constraint devices^M
at the same time by manually scheduling them in time.^M
collection_offset = “0s”^M
^M

Default flushing interval for all outputs. Maximum flush_interval will be^M
flush_interval + flush_jitter^M
flush_interval = “10s”^M

Jitter the flush interval by a random amount. This is primarily to avoid^M
large write spikes for users running a large number of telegraf instances.^M
ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s^M
flush_jitter = “0s”^M

# Retrieves SNMP values from remote agents
[[inputs.snmp]]
## Agent addresses to retrieve values from.
## format: agents = [“scheme://:”]
## scheme: optional, either udp, udp4, udp6, tcp, tcp4, tcp6.
## default is udp
## port: optional
## example: agents = [“udp://127.0.0.1:161”]
## agents = [“tcp://127.0.0.1:161”]
## agents = [“udp4://v4only-snmp-agent”]
agents = [“udp://10.XX.XX.XX:161”]

## Timeout for each request.
timeout = “25s”

## SNMP version; can be 1, 2, or 3.
version = 2

## Path to mib files
## Used by the gosmi translator.
## To add paths when translating with netsnmp, use the MIBDIRS environment variable
path = [“/usr/share/snmp/mibs”]

## SNMP community string.
community = “sbcgrafana”

## Agent host tag
agent_host_tag = “agent_host”

## Number of retries to attempt.
retries = 1
interval = “1m”

## The GETBULK max-repetitions parameter.
max_repetitions = 10

[[inputs.snmp.table]]
oid = “APUSBCSYS-MIB::apUsbcSysCpuTable”
name = “SBC-CPUutilization”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IF-MIB::ifTable”
name = “interface”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IP-FORWARD-MIB::ipForwardTable”
name = “SBC-InterfaceForwardTable”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IP-MIB::ipSystemStatsTable”
name = “SBC-IPSystemstatus”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IP-MIB::ipIfStatsTable”
name = “SBC-TrafficStats”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IP-MIB::ipAddressTable”
name = “SBC-AddressTable”
index_as_tag = true

[[inputs.snmp.table]]
oid = “IP-MIB::ipNetToPhysicalTable”
name = “SBC-PhysicalStatus”
index_as_tag = true

# Configuration for sending metrics to InfluxDB 2.0
[[outputs.influxdb_v2]]

## The URLs of the InfluxDB cluster nodes.
## Multiple URLs can be specified for a single cluster, only ONE of the
## urls will be written to each interval.
## ex: urls = [“https://us-west-2-1.aws.cloud2.influxdata.com”]
urls = [“http://10.XX.XX.XX:8086”]

## Token for authentication.
token = “XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxx”

## Organization is the name of the organization you wish to write to.
organization = “ProdInfluxDB”

## Destination bucket to write into.
bucket = “Prod-SNMP”

## The value of this tag will be used to determine the bucket. If this
## tag is not set the ‘bucket’ option is used as the default.
# bucket_tag = “”
## If true, the bucket tag will not be added to the metric.
# exclude_bucket_tag = false
## Timeout for HTTP messages.
# timeout = “5s”
28

@Hipska please check now, I appreciate your support on this. Thanks

29

Thanks, that is indeed better to understand, but there are still problems with your config file:

  1. There are a lot of ^M characters, those are not valid, please remove.
  2. There are incorrect quote signs like for example “10s” please replace by ".
30

The quote is correct in my config file, in the code blocks is looks different. Also I tried to remove ^M but issue remains same.

Even though interval has mentioned 1m in the config, but data is showing every 5 min interval, if I add more agents it shows 10-20 min interval data.

31

@Hipska
I changed the agent parameter as per below, I see some improvement and currently it shows 5-6 min interval data even though Interval data set for 1m.
metric_batch_size = 15000
metric_buffer_limit = 80000
flush_interval = “40s”

32

Okay, I also see that you are collecting 7 full tables, could you remove them and re-add one by one to see which one might introducing this delay? I know some snmp devices are having issues to quickly return all table data.

It might be interesting to specify specifically which fields and tags you want from the slow table(s) in order to reduce the load on the device.

33

Yes I am reducing the number of tables also removed the input for local system. I will let you know in next few days. Thanks

34

@Hipska I am able to fix the issue by making changes in the configuration file as per below.

  1. I have added multiple telegraf configuration file and each file has 10+ agents
  2. Removed the RFCs from input.table configuration (example rfc4022), which was utilizing more data. And added only necessary inputs field instead of adding entire RFC table in the configuration.
  3. Increased the metric_batch and metric_butter limit based on the utilization.
1 Like
35

anybody know how to change flush interval when using [inputs.snmp]?

I modified flush_interval at /etc/telegraf/telegraf.conf, but it didn’t make difference.
[[inputs.snmp.table]]
name = “docsRphyRpdDevL2tpSessionStatsTable”
oid = “DOCS-RPHY-MIB::docsRphyRpdDevL2tpSessionStatsTable”
index_as_tag = true
flush_interval=“180s”------>add this line, doesn’t make difference.

it read configuration file from below location, but how do i modify the config?
where is the configure file located?

cccc@sqa-tb1:/usr/lib/influxdb/scripts$ telegraf --config http://10.201.15.102:8086/api/v2/telegrafs/0c943b69bbb51000

2024-02-13T06:12:01Z I! Loading config: http://10.201.15.102:8086/api/v2/telegrafs/0c943b69bbb51000
2024-02-13T06:12:01Z I! Starting Telegraf 1.29.4 brought to you by InfluxData the makers of InfluxDB
2024-02-13T06:12:01Z I! Available plugins: 241 inputs, 9 aggregators, 30 processors, 24 parsers, 60 outputs, 6 secret-stores
2024-02-13T06:12:01Z I! Loaded inputs: snmp
2024-02-13T06:12:01Z I! Loaded aggregators:
2024-02-13T06:12:01Z I! Loaded processors:
2024-02-13T06:12:01Z I! Loaded secretstores:
2024-02-13T06:12:01Z I! Loaded outputs: influxdb_v2
2024-02-13T06:12:01Z I! Tags enabled: host=sqa-tb1
2024-02-13T06:12:01Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:“sqa-tb1”, Flush Interval:10s

36

The flush_interval is a setting for output plugins, not input plugins.