Influx Data not showing for last 5 min, but for last 15 min shows

InfluxDB 1
, ,
1

I’m using “InfluxDB shell version: 1.7.5” and “Telegraf v1.5.0” and transferring data from app-servers to InfluxDB through telegraf.

I am getting no data if I am using time range time >= now() -5m . But if I am using time >= now() -15m in the query data is coming. This means that the last 15 min or above time range showing data in the result of a simple select query on influx db but less than 5min it is showing no data (last 15 min data contains last 5 min data).

Already tried executing the query directly on influxdb rather than via grafana, Same observation. Issue seems to be with influx**,** Can somebody help me to resolve this issue?

Screenshot 2022-07-11 at 8.02.46 PM
Screenshot 2022-07-11 at 8.02.46 PM1920×998 90.4 KB

2

I doubt the DB is the issue… I’d start investigating the data gathering:

  • How often are the data gathered and flushed by Telegraf?
  • Which kind of data are we talking about? is it a continuous stream like CPU monitoring or something event-related like error tracking?
  • Are you using CQs (continuous queries)?
  • Are you reading from the correct Retention Policy?

Also, what do you get when you actually “find” data?

3

Hi Thanks for the reply.

  1. Below is the telegraf configuration file for data gathered and flushed by telegraf.
[global_tags]
  dc = "ap-south-1" # will tag all metrics with dc=us-east-1
  host  = "$HOSTNAME"

[agent]
  interval = "1m"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "30s"
  flush_jitter = "0s"
  precision = ""
  logfile = "/var/log/telegraf/telegraf.log"
  hostname = ""
  omit_hostname = false

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

[[outputs.influxdb]]
  urls = ["http://xxx:8086"] # required
  database = "stage-db-xxx" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"
  content_encoding = "gzip"

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false

[[inputs.disk]]
  mount_points = ["/","/data"]
  fielddrop=["used","inodes_used"]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

[[inputs.mem]]
  fielddrop=["active","buffered","cached","free","inactive","used","used_percent"]

[[inputs.processes]]
  # no configuration

[[inputs.swap]]
  fielddrop=["free","total"]

[[inputs.system]]
  fielddrop=["n_users","uptime_format"]

[[inputs.net_response]]
  protocol = "tcp"
  address = ":22"
  timeout = "5s"
  read_timeout = "5s"

[[inputs.net_response]]
  protocol = "tcp"
  address = ":80"
  timeout = "5s"
  read_timeout = "5s"

[[inputs.net_response]]
  protocol = "tcp"
  address = ":3306"
  timeout = "5s"
  read_timeout = "5s"

[[inputs.mysql]]
  servers = ["telegraf:xyz/?tls=false"]
  gather_slave_status = true
  1. General metrics we are collecting like CPU, MEM etc.
  2. I don’t think so, can you please help me with what do you mean by CQ’s.
  3. Can you share with me the query to get this retention policy, So that I can share with you the O/P.

Note: for last 15min the influx query O/P is

ubuntu@ip-xxx:~$ curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=xxx-master-prod" --data-urlencode "q=SELECT \"used_percent\" FROM \"disk\" WHERE \"host\"='ip-xxxxap-south-1.compute.internal' AND time >= now() -15m GROUP BY \"path\""
{
    "results": [
        {
            "statement_id": 0,
            "series": [
                {
                    "name": "disk",
                    "tags": {
                        "path": "/"
                    },
                    "columns": [
                        "time",
                        "used_percent"
                    ],
                    "values": [
                        [
                            "2022-07-12T13:37:28Z",
                            82.03197951483892
                        ],
                        [
                            "2022-07-12T13:37:28Z",
                            82.03197951483892
                        ],
                        [
                            "2022-07-12T13:38:29Z",
                            82.03255640368774
                        ],
                        [
                            "2022-07-12T13:38:29Z",
                            82.03255640368774
                        ],
                        [
                            "2022-07-12T13:39:27Z",
                            82.03955860352
                        ],
                        [
                            "2022-07-12T13:39:27Z",
                            82.03955860352
                        ],
                        [
                            "2022-07-12T13:40:28Z",
                            82.0397335055665
                        ],
                        [
                            "2022-07-12T13:40:28Z",
                            82.0397335055665
                        ],
                        [
                            "2022-07-12T13:41:20Z",
                            82.03991740463185
                        ],
                        [
                            "2022-07-12T13:41:20Z",
                            82.03991740463185
                        ]
                    ]
                }
            ]
        }
    ]
}

And for last 5min influx query O/P is:

ubuntu@ip-xxx:~$ curl -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=xxx-master-prod" --data-urlencode "q=SELECT \"used_percent\" FROM \"disk\" WHERE \"host\"='ip-xxxap-south-1.compute.internal' AND time >= now() -5m GROUP BY \"path\""
{
    "results": [
        {
            "statement_id": 0
        }
    ]
}

** Have observed that the last 15min query o/p contains the data of last 4-5min as well.

4

I’ll add some info about default values not specified in what you shared, just to have the full picture.

  • The telegraf flush_interval = "10s" by default
  • You are writing to the DB without specifying any Retention Policy, meaning the default one will be used.

To check the current RPs run SHOW RETENTION POLICIES ON _DbName_, you will also see which one is the default one
To check for CQs run SHOW CONTINUOUS QUERIES ON _DbName_

Check the telegraf log for errors as that might be an issue as well, given your current configuration you should also see the number of points sent to influxDB every time a flush happens.

5 
> SHOW RETENTION POLICIES ON "xxx-master-prod"
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true

CQs is

name: xxx-master-prod
name query
---- -----

There are no error logs at telegraf side, I believe telegraf is working fine since I can see the 5 min data as well when do the query for the last 15 min data.

6

I think you are right, the only point left is about timezone or time in general… Grafana does a pretty good job at querying data based on your timestamp, but I’m wondering… is the clock on the server ok? or is it somehow “imprecise” (I’ve seen servers whose clock was off by almost an hour…)
Also, remember that Telegraf and InfluxDB timestamps are always UTC, if you are in a different timezone apply the proper offset or timezone (even tho I doubt this is the case)

7

Hi Dear it works for me time was not correct on my influx DB server it was 10 min ahead of than actual UTC time.

Thank you so much for your efforts :slight_smile:

It would be nice if you can suggest some retention approach on older than 15 days data on influx db.

8

Your request is too broad and vague, I suggest you start by having a look here:

once you have clearer ideas create a new question and I’ll be glad to help