InfluxDb usage statistics

Store
,
1

Hello

First : fantastic suits of tools !!! i love so much TICS.

I would like to know if in influxDB there is a way to get a quick shot of the size usage of hosts/tags/data retention?

my scenario is for example, to see how much size in the DB an host is occupying or a specific tag.
for data retention this allow to anticipate on disk space, etc…

other example is to get per host the space disk in the influxb, like that we can anticipate space growing when adding more hosts.

hope i was enough clear :slight_smile:

Tonic8

2

@tonic8 InfluxDB exposes a bunch of this information thought the _internal database. There is also a telegraf plugin that exposes more of this information.

I dont know if you are going to be able to get a clean number for per host, but I should give you some better numbers on usage. Hope this helps!

3

Hello jackzampolin

Thanks for the prompt response, i will check that part, either the internal and the plugin.

At least it let me have some infos.

:slight_smile:

regards
Tonic8

4

Hello @tonic8,

Where you able to find your answers? If so, can you please share what you did for the community?

If not, I just found this:

$ influx
Connected to http://localhost:8086 version 1.5.1
InfluxDB shell version: 1.5.1
> use _internal
Using database _internal
> SELECT last("diskBytes") FROM "monitor"."shard" WHERE ("database" =~/internal/) AND time >= now() -1h GROUP BY "database", "path" fill(null)
name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/205
time                last
----                ----
1523316450000000000 3739493

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/208
time                last
----                ----
1523316450000000000 3741307

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/212
time                last
----                ----
1523316450000000000 3685745

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/216
time                last
----                ----
1523316450000000000 3799445

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/219
time                last
----                ----
1523316450000000000 3839261

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/222
time                last
----                ----
1523316450000000000 3831156

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/226
time                last
----                ----
1523316450000000000 3922560

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/229
time                last
----                ----
1523316450000000000 4558010

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/232
time                last
----                ----
1523316450000000000 3880661

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/235
time                last
----                ----
1523316450000000000 3854180

name: shard
tags: database=_internal, path=/var/lib/influxdb/data/_internal/monitor/240
time                last
----                ----
1523316450000000000 26711342

Which matches pretty closely to:

/var/lib/influxdb/data/_internal/monitor # ls
205  208  212  216  219  222  226  229  232  235  240

/var/lib/influxdb/data/_internal/monitor # du -s *
3676    205
3680    208
3624    212
3736    216
3776    219
3768    222
3856    226
4480    229
3816    232
3788    235
9616    240

The reason the last shard will look different is the shard size in Influx reports back the shard plus the WAL size… it is a InfluxDB’ism. I’m not the best at InfluxQL to join and figure out how to minus the WAL size, maybe someone else can do that. :slight_smile:

Remember if a new shard was created, the values in the last hour could appear off. If a new shard creation happened and an old shard dropped in the time frame, it’ll still show old shards.

Inside of Grafana, you can make a graph pretty easily to watch these values.

SELECT "diskBytes" FROM "monitor"."shard" WHERE ("database" = '_internal') AND $timeFilter GROUP BY "path"

You can swap out the database (ie. “_internal”) for your alternate database name.

Also, you could monitor the size on disk on the host… for example if the data is being saved to a /var/lib/influxdb/ directory, this dir could be monitored.

Good to check out is:

$ influxd config

within influx
> SHOW STATS
and
> SHOW DIAGNOSTICS

Hope this helps… someone.

1 Like
5

HI Peter,

The answer for this question was very clear and gave steps how to resolve this issue easily.
I am also facing similar type of issue but not same. Hope you can help in resolving my issue, Please find details below…

We do performance testing using Gatling open source tool. To get graphical interpretation of test we use Grafana and as an input to Grafana we use Influx DB. Gatling, Grafana and Influx DB are installed in different AWS instances.

Now my problem is how to send performance metrics like Memory, Disk space and CPU Utilization from AWS instance (Where gatling is installed and test is running) to Influx DB?

To enable monitoring metrics in AWS machine(Where gatling is installed and test is running) i used below commands

sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https
sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64

Can you please help me in resolving this issue?

Thanks,
Swetha. G

6

Need to run Telegraf service locally in your AWS instance. It will collect the system level data and sent it to Influx that you specified in Telegraf’s conf file.

7

Thanks for sharing it.
I was struggling to find a way to get a DB size and was confused why the query to shards always returned extremely small value. Turned out it was always returning the size of the single shard, not sum of all of them :roll_eyes:

So, a simple query to get the size of all shards is:

SELECT sum(DiskSizeMB) from (SELECT (last("diskBytes") / 1024 /1024) AS "DiskSizeMB" FROM "_internal"."monitor"."shard" WHERE ("database" =~/internal/) AND time >= now() -10m GROUP BY "database", "path" FILL(null))

But I have found it is pretty hard to get the sum working on the Chronograf UI to get nice chart of the size in time and I am not sure I am doing it the right way, but I wanted to share it anyway as it could help some other desperate soul:

SELECT sum(disk_size) from (SELECT mean("diskBytes") AS "disk_size" FROM "_internal"."monitor"."shard" WHERE "database"='_internal' AND time > :dashboardTime: AND time < :upperDashboardTime: GROUP BY time(:interval:), "database", "path" FILL(null)) GROUP BY time(:interval:)

The trouble is, that summing those shards tells me my DB has size of 570MB, but executing du like below, tells me the DB has 715MB with 683MB in default RP folder and rest in _series folder.

du -shcx influxdb/data/DB_NAME/*

So the DB is not measuring everything it seems…

1 Like
8

The “_internal” database doesn’t seem to exist in my InfluxDB server.

Oh well.