tl;dr: There is a telegraf documentation gap. There is no relationship made from telegraf keys to kernel value names.
This horse isn’t quite dead yet.
Indeed all measurement are in milliseconds.
After reading the linux document stat.txt, it gives more details about what the counters actually are and how to interpret them. It pertains to /sys/block/device/stat, but as stated previously - “don’t break user space” - these are the same values from /proc/diskstats anyway.
My misunderstanding comes from mapping of telegraph “key” to the kernel counters - and fundamentally understanding what the kernel counter actually means (due to this disconnect)
For example telegraf “read_time” actually maps to (well - I think it does…) - excerpt from stat.txt
read ticks, write ticks
These values count the number of milliseconds that I/O requests have
waited on this block device. If there are multiple I/O requests waiting,
these values will increase at a rate greater than 1000/second; for
example, if 60 read requests wait for an average of 30 ms, the read_ticks
field will increase by 60*30 = 1800.
This means the read_tick (aka read_time) is inclusive of the items on the disk I/O queue. That’s why the value appears to be very high. In order to understand a single I/O service time one would need the queue length as well. However - telegraf doesn’t seem to expose the kernel value for “in_flight” that contains the number of I/O’s currently in flight.I believe this to be the queue depth. (if it does - I don’t know what it’s called - see, there is that documentation problem I’m talking about…)
Being an average queue length and not an instantaneous queue length - we need to consider utilizing Little’s Law from Queuing Theory, the result is only valid when the system is in equilibrium state - when average and instantaneous queue lengths are equal.
Ultimately - this is a documentation shortfall of the diskio plugin in my opinion, which should clearly state the “mapping” of telegraf keys to the kernel counters. This would alleviate the confusion.