I’ve encountered quite a strange problem with InfluxDB 1.5.2 (inmem indexes), whereby data points are correctly being recorded, and can be queried used a WHERE-clause, however the “host” tag values are apparently not being indexed.
The following query successfully retrieves points:
> select * from infiniband_ib_data where host =~ /^gw305a/ and time > now() - 5m
name: infiniband_ib_data
time host plugin plugin_instance rcv type type_instance xmit
---- ---- ------ --------------- --- ---- ------------- ----
1527070893000000000 gw305a-2.fra infiniband mlx4_0 312183966 ib_data 1 312185110
1527070953000000000 gw305a-2.fra infiniband mlx4_0 312227526 ib_data 1 312228670
1527071013000000000 gw305a-2.fra infiniband mlx4_0 312271086 ib_data 1 312272230
1527071073000000000 gw305a-2.fra infiniband mlx4_0 312314718 ib_data 1 312315862
1527071133000000000 gw305a-2.fra infiniband mlx4_0 ib_data 1 312359422
However, the following produces no results:
> show tag values from infiniband_ib_data with key = host where host =~ /^gw305a/
Other hosts are also successfully logging data, and they do show up in the “show tag values” query. I also tried reconfiguring the sender (collectd, with custom write_influxdb output plugin) to write to different InfluxDB instance, where it worked as expected. This seems to indicate something fishy with our main InfluxDB instance.
I also tried deleting individual series, and even the entire measurement, expecting InfluxDB to index the tag values when it saw new data. Sadly this also did not work - data is recorded, but the host tag value for that specific host is not indexed.
Even influx_inspect does not indicate any indexes with that particular host name, so I’m baffled as to how the WHERE-query is even able to work.
Since our Grafana dashboards have a “host” drop-down list, which is populated by a “show tag values” query, it means this host is not selectable in the dashboard.
I’d like to avoid the extreme solution of dropping and restoring the entire DB, so any pointers would be gratefully appreciated.
