Map function triggering panic index out of range [1] with length 1

InfluxDB 2
1

I have two buckets that log data from systems metrics and metrics_pg using telegraf to with the input modules cpu, mem, system, disk and others enabled.

I run a query using the python API every 10 minutes which computes the percent used memory per host and send it to a slack channel.

mem_available=from(bucket: "metrics")
    |> range(start: -5m)
    |> filter(fn: (r) => r["_measurement"] == "mem")
    |> filter(fn: (r) => r["_field"] == "available")
    |> aggregateWindow(every: 5m, fn: last, createEmpty: false)
mem_total=from(bucket: "metrics")
    |> range(start: -5m)
    |> filter(fn: (r) => r["_measurement"] == "mem")
    |> filter(fn: (r) => r["_field"] == "total")
    |> aggregateWindow(every: 5m, fn: last, createEmpty: false)
percent_used=join(
        on: ["_time", "_stop", "_start", "host"],
        tables: {mem_total, mem_available}
    )
    |> map(fn: (r) => (
        {
            _time: r._time,
            _value: (r._value_mem_total - r._value_mem_available) * 100 /r._value_mem_total,
            host: r.host
        }
    ))
percent_used

This has been working fine since it was created a month ago but as of yesterday the influx query for the metrics bucket fails, however the same query changing the bucket to metrics_pg works fine.

Running this query in the influxdb2 web UI returns the error

panic: runtime error: index out of range [1] with length 1

The influxdb log has the following error for the query

ts=2020-12-22T05:01:20.615541Z lvl=info msg="Dispatcher panic" log_id=0R6Twn5W000 service=storage-reads component=dispatcher error="panic: runtime error: index out of range [1] with length 1" stacktrace="goroutine 58898134 [running]:
runtime/debug.Stack(0xc0083d1740, 0x39b7c00, 0x271f737)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/influxdata/flux/execute.(*poolDispatcher).Start.func1.1(0xc0083d17a0)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/dispatcher.go:81 +0x1e5
panic(0x259cc60, 0xc013e58980)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/apache/arrow/go/arrow/bitutil.BitIsNotSet(...)
    /go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20200923215132-ac86123a3f01/bitutil/bitutil.go:48
github.com/apache/arrow/go/arrow/array.(*array).IsNull(...)
    /go/pkg/mod/github.com/apache/arrow/go/arrow@v0.0.0-20200923215132-ac86123a3f01/array/array.go:112
github.com/influxdata/flux/execute.ValueForRow(0x3a2c880, 0xc0145d25a0, 0x8, 0x1, 0xc00dc8a2d0, 0xc00dc8a2d0)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/table.go:491 +0xcf1
github.com/influxdata/flux/execute.(*rowFn).eval(0xc016f6ab40, 0x3a07c60, 0xc013b36930, 0x8, 0x3a2c880, 0xc0145d25a0, 0x0, 0x3a3db01, 0x2, 0x3, ...)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/row_fn.go:190 +0xbf
github.com/influxdata/flux/execute.(*RowMapPreparedFn).Eval(0xc016f6ab40, 0x3a07c60, 0xc013b36930, 0x8, 0x3a2c880, 0xc0145d25a0, 0x3a32fa0, 0xc002d7dcc0, 0x0, 0x0)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/row_fn.go:285 +0x75
github.com/influxdata/flux/stdlib/universe.(*mapTransformation).Process.func1(0x3a2c880, 0xc0145d25a0, 0xc0180cdfa0, 0xc000c1bc68)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/stdlib/universe/map.go:155 +0x105
github.com/influxdata/flux/execute.(*ColListTable).Do(0xc0145d25a0, 0xc00e4e9e60, 0xa, 0x10)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/table.go:1410 +0x74
github.com/influxdata/flux/stdlib/universe.(*mapTransformation).Process(0xc00589bcc0, 0x775a8cde7841293f, 0xc6d42d56a939ef85, 0x3a0f8a0, 0xc0145d25a0, 0x5, 0x0)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/stdlib/universe/map.go:152 +0xeb
github.com/influxdata/flux/execute.processMessage(0x3a07c60, 0xc013b36930, 0x3a21460, 0xc00589bcc0, 0x39de1a0, 0xc017efad00, 0xc0083d1600, 0x0, 0x0)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/transport.go:243 +0x30d
github.com/influxdata/flux/execute.(*consecutiveTransport).processMessages(0xc00f20c620, 0x3a07c60, 0xc013b36930, 0xa)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/transport.go:191 +0xb8
github.com/influxdata/flux/execute.(*poolDispatcher).run(0xc0083d17a0, 0x3a07c60, 0xc013b36930)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/dispatcher.go:132 +0x65
github.com/influxdata/flux/execute.(*poolDispatcher).Start.func1(0xc0083d17a0, 0x3a07c60, 0xc013b36930)
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/dispatcher.go:86 +0x87
created by github.com/influxdata/flux/execute.(*poolDispatcher).Start
    /go/pkg/mod/github.com/influxdata/flux@v0.95.0/execute/dispatcher.go:63 +0x7e

running the queries for mem_available, mem_total and the query except for the map function works fine.

It seems like there is something in this map function that triggers the panic only for the metrics bucket

    |> map(fn: (r) => (
        {
            _time: r._time,
            _value: (r._value_mem_total - r._value_mem_available) * 100 /r._value_mem_total,
            host: r.host
        }
    ))

Any help will with this will be much appreciated

2

Hello @reshadp,
It looks like you’re missing a parentheses

 |> map(fn: (r) => (
        {
            _time: r._time,
            _value: (r._value_mem_total - r._value_mem_available) * 100 <HERE> /r._value_mem_total <OR HERE>,
            host: r.host
        }
    ))

Does that fix it? If not, can you verify that you aren’t dividing by 0?

3

Hi @Anaisdg,

Thanks for your response, I tried all three of these lines, but got the same results.

_value: ((r._value_mem_total - r._value_mem_available) * 100 /r._value_mem_total ),
_value: ((r._value_mem_total - r._value_mem_available) * 100) /r._value_mem_total,
_value: (((r._value_mem_total - r._value_mem_available) * 100) /r._value_mem_total ),

I also eyeballed all the mem_total values, none of them are zero.

However something odd when looking at the csv, for one of the hosts there seem to be extra fields in the columns in the mem_total table.
I was experimenting with adding tags (location) via telegraf, and that is only done for one host right now.
Could that be causing this?

As a workaround I am thinking of just exposing the results of the join to python directly and running the calculation there.
May be easier to debug in the future.

Do let me know if there is anything else you can think of.
Thanks for your help.

4

Hello @reshadp,
Can you send me some of your annotated csv so I can try it on my end? Perhaps you can limit the size by adding |>limit(n:5)?
Thanks

5

Hello @Anaisdg,

I took a look at the 3 csv’s individually.
In these host server04 is the host I was testing additional tags on.

When I output the table for mem_available or mem_total I get the additional tags for this host.
If I try to output percent_used without the map statement I get output without a row for server04

Further if add the following filter anywhere before the map, it seems to work fine.

|> filter(fn: (r) => r["host"] != "server04")

I am attaching the 3 CSV files for you to verify this behaviour.

Thanks,
Reshad

tmp.tar.gz (911 Bytes) tmp.tar.gz (911 Bytes)

1 Like
6

Interestingly if I change the line to filter only for server04 it works too.
The map/join only fails if I do not have any filter for host server04