Influxdb 2.1.1 Internal Server Error (Timestamp related)

Hello,
I’m using curl to add data to my bucket and I’m getting a runtime error “integer divide by zero”

HTTP/1.1 500 Internal Server Error
Content-Length: 168
Content-Type: application/json; charset=utf-8
Date: Wed, 23 Feb 2022 22:09:04 GMT
X-Influxdb-Build: OSS
X-Influxdb-Version: 2.1.1
X-Platform-Error-Code: internal error

{“code”:“internal error”,“message”:“a panic has occurred: /api/v2/write?org=myorg\u0026bucket=test\u0026precision=s: runtime error: integer divide by zero”}

I only get this error if the date is in 2022 but if I change the year to 2021 it works flawlessly

Works (2021)
Here are the commands that work followed by the one that doesn’t
curl -ki -XPOST 'https://influx.server.lan/api/v2/write?org=myorg&bucket=test&precision=s' --header "Authorization: Token xyz" --data-raw "data amount=0.03 1613942394"

Doesn’t work (2022)
curl -ki -XPOST 'https://influx.server.lan/api/v2/write?org=myorg&bucket=test&precision=s' --header "Authorization: Token xyz" --data-raw "data amount=0.03 1645478394"

I spent a long time trying to figure out if I had a typo but to confirm I simply copied the old line exactly as it is and only changed the timestamp and boom got the error message. What’s going on? Is this some kind of bug or am I missing something?

Thank you in advance

Hi @t481,
Let me see if I can get one of the edge team to check this out for you.

Thanks,
Jay

1 Like

I tried:

curl -vvv -XPOST 'http://localhost:8086/api/v2/write?org=myorg&bucket=test&precision=s' --header "$AUTH" --data-raw "data amount=0.03 1645478394"
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying ::1:8086...
* Connected to localhost (::1) port 8086 (#0)
> POST /api/v2/write?org=myorg&bucket=test&precision=s HTTP/1.1
> Host: localhost:8086
> User-Agent: curl/7.77.0
> Accept: */*
> Authorization: Token hHHJz7ZEiPn2wv7T0gi0V1ox42vN1mCJYMdYWWG4o-S79g70d9bMG9Hh039C_7rQ3DOlH9kPUyBuiI9mUIgadg==
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< X-Influxdb-Build: OSS
< X-Influxdb-Version: dev
< Date: Thu, 24 Feb 2022 14:26:41 GMT
<
* Connection #0 to host localhost left intact

I don’t get the same error. Can you share the logs from the OSS instance? There will hopefully be a better panic stacktrace there which would give more details.

Here’s the log from "docker logs influxdb | grep “panic”
I hope this makes sense

2022-02-23T19:14:29.427377Z lvl=error msg="a panic has occurred" log_id=0Zqvtqf0000 handler=panic error="/api/v2/write?org=myorg&bucket=test&precision=s: runtime error: integer divide by zero" stacktrace="goroutine 10796 [running]:\nruntime/debug.Stack()\n\t/home/circleci/.tools/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/influxdata/influxdb/v2/http.baseHandler.panic({{0x4058600, 0xc0084cf860}}, {0x4077400, 0xc0018d5ae0}, 0x6, {0x3c52600, 0x53e34a0})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/router.go:96 +0x1b3\ngithub.com/influxdata/httprouter.(*Router).recv(0xc0089410e0, {0x4077400, 0xc0018d5ae0}, 0xd)\n\t/home/circleci/go/pkg/mod/github.com/influxdata/httprouter@v1.3.1-0.20191122104820-ee83e2772f69/router.go:361 +0x5e\npanic({0x3c52600, 0x53e34a0})\n\t/home/circleci/.tools/go/src/runtime/panic.go:1038 +0x215\ngithub.com/influxdata/influxdb/v2/v1/services/meta.(*ShardGroupInfo).ShardFor(...)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/services/meta/data.go:1439\ngithub.com/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).MapShards(0xc0009c7880, 0xc000306658)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/coordinator/points_writer.go:237 +0x678\ngithub.com/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).WritePointsPrivileged(0xc0009c7880, {0x40a6bc0, 0xc015af9c80}, {0xc017df7550, 0x415f07}, {0x238fa4a, 0x3eadf80}, 0x1, {0xc009240080, 0x1, ...})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/coordinator/points_writer.go:361 +0x189\ngithub.com/influxdata/influxdb/v2/v1/coordinator.(*PointsWriter).WritePoints(0xc000306748, {0x40a6bc0, 0xc015af9c80}, {0xc017df7550, 0x203000}, {0x238fa4a, 0x100c0018d5d80}, 0x0, {0x40b92c8, 0xc00c5bc0c0}, ...)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/v1/coordinator/points_writer.go:340 +0x5c\ngithub.com/influxdata/influxdb/v2/storage.(*Engine).WritePoints(0xc0001d4c00, {0x40a6bc0, 0xc015af9c80}, 0xc0018d5d80, 0x0, {0xc009240080, 0x1, 0x1})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/engine.go:265 +0x204\ngithub.com/influxdata/influxdb/v2/storage.(*LoggingPointsWriter).WritePoints(0xc011917ad0, {0x40a6bc0, 0xc015af9c80}, 0xd6d1f4caa7dafb50, 0xc0187e80c9, {0xc009240080, 0x1, 0x1})\n\t/home/circleci/go/src/github.com/influxdata/influxdb/storage/points_writer.go:38 +0xd1\ngithub.com/influxdata/influxdb/v2/http.(*WriteHandler).handleWrite(0xc0155ced20, {0x4077400, 0xc0018d5ae0}, 0x90)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/write_handler.go:192 +0x54b\nnet/http.HandlerFunc.ServeHTTP(0x100000000000000, {0x4077400, 0xc0018d5ae0}, 0x90)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/httprouter.(*Router).Handler.func1({0x4077400, 0xc0018d5ae0}, 0x3c4d440, {0x0, 0x3b69c00, 0xc00a24fee0})\n\t/home/circleci/go/pkg/mod/github.com/influxdata/httprouter@v1.3.1-0.20191122104820-ee83e2772f69/router.go:325 +0x2c4\ngithub.com/influxdata/httprouter.(*Router).ServeHTTP(0xc0089410e0, {0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/pkg/mod/github.com/influxdata/httprouter@v1.3.1-0.20191122104820-ee83e2772f69/router.go:453 +0x4c3\ngithub.com/influxdata/influxdb/v2/http.(*WriteHandler).ServeHTTP(0xd, {0x4077400, 0xc0018d5ae0}, 0x321)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/write_handler.go:135 +0x27\ngithub.com/go-chi/chi.(*Mux).Mount.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:298 +0x13b\nnet/http.HandlerFunc.ServeHTTP(0x3c2d5a0, {0x4077400, 0xc0018d5ae0}, 0xc017df70f0)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/go-chi/chi.(*Mux).routeHTTP(0xc0089245a0, {0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:431 +0x1f9\nnet/http.HandlerFunc.ServeHTTP(0xc015af9890, {0x4077400, 0xc0018d5ae0}, 0xc0003070b8)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/kit/transport/http.SetCORS.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:36 +0x308\nnet/http.HandlerFunc.ServeHTTP(0xe, {0x4077400, 0xc0018d5ae0}, 0xc000a45b80)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/go-chi/chi/middleware.StripSlashes.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/middleware/strip.go:25 +0x102\nnet/http.HandlerFunc.ServeHTTP(0xc015af9890, {0x4077400, 0xc0018d5ae0}, 0x30)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/kit/transport/http.SkipOptions.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:81 +0xb9\nnet/http.HandlerFunc.ServeHTTP(0xc015af98f0, {0x4077400, 0xc0018d5ae0}, 0xc000307238)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/http.panicMW.func1.1({0x4077400, 0xc0018d5ae0}, 0xc0003072f0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/router.go:126 +0x9b\nnet/http.HandlerFunc.ServeHTTP(0x90, {0x4077400, 0xc0018d5ae0}, 0x1000000023dd680)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/go-chi/chi.(*Mux).ServeHTTP(0xc0089245a0, {0x4077400, 0xc0018d5ae0}, 0xc006cd2700)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:70 +0x40e\ngithub.com/influxdata/influxdb/v2/kit/feature.(*Handler).ServeHTTP(0xc008998d00, {0x4077400, 0xc0018d5ae0}, 0xc006cd2600)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/feature/middleware.go:43 +0x3c2\ngithub.com/influxdata/influxdb/v2/http.(*AuthenticationHandler).ServeHTTP(0xc0155ceaf0, {0x4077400, 0xc0018d5ae0}, 0xc006cd2500)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/authentication_middleware.go:127 +0x467\ngithub.com/influxdata/influxdb/v2/kit/transport/http.SetCORS.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2500)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:36 +0x308\nnet/http.HandlerFunc.ServeHTTP(0xc015af9890, {0x4077400, 0xc0018d5ae0}, 0xc017df70f1)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/kit/transport/http.SkipOptions.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2500)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:81 +0xb9\nnet/http.HandlerFunc.ServeHTTP(0x33d8a90, {0x4077400, 0xc0018d5ae0}, 0xc)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/http.(*PlatformHandler).ServeHTTP(0xc008998e00, {0x4077400, 0xc0018d5ae0}, 0xc006cd2500)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/platform_handler.go:81 +0x1d2\ngithub.com/go-chi/chi.(*Mux).Mount.func1({0x4077400, 0xc0018d5ae0}, 0xc006cd2500)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:298 +0x13b\nnet/http.HandlerFunc.ServeHTTP(0x203000, {0x4077400, 0xc0018d5ae0}, 0xc0018d5ac0)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/kit/transport/http.Metrics.func1.1({0x407c1d0, 0xc0001b2700}, 0xc006cd2500)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:66 +0x1e2\nnet/http.HandlerFunc.ServeHTTP(0xc000307830, {0x407c1d0, 0xc0001b2700}, 0x702dc5)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/kit/transport/http.Trace.func1.1({0x407c1d0, 0xc0001b2700}, 0x1)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/kit/transport/http/middleware.go:106 +0x30c\nnet/http.HandlerFunc.ServeHTTP(0xc0168fd170, {0x407c1d0, 0xc0001b2700}, 0xc00a832678)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/go-chi/chi.(*ChainHandler).ServeHTTP(0x3c2d5a0, {0x407c1d0, 0xc0001b2700}, 0xc0187e80a5)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/chain.go:31 +0x2c\ngithub.com/go-chi/chi.(*Mux).routeHTTP(0xc008946f00, {0x407c1d0, 0xc0001b2700}, 0xc006cd2400)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:431 +0x1f9\nnet/http.HandlerFunc.ServeHTTP(0x88, {0x407c1d0, 0xc0001b2700}, 0x90)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/influxdata/influxdb/v2/http.(*AddHeader).Middleware.func1({0x407c1d0, 0xc0001b2700}, 0x53c4e01)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/handler.go:102 +0x77\nnet/http.HandlerFunc.ServeHTTP(0x40a6b18, {0x407c1d0, 0xc0001b2700}, 0x53c4e90)\n\t/home/circleci/.tools/go/src/net/http/server.go:2046 +0x2f\ngithub.com/go-chi/chi.(*Mux).ServeHTTP(0xc008946f00, {0x407c1d0, 0xc0001b2700}, 0xc006cd2300)\n\t/home/circleci/go/pkg/mod/github.com/go-chi/chi@v4.1.0+incompatible/mux.go:86 +0x3c2\ngithub.com/influxdata/influxdb/v2/http.(*Handler).ServeHTTP(0xc0187e80b3, {0x407c1d0, 0xc0001b2700}, 0x0)\n\t/home/circleci/go/src/github.com/influxdata/influxdb/http/handler.go:166 +0x2f\nnet/http.serverHandler.ServeHTTP({0x4072558}, {0x407c1d0, 0xc0001b2700}, 0xc006cd2300)\n\t/home/circleci/.tools/go/src/net/http/server.go:2878 +0x43b\nnet/http.(*conn).serve(0xc00899b5e0, {0x40a6bc0, 0xc0168fd4d0})\n\t/home/circleci/.tools/go/src/net/http/server.go:1929 +0xb08\ncreated by net/http.(*Server).Serve\n\t/home/circleci/.tools/go/src/net/http/server.go:3033 +0x4e8\n"

@Samuel_Arnold Any news?

can anyone help?

I’ve tried this on 3 different devices (RPi, and 2 different laptops) and I’m getting the same panic integer divide by zero

I tried using the UI and manually entered “data amount=0.03 1645478394” after selecting the bucket and I get

but I cannot find anything in the bucket afterwards. Still empty!

I even tried deleting the buckets and restoring from backup so as to have a “clean” version to no avail. Created a “test” bucket and entered the data there and still can’t see anything if done via the interface or the integer divide by zero via CLI

Still experiencing the issue!
Bizarrely, I’ve attempted to add the same data to a “test” bucket and it seems to work.

Does this mean that my original bucket is corrupted? Do I need to repair the bucket if that’s even possible? I can retrieve the data and I can also write data provided it’s older than 2022.

Any suggestions?

Hello all
Finally managed to resolve the issue.

  • Created a new bucket

  • Ran the following query to copy the contents to the new bucket

    from(bucket: "original_bucket")
      |> range(start: 1970-08-28T22:00:00Z)
      |> to(bucket: "new_bucket", org: "myorg")
    
  • Renamed the old bucket (could have deleted it too)

  • Renamed the new one to match the old bucket name

  • Re-created authorisation keys for the bucket

  • Ran the curl command to post the data and voila it worked

2 Likes

Hello, @Jay_Clifford

I have recently encountered the same problem, happening in two buckets in the database.

It is possible to push data until 2022-12-11, but not from 2022-12-12.

This solution presented here works as well, however, my buckets contain quite a large amount of data.

Therefore, is there any suggestion on how to have a better way to debug it? The logs are not very useful to be honest.

I tried several things before as well, deleting previous data in the bucket, changing input data, etc. Unfortunately, nothing works.

I’m using an influx 2.4 version

Regards

Hi @giuliano.lm,
Can you provide a sample of the data you are trying to push to InfluxDB and the method you are using?

@Jay_Clifford

I probably can share with you part of the backup bucket, thus you could replicate my error.

I’m basically using the influx_client python library.
After setting up the connection:
write_api = client.write_api(write_options=WriteOptions(batch_size=5000))

Write data into database

write_api.write(bucket=database, record=data,data_frame_measurement_name=table,data_frame_tag_columns=tag_list,writing_precision=‘s’)

Anyway, if I put in debug mode, and copy/paste the exact line protocol in the UI, it gives an “Unknown error” as well.

I will check which data I would be able to forward you, and PM you asap.

Regards

Hi!

Any advances here? We have hit the same stone with 20 out of an estimate of 5 million records that have been ingested in the last batch. Copying and renaming the bucket is a bit out of the question, due to size and operational constraints.

Hello,

I’ve sent the example buckets to @Jay_Clifford a while ago, but unfortunately didn’t hear anything back.

Perhaps someone else from influx could assist here? @Anaisdg

Regards

Hello

I’m having the same issue. I got this error when timestamp are higher than 1699833600 (Mon Nov 13 2023 00:00:00 GMT+0000), the second before works (Sun Nov 12 2023 23:59:59 GMT+0000).

Having a newly fresh bucket doesn’t show the problem.

I can reproduce this issue with influxdb 2.7.4 (19e5c0e) and 2.7.3 ( ed645d9)

I can confirm that it still happens.
Setup:
InfluxDB v2.7.1
Server: 407fa62
Frontend: 9d25a2f
C# API.

Setup two influxdb running in containers. Let’s call them source and target.
Create a few buckets and store some data with second precision.

Back Data from source instance of InfluxDB and restore to target instance.

If you try to add new records, you run into precions issue divide by zero…

Work Around: Copy data from one bucket other as suggested above.

If required, i can share data to reproduce it.