Error when writing data : unexpected error writing points to database: [shard 2613] unexpected end of JSON input

InfluxDB 2
1

We encounter problems trying to write data using the python client through our influxdb v2 instance.
We need to fill a gap in data archives.

The error message when I try to write data

unexpected error writing points to database: [shard 2613] unexpected end of JSON input

Full debug error message

[DEBUG] INFLUXDB_ADRESS “POST /api/v2/write?org=MY_ORG&bucket=MY_BUCKET&precision=s HTTP/1.1” 500 124
[DEBUG] response body: {“code”:“internal error”,“message”:“unexpected error writing points to database: [shard 2613] unexpected end of JSON input”}
[ERROR] The batch item wasn’t processed successfully because: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({‘Server’: ‘nginx/1.18.0 (Ubuntu)’, ‘Date’: ‘Wed, 08 Dec 2021 14:39:23 GMT’, ‘Content-Type’: ‘application/json; charset=utf-8’, ‘Content-Length’: ‘124’, ‘Connection’: ‘keep-alive’, ‘X-Influxdb-Build’: ‘OSS’, ‘X-Influxdb-Version’: ‘2.1.1’, ‘X-Platform-Error-Code’: ‘internal error’})
HTTP response body: {“code”:“internal error”,“message”:“unexpected error writing points to database: [shard 2613] unexpected end of JSON input”}
[INFO] the batching processor was disposed

No similar errors were found on community sites.
Added to the vague message itself, the message error is not clear at all.

We first suspected shard datatype error, but we tried to write strings, integers, floats and keep receiving this error. We keep thinking this is a tied problem but miss clear information with the error message.

No error if I use the same command line and write the same data :

  • in another bucket
  • in another instance of influxDB based on the backup of the main DB, exactly in the same measurement

We even deleted all data in the whole month of March 2019, and tried to rewrite it all.
It works fine for the first days and the same error happens on 2019-03-11.

Can someone clarify this error, to permit us to fill data ?

Example code used :

from influxdb_client import InfluxDBClient, WriteOptions

cli = "measurement,tag1=tag1_value field=-30 1552262460"

client = InfluxDBClient(
        url = "XXX",
        token = "MY_TOKEN",
        org = "ORG",
        debug = True,
        verify_ssl = False
    )

write_client = client.write_api(
            write_options = WriteOptions(
                batch_size = 5_000,
                max_retries = 2,
                max_retry_delay = 30000
            )
        )

write_client.write(
                bucket = "MY_BUCKET",
                org = "MY_ORG",
                record = cli,
                write_precision = "s"
            )
write_client.close()
client.close()

We’ll do everything possible to succeed writing data in these gaps periods.

1 Like
2

Similar issues, with the .net client. Randomly get “Unexpected end of JSON input”

3

We found the reason digging in raw files stored in the databases.
Two “*.tsm” files associated to the 2 shards where we faced this problem were “locked”.

We have the chance to work with a “dev” server generated each day from the backup of the “production” server. We did not face this problem on this server.
So we copied the “*.tsm” files from “dev” to “prod”, taking care of the files right access, and resolved the problem.
We can now write on these 2 shards.

Sadly, we do not have any ideas why these 2 shards were corrupted.

4

I’m facing the same problem after my Influxdb VM ran out of disk space. I assume the tsm files got corrupted, now I can’t write new data. Is there a way to delete or truncate the corrupted files, to allow new write to work again?

1 Like
5

Were you able to solve it somehow? Have the same problem right now :smiling_face_with_tear: