InfluxDB Client failure when writing large amounts of data

InfluxDB 2
, , , ,
5

AN IMPORTANT NOTE: I should mention that this program has actually been failing before I reach the flush stage, not after, so the title was inaccurate. I have changed the title to reflect this, but I won’t edit the preceeding posts as that would change the context of this discussion. Does this mean I need to do something to increase the batch size, since it is overfilled?

Original post:

A written point is structured as:

p := influxdb2.NewPointWithMeasurement("ObjectData").
		AddTag("name", object.Name). 
		AddField("var1", var1).
		AddField("var2", var2).
		...
		AddField("var8", var8). 
		SetTime(timestamp)

Vars 1-6 are float64, with the last two Vars being int .

I should have mentioned earlier, but there’s also another type of point that gets written occasionally (but not every time). This is associated with another separate group of objects, which is considerably larger than the ones I am mentioning here. I’m not sure of the exact contribution to writes here, but it would be a bit larger than the satellites…in any case, it’s going to be contributing to load for sure.

Those points are similar to that above, except there are only three Fields: one name string, and two float64s.

The error looks like this:

 2020/07/22 09:50:00 Post "http://localhost:9999/api/v2/query?org=MyOrg": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
 2020/07/22 09:50:00 http: panic serving MyIPAddress:SomePortNumber: Post "http://localhost:9999/api/v2/query?org=MyOrg": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

    goroutine 19 [running]:
    net/http.(*conn).serve.func1(0xc000204280)
            /usr/lib/go-1.14/src/net/http/server.go:1772 +0x139
    panic(0xbcbb40, 0xc194ab39b0)
            /usr/lib/go-1.14/src/runtime/panic.go:975 +0x3e3
    log.Panicln(0xc049fa9420, 0x1, 0x1)
            /usr/lib/go-1.14/src/log/log.go:365 +0xac
    github.com/exampleOrg/sim.MyQueryFunction(0xe001c0, 0xc0002cb240, 0x0, 0xed6a97c97, 0x14d6f00, 0xc0001009b0, 0x3e8, 0xc194addf18)

… (traces through my personal functions)… then into:

    net/http.HandlerFunc.ServeHTTP(0xc00037a0a0, 0xe05280, 0xc0001935e0, 0xc0002e5000)
            /usr/lib/go-1.14/src/net/http/server.go:2012 +0x44
    net/http.(*ServeMux).ServeHTTP(0x14d6e60, 0xe05280, 0xc0001935e0, 0xc0002e5000)
            /usr/lib/go-1.14/src/net/http/server.go:2387 +0x1a5
    net/http.serverHandler.ServeHTTP(0xc000193500, 0xe05280, 0xc0001935e0, 0xc0002e5000)
            /usr/lib/go-1.14/src/net/http/server.go:2807 +0xa3
    net/http.(*conn).serve(0xc000204280, 0xe06a40, 0xc0002caf80)
            /usr/lib/go-1.14/src/net/http/server.go:1895 +0x86c
    created by net/http.(*Server).Serve
            /usr/lib/go-1.14/src/net/http/server.go:2933 +0x35c
    2020/07/22 09:50:47 [E]! Write error: Post "http://localhost:9999/api/v2/write?bucket=MyBucket&org=MyOrg&precision=ns": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    write error: Post "http://localhost:9999/api/v2/write?bucket=MyOrg&org=MyBucket&precision=ns": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    signal: killed

Increasing the timeout via SetHttpRequestTimeout did not solve the query error issue, but it did prevent write error from appearing. Increasing the Batch size to 10,000, RetryBufferLimit to 50,000 and Flush Interval to 1000 didn’t fix the query error.