Problem writing to InfluxDB Cloud from AWS Lambda

[PMike]

I have AWS lambda function (US-EAST Ohio) that sends data into InfluxDB cloud (free tier, US-WEST). Using NodeJS @influxdata/influxdb-client.

The problem is that some writes show up in the database and some don’t. Client library reports success in either case.

Even more interesting is that when I run exactly the same function, with exactly the same input using AWS Lambda docker support from my laptop - it ingests perfectly.

The code that commits the points to the database is based on the example, except for “await”:

<omitted point prep>
await writeApi
      .close()
      .then(() => {
         StatusCode = 200;
         Body = "Statistics were successfully registered";
         console.log("Successfully wrote into InfluxDB")
      })
      .catch(e => {
         StatusCode = 400;
         Body = `Failed wriitng statistics: ${e}`
         console.error(e)
         console.log('\InfluxDB ERROR')
      })

I validate the number of ingested points by using a simple Flux query:
from(bucket: “Bucket-Name”)
|> range(start: -30d)
|> filter(fn: ® => r._field == “Field-Name” )
|> group(columns: [“time”])
|> count()
|> yield(name: “count”)

To sum up, same code, same input from my laptop works, from AWS - sometimes, generally once and twice and then doesn’t for quite a long time. InfluxDB client library reaches success branch in both cases…

Any idea what that could be?

[sranka]

WriteApi asynchronously writes points to InfluxDB. It buffers the points to write and send them to InfluxDB server upon reaching a time (1 minute) or buffer size (1000) limit, or upon calling flush or close programmatically. If any write to InfluxDB fails, a retry operation is by default automatically scheduled by the client library. The writeApi.close writes the points that were not written yet (buffered) and terminates all scheduled retries. The termination of retries is not an error. You can await writeApi.flush(true) to try to execute the retries before closing the client, this operation will fail upon any write failure, true let you await also for retries to succeed or fail.

To better understand the concepts behind the scene, see also https://github.com/influxdata/influxdb-client-js/blob/master/examples/writeAdvanced.js that shows how you can control the way of how points are written into InfluxDB.

You can also post a new issue in Issues · influxdata/influxdb-client-js · GitHub + attach a relevant lambda execution log, so I could answer better. The client always prints out an error message with a root cause upon write failure.

[PMike]

Thanks for the quick response @sranka!
I finally got it working by replacing .close() with .flush().

Apparently AWS Lambda attempts to optimize execution and doesn’t reload the entire file (that has writeAPI init code) on the subsequent calls. So I was practically closing the connection after the first call and it would obviously not be able to send any further points to Influx.

The interesting part is that local AWS SAM environment doesn’t have this optimization, so all of the file contents get reexecuted, API is reinitialized, points go out. Clearly not a nice behavior of AWS SAM debugging environment.

Thanks again for the description and flush() pointer, I hope this info would be useful to those who hit this problem in the future.