For context this relates to an upload service included in the IoTaWatt energy monitor. The device has supported uploading to the OSS 1.x version for a couple of years now. Recently made the changes to upload to 2.0. The device can push data to various upstream servers seamlessly, so sometimes that may involve uploading a backlog after a communications outage.
Now I’d like to provide the ability to upload essentially all of the data collected so that larger users can aggregate their data in the cloud version. Problem is that the line protocol is way verbose and repetitive. The CSV format would be very efficient.
I can see where CSV can be pulled from files, but have not come across a simple way to just push annotated CSV to an influx endpoint. Not really interested in agents, python scripts or other middleware. I can build clean annotated CSV to send directly.
Surely this is possible given the wholesale conversion to CSV and tables in flux. Can someone point me at the best (any) way to do this?
The first suggestion pulls the data from a file, I want to push a CSV table with a HTTP Post.
The second, example is getting closer, but still seems to require that I pass it a file rather than just push a HTTP Post payload with the CSV.
I did find the Telegraf HTTP_LISTENER_V2 that by the brief description I think I can push the CSV to Telegraf and it will expand to Write protocol and send to influx_v2. Does that sound right?
It took a long time to pour through the long list of Telegraf plugins to discover this, and I when I have more time I’ll explore whether it can suit my needs.
There’s two solutions that we’ll likely have for this. First, we are working on making it possible to perform remote writes using all versions of Flux. I think the cloud version can do remote writes, but OSS and the flux CLI can’t. Once we’ve done this, it would be possible to use the flux command line client to push csv from your local disk since that CLI client has access to the filesystem service.
The second is building this functionality directly into the influx write command with some kind of --format flag or similar. If you want to file an issue for that, we’d welcome contributions or someone would pick it up. I don’t have any timeline for when we might have time to pick it up and implement it, but I think it would be something that we’d work on at some point.
