Writing a Pandas Dataframe into InfluxDB Cloud

Telegraf
, ,
1

Hey everyone,

My team recently set up a trial w/ InfluxDB 2.0 in order to test them out for use as a Time series database for our work. I have been trying to write a dataframe into Our InfluxDB cloud bucket similar to the solution provided here. and although we don’t get any errors and connect correctly (The only difference being the URL we use is an influxDB cloud url and not a local server, none of our data writes in. We tried various dataframes down to the simplest possible and it didn’t work.

Is this not possible to do to the cloud? The documentation I find suggests writing Data Points, InfluxDB line protocol, or batch writes in the line protocool format is the only way to do so.

Many thanks in advance if anyone could shed a light on this!

2

Hi @Maz,

could you share a debug output from client?

You could enable debug mode by: InfluxDBClient(url="http://localhost:9999", token="my-token", org="my-org", debug=True)

Regards

3

Code snippets posted in the forum, do not necessarily have to be error-free :wink:
I think there is a typo - also in the official documentation.
Should be: record=_data_frame

sure, that is possible

4

Hi Bednar.

As far as I can tell, there aren’t any issues?

Here you go:
send: b’POST /api/v2/write?org=(REDACTED)&bucket=(REDACTED)&precision=ns HTTP/1.1\r\nHost: us-east-1-1.aws.cloud2.influxdata.com\r\nAccept-Encoding: identity\r\nContent-Length: 56\r\nContent-Encoding: identity\r\nContent-Type: text/plain\r\nAccept: application/json\r\nAuthorization: Token (REDACTED) \r\nUser-Agent: influxdb-client-python/1.14.0\r\n\r\n’
send: b’Speed,DateTime=2021-02-18\ 12:56:44.859396 Speed=“0.0” 0’
reply: ‘HTTP/1.1 204 No Content\r\n’
header: Date: Thu, 18 Feb 2021 17:56:51 GMT
header: Connection: keep-alive
header: trace-id: 13e222fb08ce8d55
header: trace-sampled: false
header: Strict-Transport-Security: max-age=15724800; includeSubDomains

I removed the Token, Bucket name, and username for this output

Thanks!

5

If you look at debug output you could see that following LineProtocol was written into InfluxDB: Speed,DateTime=2021-02-18\ 12:56:44.859396 Speed=“0.0” 0.

Could you share how your code and DataFrame looks like?

Here is an example how to ingest DataFrame:


"""
Write Pandas DataFrame
"""
_now = pd.Timestamp().now('UTC')
_data_frame = pd.DataFrame(data=[["coyote_creek", 1.0], ["coyote_creek", 2.0]],
                           index=[now, now + timedelta(hours=1)],
                           columns=["location", "water_level"])

write_api.write("my-bucket", record=data_frame, data_frame_measurement_name='h2o_feet',
                    data_frame_tag_columns=['location'])
6

The written line protocol does not look like a valid line protocol to me.

Edit:

  • DateTime seems to be a “tag” - should be either a “field” or the timestamp?
  • Timestamp is 0

Yes that would be helpful

7

Hello,

Are there any further conclusions or solutions? I have the same problem using influxdb-client 1.16.

writing a dataframe to an OSS 1.8 influxDB works just fine. But writing to influxDB cloud 2.0 does not seem to work.

Below the head of the dataframe:

                    description machinePart machineSide    sensor    value

_time
2020-11-19 16:57:05.740 OdoMeter JacquardL Undefined JQ.L.ODO 3839202
2020-11-19 16:57:10.740 OdoMeter JacquardL Undefined JQ.L.ODO 3839216
2020-11-19 16:57:15.740 OdoMeter JacquardL Undefined JQ.L.ODO 3839231
2020-11-19 16:57:20.740 OdoMeter JacquardL Undefined JQ.L.ODO 3839246
2020-11-19 16:57:25.740 OdoMeter JacquardL Undefined JQ.L.ODO 3839260

Thank you in advance for taking a look at this.