POST data from a text file into database

1

We have a script that writes data to a text file in the proper line protocol format when then internet connection of our device is interrupted. The goal is to write that file to our database when the connection is restored.

We have successfully written to the text file and, via Postman, able to write that file to the database when the binary option is selected in Postman and the file loaded in the body.

We are now attempting to use our script to automate the writing of the text file to the database. When connected to the internet our script successfully uses the HTTP POST method to write a line protocol entry into the database using the Content-Type text/plain.

However, using that same Content-Type to write the file to the database produces a 400 Bad Request response. I note in the Influx documentation that --data-binary is used in example of writing a file to the database, and this seems consistent with the option in Postman that has proven successful in writing the file.

I cannot seem to find the proper Content-Type to use in the script to write the text file to the database. I have also tried application/x-binary and application/octet-stream with no success.

2

Hello @dnederveld,
Thanks for your question. Have you tried using the tauk or file telegraf plugins? The tail plugin allows you to tail txt file and insert newly written lines. The file plugin is just a simple input plugin with line protocol as one of the ingest formats. I recommend checking out the tail plugin first.

3

Thank you for the suggestion, but I don’t believe the issue is with Telegraf since I am able to POST a txt file to the database using the Postman app. I now believe the issue is with script language of the device which contains the txt file I am wishing to POST. I just don’t have the syntax properly configured.

4

Hey. I should have been more clear. I highly recommend using telegraf as your collection agent because it offers you more flexibility with data ingest and could be easier than using postman (especially in the long run).

What version of influxdb are you using? Let me make sure I am understanding you correctly, you just wish to import a txt file into influxdb without postman now are unsuccessful? Are you wanting to make a curl request? Or import directly? Have you tried?

influx -import -path=$yourfilepath -precision=ns

Can you please share your script?

5

Thank you for your questions, and please excuse the long post while I give you as much information as I can.

I am running telegraf (v. 1.12.4) and influxdb (v. 1.7.8) on a Linux instance on AWS. I have only used Postman to test the HTTP POST command to test sending a text string of InfluxDB line protocol or a text file. I have successfully sent InfluxDB line protocol strings to my database using both Postman and my script. I assumed telegraf is being using as the collection agent in both those means of sending the InfluxDB line protocol string.

So, I am using Postman only to test things outside my script and, yes, I am trying to do the same task with my script outside of Postman.

My script is BASIC IDE as that is the language my device use where the data resides. The script I use to write a single InfluxDB line protocol string is as follows:

url$ = “<>/write?db=database”
method$ = “POST”
header$ = “Content-Type=text/plain”
REQUESTHTTPX url$, method$, header$, dataToSend$

The REQUESTHTTPX command is specific the scripting language of the device. The above command successfully writes to the database the InfluxDB line protocol string in the dataToSend$ variable.

The same REQUESTHTTPX command is used to write a file, as follows:

url$ = “<>/write?db=database”
method$ = “POST”
header$ = “Content-Type=text/plain”
REQUESTHTTPX url$, method$, header$, “”, fileToSend$

Note the “” in place of where the data string had been placed and the new variable related to the file being sent. The following is the documentation for the REQUESTHTTPX command:

REQUESTHTTPX http[s]://S1, S2 [, S3 [, S4 [, S5 [, S6 [, S7]]]]]

S1 is the Server
It is the URL of the targeted request.

S2 is the Method
It’s the REST API HTTP verb. This can be “Get”, “Post” … (Using Post, of course)

S3 are the Headers (optional)

S4 is the Post-Data (optional)
This is only used for writing a single InfluxDB line protocol string, and it works successfully. When sending a file this is “”.

S5 is the File-Data (optional)
This is only used when exporting a file, which is the option I have been unsuccessful with thus far.

S6 and S7 are optional and unrelated to any task being used.

When I test writing an InfluxDB line protocol string in Postman I select the ‘raw’ option under the Body tab. To send a file I select the ‘binary’ option. As stated previously, both tasks are successful using the appropriate option.

PS The device containing the data and the script sending the data is an eWON Flexy 205. The reference for the scripting language is Prog. Ref. Guide

PSS I notice no matter what I use for the Content-Type header, even gibberish, I still receive a 204 response code from the Influx database, as if the data was written from the file.

6

@Anaisdg, are you suggesting I enable something in Telegraf to ingest this text file? Have I been using the REST API of Inlfux and, if so, does that bypass Telegraf? Why, do you think, I can use the API in Telegraf but not my script?

Please see my previous post for details on my script.

7

@Anaisdg…

Below is the response from my Influx instance from the execution of the REQUESTHTTPX command on my device to send a single string of line protocol data to the database, and a successful writing or ingesting of the data into the database:

image
I have been able to successfully replicate this action over and again.

Below is the response from my Influx instance from the execution of the REQUESTHTTPX command on my device to send a text file of multiple line protocol strings of data to the database server, but the data is not successfully written or ingested into the database:

image
Note, the responses are virtually identical even though only the first one is successful.

When I send the same exact text file via the Postman app I receive this response:

image
image.png711×293 6.54 KB

Using the Postman app the data is successfully written or ingested into the database. Note, however, the responses sent to my device include connection: close whereas that is not included in the response to Postman.

The connection: close response sent to my device is sent both from the successful writing of the single line protocol string and the unsuccessful writing of the text file of multiple line protocol strings, so I’m not sure that explains anything.

8

Can you please share the strings you are trying to send? Multiple points should be in one string. No you’re right, telegraf doesn’t bypass the api, but I believe that part of the reason why telegraf is so popular is because it takes a lot of these hurdles out of the equation. Also as your ingest requirements grow you can fine tune the ingestion with telegraf to be more performant.

9

@Anaisdg, below is an example of a single single that I can write via Postman or my device:

DEVELOPMENT_DATA,unit=DEVELOPMENT val1=584.00,val2=3562.53,val3=1,CommConn=0 1572275666000000000

Below is an example of the contents of a text file I can send via Postman and it writes to the database. When I send from my device I receive a response that it is sent but the data is not written to the database.

DEVELOPMENT_DATA,unit=DEVELOPMENT val1=584.00,val2=3562.53,val3=1,CommConn=0 1572275666000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=578.00,val2=3572.94,val3=1,CommConn=0 1572275671000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=591.00,val2=3578.96,val3=1,CommConn=0 1572275676000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=576.00,val2=3586.59,val3=1,CommConn=0 1572275681000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=571.00,val2=3590.74,val3=1,CommConn=0 1572275686000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=593.00,val2=3595.50,val3=1,CommConn=0 1572275691000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=580.00,val2=3597.73,val3=1,CommConn=0 1572275696000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=575.00,val2=3599.66,val3=1,CommConn=0 1572275701000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=572.00,val2=3600.00,val3=1,CommConn=0 1572275706000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=582.00,val2=3599.07,val3=1,CommConn=0 1572275711000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=584.00,val2=3562.53,val3=1,CommConn=0 1572275666000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=578.00,val2=3572.94,val3=1,CommConn=0 1572275671000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=591.00,val2=3578.96,val3=1,CommConn=0 1572275676000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=576.00,val2=3586.59,val3=1,CommConn=0 1572275681000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=571.00,val2=3590.74,val3=1,CommConn=0 1572275686000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=593.00,val2=3595.50,val3=1,CommConn=0 1572275691000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=580.00,val2=3597.73,val3=1,CommConn=0 1572275696000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=575.00,val2=3599.66,val3=1,CommConn=0 1572275701000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=572.00,val2=3600.00,val3=1,CommConn=0 1572275706000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=582.00,val2=3599.07,val3=1,CommConn=0 1572275711000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=584.00,val2=3562.53,val3=1,CommConn=0 1572275666000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=578.00,val2=3572.94,val3=1,CommConn=0 1572275671000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=591.00,val2=3578.96,val3=1,CommConn=0 1572275676000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=576.00,val2=3586.59,val3=1,CommConn=0 1572275681000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=571.00,val2=3590.74,val3=1,CommConn=0 1572275686000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=593.00,val2=3595.50,val3=1,CommConn=0 1572275691000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=580.00,val2=3597.73,val3=1,CommConn=0 1572275696000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=575.00,val2=3599.66,val3=1,CommConn=0 1572275701000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=572.00,val2=3600.00,val3=1,CommConn=0 1572275706000000000
DEVELOPMENT_DATA,unit=DEVELOPMENT val1=582.00,val2=3599.07,val3=1,CommConn=0 1572275711000000000

I set up syslog this morning so I can see in Chronograf that the POST write commands are being seen by telegraf.

1 Like
10

The only thing I can think of here is that, since this appears to be a Windows box of some sort, the line ending is not the expected \n that a typical UNIX file would have for the newline character. It May be the typical Windows ^M or even \r but it would seem that Influx is not seeing this as a valid input.

dg

11

@davidgs, good thought but the device, I believe, runs a Linux OS because the SD card I have in the device which contains the text file had to be formatted as a Linux storage device (not FAT or other Windows-like formatting).

The construct of the text file is a multiple lines of line protocol strings, each followed by a line feed (ASCII character 10 or \n as you noted).

The mystery is why I can write the file to the database from Postman but not from my device, both using the HTTP POST command.

One thing I would like to discover is whether the text file is actually being transmitted from my device to the database. The syslog entry only shows that the API request was made but I do not see where I can check whether the file is being sent. In other words, how can I check what is actually being written to the /write endpoint.

12

I have tested the script with a different REST API tester to see what is transmitted. On that tester no content seems to be received. I am beginning to believe my script is not actually transmitting the data file.

Is there a means to check the logs on Influx to see if the file is being transmitted?

13

I wonder if you could try altering the script file to run a while loop and process the file one line at a time and see if that works?

Sorry, I’m grasping at straws here.

dg

14

Thanks for the suggestion. That was going to be my last resort. and I really didn’t want to interrupt my script for long periods of time to use that method.

My script is sending data to the Influx database at regular intervals. When that connection is interrupted I saved the data strings to a file on a SD card on my device. When the connection returns I wanted to send that file to Influx to have ingest the data that was collected while the connection was lost.

Everything I did worked, except properly formatting the REST API POST command on my device to send the file in a manner Influx could use to ingest the data. I received a ‘204’ response from Influx each time I executed the command, but no data was ingested and no other information could tell be where the process failed.

I finally was able to use a couple REST API test sites that could provide more information on how the command on my device was actually sending, or not sending, the file. I found out, with help from the device manufacturer, that the file was not actually being transmitted. With their help I was able to formulate the command on my device to properly send the file to Influx.

When I first test it I had a test file of about 30 hours of data, consisting of 23,000+ lines of line protocol strings. From my device the file was sent and the data ingested in about 3 secs, or less.

1 Like
15

It could also be that the file is simply too large. Maybe try, just to validate, splitting the file up into smaller parts? If that’s the problem, then we would have to find a way to do that programmatically in your script.

dg

16

@davidgs, file size was not an issue, it was the construct of the POST command in my script. I tested it again yesterday with a file containing five days of data (4MB) and it ingested into the database in 14 secs.

2 Likes
17

That’s great to hear! Sounds like you’re all set now.

dg