Insert data into influxdb from node / windows causes problems

Store
,
1

I’m trying to insert data into InfluxDb using node/windows but there is a (seemingly well-know) bug that I am having trouble wrapping my mind around. I am trying to do a batch insert into my local db by loading data from a file and sending to my database using a POST. Here is the data that I am trying to insert:

test_data.txt

test_measurement test=val1 1516665684
test_measurement price=val2 1516665685
test_measurement price=val3 1516665686

Using the code:

var http = require(“http”);
var fs = require(‘fs’);

fs.readFile(‘test_data.txt’, { encoding: ‘utf8’ }, function (err, data) {
var options = {
“method”: “POST”,
“hostname”: “localhost”,
“port”: “8086”,
“path”: “/write?db=test_db”,
“headers”: {
“cache-control”: “no-cache”
}
};

var req = http.request(options, function (res) {
var chunks = ;

res.on("data", function (chunk) {
  chunks.push(chunk);
});

res.on("end", function () {
  var body = Buffer.concat(chunks);
  console.log(body.toString());
});

});

req.write(data);
req.end();
});

And I get the error

{"error":"unable to parse 'test_measurement test=val1 1516665684\r': bad timestamp\nunable to parse 'test_measurement price=val2 1516665685\r': bad timestamp\nunable to parse 'test_measurement price=val3 1516665686': bad timestamp"}

From my research, I understand that windows uses \r\n characters as a carriage return, but I’m having trouble wrapping my mind around how to fix this. I’ve made attempts to remove all \r and \n characters when I read the file in, but it doesn’t seem to remove them, as I continue to get the same error. Even if I was able to remove them this way, how would influx know where the next insert of the batch began? It needs to know where the next line starts for a batch insert, but doesn’t like the new line characters.

Any help would be great!!

2

I’m a bit confused by some of this, so hopefully you can clarify. Valid line-protocol is of the form:

measurement,tag1=tag_value,tag2=tag_value value=9999,value2=8888 timestamp

Your data is not being sent in the right format. first, there should be a comma , not a space after the measurement name followed by the tag=value (I’m assuming that “test=val1” is your tag. After that, you would insert your field values as field=value pairs, finally followed by a timestamp – though the timestamp is optional.

See the line-protocol docs here :Writing data with the HTTP API | InfluxDB OSS 1.4 Documentation for more information.

If I’ve misinterpreted what you were doing/trying to do, please do let me know!

Best regards,
dg

1 Like
3

Hi David, I’m very new to influx so I could be wrong here, but I do not have any value that is worthy of being a tag, so i’m omitting tags. I just have a timestamp and two field values. I read that if you do not want tags, you can omit them by using the format:

measurement value1=value1,value2=value2 timestamp

And it has been working great! So I do not believe that this is a line-protocol format issue. The issue seems to be something to do with windows adding \r\n at the end of each line to represent a new line and influxdb not being capable of parsing that.

4

True, you can omit the tags.

The ‘invalid boolean’ is actually indicative of the value not being correct. Try replacing val1, val2, etc. with numeric values, or quoting the string values and see if that changes things.

dg

1 Like
5

Wow, sorry about that! It actually says ‘bad timestamp’, i have no idea how ‘invalid boolean’ got in there. It says ‘bad timestamp’ due to the fact that it’s trying to read 1516665684\r as a timestamp.

Sry to mislead you there. Good catch. Edited my original question.

6

curl -i -XPOST ‘http://localhost:8086/write?db=mydb’ --data-binary ‘test_measurement test=val1 1516665684’
HTTP/1.1 400 Bad Request
Content-Type: application/json
Request-Id: 656cfa5a-079c-11e8-98d4-000000000000
X-Influxdb-Build: OSS
X-Influxdb-Error: unable to parse ‘test_measurement test=val1 1516665684’: invalid boolean
X-Influxdb-Version: v1.4.2
X-Request-Id: 656cfa5a-079c-11e8-98d4-000000000000
Date: Thu, 01 Feb 2018 22:08:13 GMT
Content-Length: 85

{“error”:“unable to parse ‘test_measurement test=val1 1516665684’: invalid boolean”}
Davids-MacBook-Pro:Downloads davidgs$ curl -i -XPOST ‘http://localhost:8086/write?db=mydb’ --data-binary ‘test_measurement test=“val1” 1516665684’
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 6d390206-079c-11e8-98d7-000000000000
X-Influxdb-Build: OSS
X-Influxdb-Version: v1.4.2
X-Request-Id: 6d390206-079c-11e8-98d7-000000000000
Date: Thu, 01 Feb 2018 22:08:26 GMT

So val1 is invalid, but “val1” is ok. That’s the origin of your invalid boolean error. You will also, apparently, need to strip the \r and replace it with \n in order to make this successful.

You should be able to use string.replace() to remove the \r from your lines in your JavaScript.

res.on("data", function (chunk) {
    chunk = chunk.replace("\r" "");
    chunks.push(chunk);
});

dg

1 Like
7

Awesome!! It’s working when stripping the \r. This will be an extra step that I assume will slow things down a bit when looking at 100 million lines of data from files, but it looks like the best solution possible.

Thanks for the help!

8

saved my bacon after spending hours thanks @davidgs !