New entry overwrites all previous data?

, ,
1

I am noticing what appears to be something that just doesn’t make sense to me.

By inserting a new entry to the database, it seems to cause all existing data to get wiped?

I have two python scripts. The first writes some historical data from a CSV file. The second polls an API and writes the current measurement.

When I run the script to write the current measurement, it causes all the previous data to vanish.

Here you can see me run the first script to write the measurements. And then I view the measurements to confirm they are in the DB.

[username@testbox scripts]$ python oldportcounts.py
[{'fields': {'ports': '2\r'}, 'time': '2017-03-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '50\r'}, 'time': '2017-03-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '62\r'}, 'time': '2017-03-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23\r'}, 'time': '2017-03-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-03-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '10\r'}, 'time': '2017-03-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4\r'}, 'time': '2017-03-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-31T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '42\r'}, 'time': '2017-04-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '20\r'}, 'time': '2017-04-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '6\r'}, 'time': '2017-04-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-04-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '16\r'}, 'time': '2017-04-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-04-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4\r'}, 'time': '2017-04-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14\r'}, 'time': '2017-04-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '22\r'}, 'time': '2017-05-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23\r'}, 'time': '2017-05-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-05-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '1\r'}, 'time': '2017-05-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-05-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-05-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60\r'}, 'time': '2017-05-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '12\r'}, 'time': '2017-05-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14\r'}, 'time': '2017-05-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60\r'}, 'time': '2017-05-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-18T07:00:00Z', 'measurement': 'portcount'}]
[username@testbox scripts]$ influx -precision rfc3339
Connected to http://localhost:8086 version 1.2.2
InfluxDB shell version: 1.2.2
> use portcount
Using database portcount
> select * from portcount
name: portcount
time                 ports
----                 -----
2017-03-16T07:00:00Z 2
2017-03-17T07:00:00Z 50
2017-03-18T07:00:00Z 0
2017-03-19T07:00:00Z 0
2017-03-20T07:00:00Z 0
2017-03-21T07:00:00Z 0
2017-03-22T07:00:00Z 0
2017-03-23T07:00:00Z 62
2017-03-24T07:00:00Z 23
2017-03-25T07:00:00Z 0
2017-03-26T07:00:00Z 0
2017-03-27T07:00:00Z 0
2017-03-28T07:00:00Z 2
2017-03-29T07:00:00Z 10
2017-03-30T07:00:00Z 4
2017-03-31T07:00:00Z 0
2017-04-01T07:00:00Z 0
2017-04-02T07:00:00Z 0
2017-04-03T07:00:00Z 0
2017-04-04T07:00:00Z 8
2017-04-05T07:00:00Z 42
2017-04-06T07:00:00Z 0
2017-04-07T07:00:00Z 20
2017-04-08T07:00:00Z 0
2017-04-09T07:00:00Z 0
2017-04-10T07:00:00Z 0
2017-04-11T07:00:00Z 8
2017-04-12T07:00:00Z 0
2017-04-13T07:00:00Z 6
2017-04-14T07:00:00Z 5
2017-04-15T07:00:00Z 0
2017-04-16T07:00:00Z 0
2017-04-17T07:00:00Z 0
2017-04-18T07:00:00Z 8
2017-04-19T07:00:00Z 16
2017-04-20T07:00:00Z 0
2017-04-21T07:00:00Z 2
2017-04-22T07:00:00Z 0
2017-04-23T07:00:00Z 0
2017-04-24T07:00:00Z 0
2017-04-25T07:00:00Z 4
2017-04-26T07:00:00Z 0
2017-04-27T07:00:00Z 14
2017-04-28T07:00:00Z 0
2017-04-29T07:00:00Z 0
2017-04-30T07:00:00Z 0
2017-05-01T07:00:00Z 0
2017-05-02T07:00:00Z 22
2017-05-03T07:00:00Z 23
2017-05-04T07:00:00Z 5
2017-05-05T07:00:00Z 1
2017-05-06T07:00:00Z 0
2017-05-07T07:00:00Z 0
2017-05-08T07:00:00Z 0
2017-05-09T07:00:00Z 0
2017-05-10T07:00:00Z 5
2017-05-11T07:00:00Z 2
2017-05-12T07:00:00Z 60
2017-05-13T07:00:00Z 12
2017-05-14T07:00:00Z 0
2017-05-15T07:00:00Z 14
2017-05-16T07:00:00Z 60
2017-05-17T07:00:00Z 0
2017-05-18T07:00:00Z 0

> 
> exit

Then I run the second script and print out the data that is being written so you can see. After it is written I go check, and all the old data is gone, and just the new measurement is there.

[username@testbox scripts]$ python portcount.py
[{'fields': {'ports': 6}, 'time': '2017-05-22T21:51:58Z', 'measurement': 'portcount'}]

 [username@testbox scripts]$ influx -precision rfc3339
    Connected to http://localhost:8086 version 1.2.2
    InfluxDB shell version: 1.2.2
    > use portcount
    Using database portcount
    > select * from portcount
    name: portcount
    time                 ports
    ----                 -----
    2017-05-22T21:51:58Z 6

The format of the data is the same. The timestamp format appears to be the same also? Yet for some reason it causes all previous data to disappear.

There is nothing in the scripts stating to drop anything.

oldportcounts.py script:

# !/usr/bin/env python
from __future__ import print_function
from functools import wraps
from pprint import pprint
import sys
import requests
import datetime
import json
import influxdb
from influxdb import client as influxdb

db = influxdb.InfluxDBClient(host='127.0.0.1', port=8086, username='root', password='root', database='portcount')

portcount = {}
with open(r'portcountlog.csv', 'r') as h:
    for line in h:
        log = line.strip('\n').split(',')
        loglist = []
        logentry = {}
        logentry['measurement'] = "portcount"
        logentry['time'] = log[0]
        logentry['fields'] = {}
        logentry['fields']['ports'] = log[1]
        loglist.append(logentry)
        print (loglist)
        db.write_points(loglist)

portcount.py:

# !/usr/bin/env python
from __future__ import print_function
from functools import wraps
from pprint import pprint
import sys
import requests
from datetime import datetime
import json
import influxdb
from influxdb import client as influxdb


# Define URL and Params for pulling CIM JSON data

url = 'http://api.website.com/api/ports/getPortActivations'

# Create variable called "resp" containing data from response we get back

resp = requests.get(url)
data = resp.json()

# Create timestamp for InfluxDB
current_time = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%SZ')

# Extract the "count" from API and store as variable.


portcount = [
        {
        'measurement':"portcount",
        'time': current_time,
        'fields':{
            'ports':data['count']},
        }]

# Define InfluxDB Client Information

db = influxdb.InfluxDBClient(host='127.0.0.1', port=8086, username='root', password='root', database='portcount')

# Write the port count to InfluxDB

print(portcount)
db.write_points(portcount)

I have to be able to load the old data and start updating the database with new data each night. Just not sure what’s going on.

2

@ddevalco That is quite odd. The only thing that looks out of place is the /r on the end of your values in the oldportcounts.py. Can you try this same test without that on the end?

3

The /r doesn’t show in the DB when I view all? I can try but either way I don’t understand why all the previous data would be gone? Seems if the /r was a problem wouldn’t that be a problem with inserting the data?

4

@ddevalco I don’t either. That would be my only question from the above code. Can you open an issue on InfluxDB with this information?

5

Yes sir. Opened one yesterday

1 Like
6

They closed my issue and said it wasn’t the python client.

I have confirmed that the same issue occurs when I manually insert via the InfluxDB CLI.

I didn’t even specify a time stamp. But by adding a new measurement all previous data is lost.

Here is exactly what was done:

[user@testbox scripts]$ python oldportcounts.py
[{'fields': {'ports': '2\r'}, 'time': '2017-03-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '50\r'}, 'time': '2017-03-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '62\r'}, 'time': '2017-03-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23\r'}, 'time': '2017-03-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-03-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '10\r'}, 'time': '2017-03-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4\r'}, 'time': '2017-03-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-03-31T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '42\r'}, 'time': '2017-04-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '20\r'}, 'time': '2017-04-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '6\r'}, 'time': '2017-04-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-04-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8\r'}, 'time': '2017-04-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '16\r'}, 'time': '2017-04-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-04-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4\r'}, 'time': '2017-04-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14\r'}, 'time': '2017-04-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-04-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '22\r'}, 'time': '2017-05-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23\r'}, 'time': '2017-05-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-05-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '1\r'}, 'time': '2017-05-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5\r'}, 'time': '2017-05-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2\r'}, 'time': '2017-05-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60\r'}, 'time': '2017-05-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '12\r'}, 'time': '2017-05-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14\r'}, 'time': '2017-05-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60\r'}, 'time': '2017-05-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0\r'}, 'time': '2017-05-18T07:00:00Z', 'measurement': 'portcount'}]
[user@testbox scripts]$ 
[user@testbox scripts]$ 
[user@testbox scripts]$ influx
Connected to http://localhost:8086 version 1.2.2
InfluxDB shell version: 1.2.2
> use portcount
Using database portcount
> select * from portcount
name: portcount
time                ports
----                -----
1489647600000000000 2
1489734000000000000 50
1489820400000000000 0
1489906800000000000 0
1489993200000000000 0
1490079600000000000 0
1490166000000000000 0
1490252400000000000 62
1490338800000000000 23
1490425200000000000 0
1490511600000000000 0
1490598000000000000 0
1490684400000000000 2
1490770800000000000 10
1490857200000000000 4
1490943600000000000 0
1491030000000000000 0
1491116400000000000 0
1491202800000000000 0
1491289200000000000 8
1491375600000000000 42
1491462000000000000 0
1491548400000000000 20
1491634800000000000 0
1491721200000000000 0
1491807600000000000 0
1491894000000000000 8
1491980400000000000 0
1492066800000000000 6
1492153200000000000 5
1492239600000000000 0
1492326000000000000 0
1492412400000000000 0
1492498800000000000 8
1492585200000000000 16
1492671600000000000 0
1492758000000000000 2
1492844400000000000 0
1492930800000000000 0
1493017200000000000 0
1493103600000000000 4
1493190000000000000 0
1493276400000000000 14
1493362800000000000 0
1493449200000000000 0
1493535600000000000 0
1493622000000000000 0
1493708400000000000 22
1493794800000000000 23
1493881200000000000 5
1493967600000000000 1
1494054000000000000 0
1494140400000000000 0
1494226800000000000 0
1494313200000000000 0
1494399600000000000 5
1494486000000000000 2
1494572400000000000 60
1494658800000000000 12
1494745200000000000 0
1494831600000000000 14
1494918000000000000 60
1495004400000000000 0
1495090800000000000 0

>
> insert portcount ports=22
> select * from portcount
name: portcount
time                ports
----                -----
1495627380301384192 22

>
7

I have eliminated the \r that was showing in my data prior to writing it with the script to rule that out as an issue.

I have also tested manually inserting a new measurement without specifying a timestamp and this issue still occurs.

Really need some guidance and help on what could be going on. I just don’t understand why all the data is being dropped when inserting a new measurement.

[username@testbox scripts]$ python oldportcounts.py
[{'fields': {'ports': '2'}, 'time': '2017-03-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '50'}, 'time': '2017-03-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '62'}, 'time': '2017-03-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23'}, 'time': '2017-03-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2'}, 'time': '2017-03-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '10'}, 'time': '2017-03-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4'}, 'time': '2017-03-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-03-31T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8'}, 'time': '2017-04-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '42'}, 'time': '2017-04-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '20'}, 'time': '2017-04-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8'}, 'time': '2017-04-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '6'}, 'time': '2017-04-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5'}, 'time': '2017-04-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '8'}, 'time': '2017-04-18T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '16'}, 'time': '2017-04-19T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-20T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2'}, 'time': '2017-04-21T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-22T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-23T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-24T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '4'}, 'time': '2017-04-25T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-26T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14'}, 'time': '2017-04-27T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-28T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-29T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-04-30T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-01T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '22'}, 'time': '2017-05-02T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '23'}, 'time': '2017-05-03T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5'}, 'time': '2017-05-04T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '1'}, 'time': '2017-05-05T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-06T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-07T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-08T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-09T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '5'}, 'time': '2017-05-10T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '2'}, 'time': '2017-05-11T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60'}, 'time': '2017-05-12T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '12'}, 'time': '2017-05-13T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-14T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '14'}, 'time': '2017-05-15T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '60'}, 'time': '2017-05-16T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-17T07:00:00Z', 'measurement': 'portcount'}]
[{'fields': {'ports': '0'}, 'time': '2017-05-18T07:00:00Z', 'measurement': 'portcount'}]
[username@testbox scripts]$ 
[username@testbox scripts]$ influx
Connected to http://localhost:8086 version 1.2.2
InfluxDB shell version: 1.2.2
> use portcount
Using database portcount
> select * from portcount
name: portcount
time                ports
----                -----
1489647600000000000 2
1489734000000000000 50
1489820400000000000 0
1489906800000000000 0
1489993200000000000 0
1490079600000000000 0
1490166000000000000 0
1490252400000000000 62
1490338800000000000 23
1490425200000000000 0
1490511600000000000 0
1490598000000000000 0
1490684400000000000 2
1490770800000000000 10
1490857200000000000 4
1490943600000000000 0
1491030000000000000 0
1491116400000000000 0
1491202800000000000 0
1491289200000000000 8
1491375600000000000 42
1491462000000000000 0
1491548400000000000 20
1491634800000000000 0
1491721200000000000 0
1491807600000000000 0
1491894000000000000 8
1491980400000000000 0
1492066800000000000 6
1492153200000000000 5
1492239600000000000 0
1492326000000000000 0
1492412400000000000 0
1492498800000000000 8
1492585200000000000 16
1492671600000000000 0
1492758000000000000 2
1492844400000000000 0
1492930800000000000 0
1493017200000000000 0
1493103600000000000 4
1493190000000000000 0
1493276400000000000 14
1493362800000000000 0
1493449200000000000 0
1493535600000000000 0
1493622000000000000 0
1493708400000000000 22
1493794800000000000 23
1493881200000000000 5
1493967600000000000 1
1494054000000000000 0
1494140400000000000 0
1494226800000000000 0
1494313200000000000 0
1494399600000000000 5
1494486000000000000 2
1494572400000000000 60
1494658800000000000 12
1494745200000000000 0
1494831600000000000 14
1494918000000000000 60
1495004400000000000 0
1495090800000000000 0

> 
> insert portcount ports=22
> select * from portcount
name: portcount
time                ports
----                -----
1495645068147793446 22

>
8

I just tried a sample of what you describe from the command line using a recent version of the OSS master. Can you tell us what version of influxdb you are running?

> create database ddevalco
> use ddevalco
Using database ddevalco
> insert portcount ports=0 1495090800000000000
> select * from portcount
name: portcount
time                ports
----                -----
1495090800000000000 0

> insert portcount ports=22 1495645068147793446
> select * from portcount
name: portcount
time                ports
----                -----
1495090800000000000 0
1495645068147793446 22
9

A field type mismatch may be the root cause here.

[{'fields': {'ports': '2'}, 'time': '2017-03-16T07:00:00Z', 'measurement': 'portcount'}]

I don’t know how the Python client works, but that looks as though ports is being written as a string, not a number.

@ddevalco can you reproduce the issue if you emit raw line protocol, rather than going through the Python client?

You can emit line protocol to a file, first couple lines for you should look like

portcount ports=2i 1489647600000000000
portcount ports=50i 1489734000000000000

And then you can write it with curl:

curl -XPOST http://localhost:8086/write?db=YOUR_DB_NAME --data-binary @/path/to/lineprotocol.file