Import Data not Working

[perdurabo93]

So I have a large import file of line protocol data that looks like this:

# DDL
CREATE DATABASE cptdata
CREATE RETENTION POLICY oneyear ON cptdata DURATION 365d REPLICATION 1

# DML
# CONTEXT-DATABASE: cptdata
# CONTEXT-RETENTION-POLICY: oneyear

lpardata,partitionName=host1,frame=frame1 phys_cores_used=0.81 1533441720
lpardata,partitionName=host2,frame=frame2 phys_cores_used=0.52 1533441780
lpardata,partitionName=host3,frame=frame3 phys_cores_used=0.01 1533441840
[...]

When I import it I get this output:

$ influx -import -path=myimportfile.influx -precision=s
2018/10/19 14:29:23 Processed 100000 lines.  Time elapsed: 488.612792ms.  Points per second (PPS): 204661
2018/10/19 14:29:23 Processed 200000 lines.  Time elapsed: 982.878739ms.  Points per second (PPS): 203483
2018/10/19 14:29:24 Processed 300000 lines.  Time elapsed: 1.407458905s.  Points per second (PPS): 213150
2018/10/19 14:29:24 Processed 400000 lines.  Time elapsed: 1.874520048s.  Points per second (PPS): 213387
2018/10/19 14:29:25 Processed 500000 lines.  Time elapsed: 2.402448304s.  Points per second (PPS): 208121
2018/10/19 14:29:25 Processed 600000 lines.  Time elapsed: 2.862350222s.  Points per second (PPS): 209617
2018/10/19 14:29:26 Processed 700000 lines.  Time elapsed: 3.324955206s.  Points per second (PPS): 210529
2018/10/19 14:29:26 Processed 800000 lines.  Time elapsed: 3.918903014s.  Points per second (PPS): 204138
2018/10/19 14:29:27 Processed 900000 lines.  Time elapsed: 4.372104188s.  Points per second (PPS): 205850
2018/10/19 14:29:27 Processed 1000000 lines.  Time elapsed: 4.845329892s.  Points per second (PPS): 206384
2018/10/19 14:29:27 Processed 2 commands
2018/10/19 14:29:27 Processed 1001035 inserts
2018/10/19 14:29:27 Failed 0 inserts

But when I go to look and see whats in my measurement, I get nothing:

$ influx -database 'cptdata' -execute 'select count(*) from lpardata' -format 'column'
$

But it DOES create the database and measurement:

$ influx -execute 'SHOW DATABASES' -format 'column'
name: databases
name
----
_internal
cptdata

$ influx -database 'cptdata' -execute 'SHOW MEASUREMENTS' -format 'column'
name: measurements
name
----
lpardata

And when I manually insert data from the shell, it works:

$ influx
Connected to http://localhost:8086 version 1.6.3
InfluxDB shell version: 1.6.3
> USE cptdata
> INSERT lpardata,partitionName=host1,frame=frame1 phys_cores_used=0.81 1533441720
> SELECT COUNT(*) FROM lpardata
name: lpardata
time count_phys_cores_used
---- ---------------------
0    1

So, where the heck is my data if it was supposedly successfully imported?

[vincentzhong]

Anyone has any idea? I ran into the same issue today, couldn’t see the data imported, but manual insert worked. Thanks.

[avi]

I’m having this problem now as well.

[MarcV]

Hi what is the result of

  Show retention policies 

While connected to your database ?

What were the timestamps of your imported data ?

[avi]

> show retention policies 
name      duration   shardGroupDuration replicaN default
----      --------   ------------------ -------- -------
autogen   0s         168h0m0s           1        true
two_years 17472h0m0s 168h0m0s           1        false

And here’s the top few lines of my import file:

# DML
# CONTEXT-DATABASE: ee_fc4_metrics
# CONTEXT-RETENTION-POLICY: two_years

diagrams_published,org=funding_circle,novelty=new avi=3i,diagrams=1i,description="UK System Landscape" 1518134400
diagrams_published,org=funding_circle,novelty=new avi=3i,diagrams=2i,description="Funding Circle App (UK)" 1518480000

[MarcV]

Do these queries return the same ?

 Select count(*) from “ee_fc4_metrics”.“autogen”.”diagrams_published”

 Select count(*) from “ee_fc4_metrics”.“two_years”.”diagrams_published”

[MarcV]

Please also have a look here about shard duration settings when importing data …

Influxdb failing to load all data points via HTTP

Hi all, I have 5,300,000 data points, all in the same series, to be loaded into Influxdb. The data points are split into files of 5,000 lines each. I use a bash script that submits a file load via curl every 30 seconds. It takes quite a few hours to run (!) but to each request I get an HTTP response code 204. When I go into influx and SELECT COUNT(*) FROM “digital” I get the count 654,363 NOT 5,300,000 The database has been created with a duration of 36500d as the data I have is old (201…

[avi]

@MarcV

Do these queries return the same ?

Aha, that solves the mystery! The first returned nothing but the second returned the value I’d have expected. This led me to read up a little more on retention policies and now I understand that when I was running queries in the shell I was unknowingly querying the default retention policy (autogen), whereas my measurements (points?) had been written to a different retention policy, the one that I’d specified.

I think I understand how to proceed at this point. Thank you Marc for your help!

[MarcV]

You are welcome Avi