The system is running on a Raspberry pi 4B (4gb ram, Raspbian linux 10 buster Linux raspberrypi 5.10.11-v7l+), InfluxDB shell v.1.8.4, Python 3.7.3, flashdrive (32GB 100mbsps writespeed). Data is written to the database via python (pandas dataframe with time index and DataFrameClient.write_points()).
After around 10h the first data starts being removed for no reason the retention policy is set on default autogen (168h). The intention is to store 10000 values per second in the database which is working the problem however starts after the 10h mark. This is the point where data is automaticaly removed, after continuous writing without any querys or other strains on the database.
Any suggestions?
thank you in advance!
Hello @Thomas10239,
Welcome! That is super strange. Can you share any logs at the time of deletion? What query are you using/how are you identifying the deletion?
Thanks 
Thanks for the reply,
I think logs are not enabled how can I do this?
The querry entered is:
SELECT first({field}) FROM {dbname}
SELECT last({field}) FROM {dbname}
The first few hours this gives:
I could run my program again with enabled logs, is there anywhere a tutorial on how this is done?
Thanks in advance
After a longer period of time this gives:
From another run, same problem (started at 14u15 April 9 2021) there are no logs from before this I think.
– Logs begin at Sat 2021-04-10 20:17:01 CEST, end at Tue 2021-04-13 10:12:19 CEST. –
Apr 10 20:25:19 raspberrypi systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Apr 10 20:25:19 raspberrypi influxd[588]: ts=2021-04-10T18:25:19.995390Z lvl=info msg=“InfluxDB starting” log_id=0TRGLUUl000 version=1.8.4 branch=1.8 commit=bc8ec4384eed25436d31045f974bf39f3310fa3c
Apr 10 20:25:19 raspberrypi influxd[588]: ts=2021-04-10T18:25:19.996410Z lvl=info msg=“Go runtime” log_id=0TRGLUUl000 version=go1.13.8 maxprocs=4
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.119809Z lvl=info msg=“Using data dir” log_id=0TRGLUUl000 service=store path=/mnt/mydisk/data
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.120498Z lvl=info msg=“Compaction settings” log_id=0TRGLUUl000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.125870Z lvl=info msg=“Open store (start)” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open op_event=start
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.463744Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000169-000000002.tsm id=1 duration=56.357ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.491398Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000177-000000002.tsm id=2 duration=80.552ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.515341Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000161-000000002.tsm id=0 duration=108.265ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.515460Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000185-000000002.tsm id=3 duration=104.746ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.576631Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000193-000000002.tsm id=4 duration=60.703ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.579247Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000201-000000002.tsm id=5 duration=59.454ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.661240Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000210-000000002.tsm id=6 duration=139.628ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.661850Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000229-000000002.tsm id=8 duration=85.054ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.672322Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000221-000000002.tsm id=7 duration=135.632ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.697077Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000237-000000002.tsm id=9 duration=73.306ms
– Logs begin at Sat 2021-04-10 20:17:01 CEST, end at Tue 2021-04-13 10:12:19 CEST. –
Apr 10 20:25:19 raspberrypi systemd[1]: Started InfluxDB is an open-source, distributed, time series database.
Apr 10 20:25:19 raspberrypi influxd[588]: ts=2021-04-10T18:25:19.995390Z lvl=info msg=“InfluxDB starting” log_id=0TRGLUUl000 version=1.8.4 branch=1.8 commit=bc8ec4384eed25436d31045f974bf39f3310fa3c
Apr 10 20:25:19 raspberrypi influxd[588]: ts=2021-04-10T18:25:19.996410Z lvl=info msg=“Go runtime” log_id=0TRGLUUl000 version=go1.13.8 maxprocs=4
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.119809Z lvl=info msg=“Using data dir” log_id=0TRGLUUl000 service=store path=/mnt/mydisk/data
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.120498Z lvl=info msg=“Compaction settings” log_id=0TRGLUUl000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.125870Z lvl=info msg=“Open store (start)” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open op_event=start
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.463744Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000169-000000002.tsm id=1 duration=56.357ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.491398Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000177-000000002.tsm id=2 duration=80.552ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.515341Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000161-000000002.tsm id=0 duration=108.265ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.515460Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000185-000000002.tsm id=3 duration=104.746ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.576631Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000193-000000002.tsm id=4 duration=60.703ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.579247Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000201-000000002.tsm id=5 duration=59.454ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.661240Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000210-000000002.tsm id=6 duration=139.628ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.661850Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000229-000000002.tsm id=8 duration=85.054ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.672322Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000221-000000002.tsm id=7 duration=135.632ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.697077Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000237-000000002.tsm id=9 duration=73.306ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.713710Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000245-000000002.tsm id=10 duration=52.293ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.714187Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/ADC_VALUES/autogen/19/000000246-000000001.tsm id=11 duration=25.649ms
Apr 10 20:25:20 raspberrypi influxd[588]: ts=2021-04-10T18:25:20.976164Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/ADC_VALUES/autogen/19 duration=727.279ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.033088Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/18/000000005-000000002.tsm id=0 duration=41.132ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.032685Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/9/000000004-000000001.tsm id=1 duration=40.599ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.037558Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/9/000000002-000000002.tsm id=0 duration=45.397ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.044135Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/7/000000003-000000002.tsm id=0 duration=49.181ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.070644Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/20/000000002-000000001.tsm id=1 duration=23.255ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.115169Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/20/000000001-000000001.tsm id=0 duration=62.580ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.211390Z lvl=info msg=“Reading file” log_id=0TRGLUUl000 engine=tsm1 service=cacheloader path=/mnt/mydisk/wal/_internal/monitor/20/_00009.wal size=10494557
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.215171Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/7 duration=499.988ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.211583Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/9 duration=496.463ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.222952Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/18 duration=507.785ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.267750Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/14/000000003-000000002.tsm id=0 duration=16.471ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.289167Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/16/000000005-000000002.tsm id=0 duration=36.363ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.299933Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/13/000000003-000000002.tsm id=0 duration=11.737ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.484768Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/16 duration=261.236ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.484833Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/14 duration=269.132ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.485132Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/13 duration=267.973ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.514089Z lvl=info msg=“Opened file” log_id=0TRGLUUl000 engine=tsm1 service=filestore path=/mnt/mydisk/data/_internal/monitor/8/000000004-000000002.tsm id=0 duration=13.257ms
Apr 10 20:25:21 raspberrypi influxd[588]: ts=2021-04-10T18:25:21.595335Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/8 duration=109.883ms
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.087240Z lvl=info msg=“Reading file” log_id=0TRGLUUl000 engine=tsm1 service=cacheloader path=/mnt/mydisk/wal/_internal/monitor/20/_00010.wal size=689298
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.345972Z lvl=info msg=“File corrupt” log_id=0TRGLUUl000 engine=tsm1 service=cacheloader error=“unexpected EOF” path=/mnt/mydisk/wal/_internal/monitor/20/_00010.wal pos=677151
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.418085Z lvl=info msg=“Opened shard” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open index_version=inmem path=/mnt/mydisk/data/_internal/monitor/20 duration=4441.772ms
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.419076Z lvl=info msg=“Open store (end)” log_id=0TRGLUUl000 service=store trace_id=0TRGLU~G000 op_name=tsdb_open op_event=end op_elapsed=5293.213ms
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425833Z lvl=info msg=“Opened service” log_id=0TRGLUUl000 service=subscriber
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425886Z lvl=info msg=“Starting monitor service” log_id=0TRGLUUl000 service=monitor
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425910Z lvl=info msg=“Registered diagnostics client” log_id=0TRGLUUl000 service=monitor name=build
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425930Z lvl=info msg=“Registered diagnostics client” log_id=0TRGLUUl000 service=monitor name=runtime
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425949Z lvl=info msg=“Registered diagnostics client” log_id=0TRGLUUl000 service=monitor name=network
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.425980Z lvl=info msg=“Registered diagnostics client” log_id=0TRGLUUl000 service=monitor name=system
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.429636Z lvl=info msg=“Starting precreation service” log_id=0TRGLUUl000 service=shard-precreation check_interval=10m advance_period=30m
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.429687Z lvl=info msg=“Starting snapshot service” log_id=0TRGLUUl000 service=snapshot
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.429743Z lvl=info msg=“Storing statistics” log_id=0TRGLUUl000 service=monitor db_instance=_internal db_rp=monitor interval=10s
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.430001Z lvl=info msg=“Starting continuous query service” log_id=0TRGLUUl000 service=continuous_querier
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.431768Z lvl=info msg=“Starting HTTP service” log_id=0TRGLUUl000 service=httpd authentication=false
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.431805Z lvl=info msg=“opened HTTP access log” log_id=0TRGLUUl000 service=httpd path=stderr
Apr 10 20:25:25 raspberrypi influxd[588]: ts=2021-04-10T18:25:25.432207Z lvl=info msg=“Listening on HTTP” log_id=0TRGLUUl000 service=httpd addr=[::]:8086 https=false
