How receive more debug info when influx fails to import valid data

InfluxDB 2
1

hallo, please is there a way to receive more info from influx import

example_1:

command and data format is valid, so it returns correctly code 204 but let’s say because time error, it quietly does not import and return no warning

curl --cacert XXX --request POST “https://XXX:8086/api/v2/write?org=foookin_paavel&bucket=panaque&precision=ms” --header “Authorization: Token TTTTT” --data-raw “humidity,host=komar,HumSenKey=16,HumUlId=50x29196a980 HumDecimal=51,HumKey=23326 1622549297766”

here the problem was that timestamp was +2h ahead (as i forgot that my server i set to different time zone, so probably there is no way to write future timestamp)

example_2:

same command but PRECISION is missing. and as influx default is NS it again receive valid data with 204 response, but does not import due wrong time precision

curl --cacert XXX --request POST “https://XXX:8086/api/v2/write?org=foookin_paavel&bucket=panaque” --header “Authorization: Token TTTTT” --data-raw “humidity,host=komar,HumSenKey=16,HumUlId=50x29196a980 HumDecimal=51,HumKey=23326 1622549297766”

thank you, pavel

2

For example 2 , there should be something output from stdout of your influxd process (mine runs as systemd service so it goes to var/log/messages and the journal) . There should be a message in there if a record cannot be inserted.

For example 1 , although unusual, I don’t believe it’s wrong to do this. There are use cases (perhaps reprocessing historical data) where inputting a timestamp different to “now” may be necessary. Agree that a “future” timestamp should probably emit some kind of warning at a minimum. Perhaps one of the staff can comment on this further. The rule of thumb through is for all client processes inserting data to convert the timestamps into Unix time (UTC) Line protocol | InfluxDB OSS 2.0 Documentation

3

hallo @FixTestRepeat, i have been also looking to logs yesterday but here i my today tests:

  • my time stamp is always unix time / i just use MS instead of NS

  • with past date i have no issues as for tests i am importing tons of data, more then one year old, especially when moving influx_1 data to influx_2

  • i have created a test bucket ‘jjj’ to egrep faster in logs. but i do not see any error/warning

  • my config.toml is set to log-level = “debug”

  • see the journal logs below

#no precision set / same curl command as in previous post
#the record cannot be found with query
Jun 02 07:41:42 komar influxd[29321]: ts=2021-06-02T07:41:42.990122Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36220 user_agent=curl took=82.900ms

#json_log_version
{
“_COMM” : “influxd”,
“_PID” : “29321”,
“SYSLOG_IDENTIFIER” : “influxd”,
“_TRANSPORT” : “stdout”,
“_EXE” : “/usr/bin/influxd”,
“_SYSTEMD_CGROUP” : “/system.slice/influxdb.service”,
“__REALTIME_TIMESTAMP” : “1622619702990192”,
“_STREAM_ID” : “d1b736cae0744e5da0d78f78ff27887d”,
“_HOSTNAME” : “komar”,
“_SYSTEMD_SLICE” : “system.slice”,
“_CAP_EFFECTIVE” : “0”,
“PRIORITY” : “6”,
“_SELINUX_CONTEXT” : “unconfined\n”,
“_MACHINE_ID” : “5baeea1b46b14f8a808cb8a6ae19ec7d”,
“MESSAGE” : “ts=2021-06-02T07:41:42.990122Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36220 user_agent=curl took=82.900ms”,
“__CURSOR” : “s=51eb37556ff24ca6ae461506f3737254;i=2173;b=75840e056e5e40f0876f943cd59866ed;m=3c5247878a6;t=5c3c397009570;x=a92feac53a610dea”,
“__MONOTONIC_TIMESTAMP” : “4145255315622”,
“_CMDLINE” : “/usr/bin/influxd”,
“_GID” : “998”,
“_SYSTEMD_UNIT” : “influxdb.service”,
“_SYSTEMD_INVOCATION_ID” : “03e11acb1df3466c881fbade1b6e17b1”,
“_BOOT_ID” : “75840e056e5e40f0876f943cd59866ed”,
“_UID” : “998”
}

#precision set to MS
#record can be found
Jun 02 07:47:01 komar influxd[29321]: ts=2021-06-02T07:47:01.299091Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel&precision=ms” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36270 user_agent=curl took=23.715ms

{
“_SYSTEMD_UNIT” : “influxdb.service”,
“_COMM” : “influxd”,
“_CAP_EFFECTIVE” : “0”,
“_UID” : “998”,
“_CMDLINE” : “/usr/bin/influxd”,
“_SYSTEMD_INVOCATION_ID” : “03e11acb1df3466c881fbade1b6e17b1”,
“SYSLOG_IDENTIFIER” : “influxd”,
“_SYSTEMD_SLICE” : “system.slice”,
“__REALTIME_TIMESTAMP” : “1622620021299161”,
“__MONOTONIC_TIMESTAMP” : “4145573624591”,
“_SYSTEMD_CGROUP” : “/system.slice/influxdb.service”,
“MESSAGE” : “ts=2021-06-02T07:47:01.299091Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel&precision=ms” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36270 user_agent=curl took=23.715ms”,
“_GID” : “998”,
“__CURSOR” : “s=51eb37556ff24ca6ae461506f3737254;i=22c0;b=75840e056e5e40f0876f943cd59866ed;m=3c537717b0f;t=5c3c3a9f997d9;x=caab35cdbb64da68”,
“_TRANSPORT” : “stdout”,
“_MACHINE_ID” : “5baeea1b46b14f8a808cb8a6ae19ec7d”,
“_STREAM_ID” : “d1b736cae0744e5da0d78f78ff27887d”,
“_EXE” : “/usr/bin/influxd”,
“_BOOT_ID” : “75840e056e5e40f0876f943cd59866ed”,
“_PID” : “29321”,
“_HOSTNAME” : “komar”,
“PRIORITY” : “6”,
“_SELINUX_CONTEXT” : “unconfined\n”
}

conan@komar:~$ influx query -o foookin_paavel --skip-verify ‘from(bucket:“jjj”) |> range(start: -24h) |> filter(fn: (r) => r["_measurement"] == “battery_adc” and r.BatUlId == “24xac494d38c” and r.BatSenKey == “24”) |> filter(fn: (r) => r["_field"] == “BatKey”) |> filter(fn: (r) => r["_value"] == 11943) |> sort(columns: ["_time"], desc:true)’ --raw
#group,false,false,true,true,false,false,true,true,true,true,true
#datatype,string,long,dateTime:RFC3339,dateTime:RFC3339,dateTime:RFC3339,double,string,string,string,string,string
#default,_result,
,result,table,_start,_stop,_time,_value,BatSenKey,BatUlId,_field,_measurement,host
,0,2021-06-01T08:04:50.775959032Z,2021-06-02T08:04:50.775959032Z,2021-06-02T07:40:16.017Z,11943,24,24xac494d38c,BatKey,battery_adc,komar

also nothing visible in /var/log/messages

#no record written
572 Jun 2 07:41:42 komar influxd[29321]: ts=2021-06-02T07:41:42.990122Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36220 user_agent=curl took=82.900ms
573 Jun 2 07:42:17 komar influxd[29321]: ts=2021-06-02T07:42:17.039981Z lvl=debug msg=“user find by ID” log_id=0TyV~fGl000 store=new took=0.034ms
574 Jun 2 07:42:17 komar influxd[29321]: ts=2021-06-02T07:42:17.041214Z lvl=debug msg=“org find” log_id=0TyV~fGl000 store=new took=0.048ms
575 Jun 2 07:42:17 komar influxd[29321]: ts=2021-06-02T07:42:17.051980Z lvl=debug msg=“bucket find” log_id=0TyV~fGl000 store=new took=0.124ms

#record ok
847 Jun 2 07:47:01 komar influxd[29321]: ts=2021-06-02T07:47:01.299091Z lvl=debug msg=Request log_id=0TyV~fGl000 service=http method=POST host=XXX:8086 path=/api/v2/write query=“bucket=jjj&org=foookin_paavel&precision=ms” proto=HTTP/2.0 status_code=204 response_size=0 content_length=99 referrer= remote=XXX:36270 user_agent=curl took=23.715ms
848 Jun 2 07:47:05 komar influxd[29321]: ts=2021-06-02T07:47:05.185712Z lvl=debug msg=“user find by ID” log_id=0TyV~fGl000 store=new took=0.034ms
849 Jun 2 07:47:05 komar influxd[29321]: ts=2021-06-02T07:47:05.187727Z lvl=debug msg=“org find” log_id=0TyV~fGl000 store=new took=0.078ms
850 Jun 2 07:47:05 komar influxd[29321]: ts=2021-06-02T07:47:05.203001Z lvl=debug msg=“bucket find” log_id=0TyV~fGl000 store=new took=0.083ms

thank you, pavel