Writing Logs manually into InfluxDB

Store
1

Hi,

is it possible to write logs (in the syslog format) straight into the database? Like writing a new point into the database? I have tested so far with the InfluxData API and was able to write information into the database
but not logs.

I haven’t found any information on this and I wonder if there is a possibility without using telegraf.

Regards,
Julian

2

Hello Julian,

I’m not sure I entirely understand what it is that you want to do. You could convert your logs to line protocol and then you could write data from the txt file that contains your logs in line protocol as described here. Does this help at all? Why don’t you want to use the syslog telegraf input?

Best,
Anais

3

It is possible to write logs straight into the database for use with the Chronograf log viewer.

Normally Telegraf is responsible for accepting messages in syslog format and converting them to line protocol to be written to InfluxDB. It inserts all syslog messages into a measurement called syslog, which is what Chronograf looks for when it is populating the log viewer with data. The viewer has drop-down menus at the upper right for selecting the InfluxDB instance and database to use.

The schema looks like this:

  • Tags:
    • appname
    • facility
    • host
    • hostname
    • severity (needs to match the syslog severity level keyword to display properly in Chronograf)
  • Fields:
    • facility_code (integer)
    • message (string)
    • procid (string)
    • severity_code (integer)
    • timestamp (integer)
    • version (integer)

Here are a few examples of logs in line protocol:

syslog,appname=myapp,facility=console,host=myhost,hostname=myhost,severity=warning facility_code=14i,message="warning message here",severity_code=4i,procid="12345",timestamp=1534418426076077000i,version=1i

syslog,appname=mysecondapp,facility=console,host=myhost,hostname=myhost,severity=crit facility_code=14i,message="critical message here",severity_code=2i,procid="12346",timestamp=1534418426076078000i,version=1i

You can get more information on things like severity codes and facilities on the syslog Wikipedia page.

You can also use this schema with the Telegraf tail plugin and the grok parser to read logs into InfluxDB from log files without using syslog. We’re working on a blog post that covers this use case in more detail.

1 Like
4

Thanks, that helped a lot!
I was able to send some logs into the telegraf db for testing.
So far so good but the log points aren’t getting visualized.

LogViewer
LogViewer1824×457 37.6 KB

I’ve set in my .NET Application every Tag and Field that’s needed.
I’m using the InfluxData.Net API by the way.

Any idea why it is not getting visualized?

5

App
App.PNG1113×431 12.8 KB

Here is the code I’m using to create the point for the db.

6

Awesome!

The time range in your log viewer is set to now with a -1m range; looks like your logs might be outside that time range.

Another thing to keep in mind is that Chronograf uses the timestamp field to visualize the logs. This is the time that the log was written, as opposed to the internal InfluxDB timestamp for the measurement, which indicates when the log was received. The time the log was written is more valuable for diagnostic purposes and correlation with other metrics and logs.

7

Whoops, i guess that Screenshot was a bit confusing.

LogViewer2
LogViewer2.PNG1814×855 126 KB

Here you can see that although I changed the range it is not showing my logs.
The one green column is from logs that rsyslog sent over telegraf.
I had sent some logs over rsyslog to see which timestamp InfluxDB is using to store syslogs.

8

Hm, I’m not sure why that would be. Could be a bug. I’ll ping someone on the Chronograf team; are you seeing any errors in the console?

9

Thank you!
I’m not seeing any errors in my console so far.

10

I’ve figured it out myself.

I needed to comment out the optional timestamp which was in my code.

LogViewer3
LogViewer3.PNG1406×480 20.3 KB

Now it’s getting visualized!

1 Like