Hi there,
My goal is to store 2 log files (each file contains maybe around 200k lines) into InfluxDB and view both the logs all in one single interface. These log files are currently saved in a CSV format, with maybe around 20 DIFFERENT columns for each log. I would like to retain all of these columns (some as indexed tags, and the rest as fields) when I store them into InfluxDB. The columns that will be tagged are the ones that BOTH logs share in common, whereas the ones that will be fields are the ones that differ for both logs. An example of the column headers for one of my log files would be like “ts,type,ID,data,location,flag,direction,…”, which consists of a mix of numbers, strings, booleans, so it’s not exactly like a typical textual log where there is just a timestamp and a text message.
I am aiming towards something like the Log Viewer feature in Chronograf 1.6+ but because that is only limited to the syslog plugin, it isn’t very applicable to my log files (with 20 differing columns each).
TICK stack
Using Chronograf, I can just use a basic InfluxQL query such as “Select * from db.autogen.a, db.autogen.b”, and display the results in Table view to show all the logs. However, it’s hard to visualize the text because if each measurement has 20 columns, then the entire query would return 40 columns, and you would have to horizontally scroll to see the information you need. Not to mention that you cannot really use the Query Builder to build a query on two different measurements at the same time (whereas InfluxDB v2.0 can do that).
Another issue I have with Chronograf is where it would display a string stored value like “5e23105” as “Infinity” because it treats it as an exponential and calculates it. Using telegraf and the inputs.logparser plugin, I specifically set all the columns to a string type, but Chronograf is automatically converting the string values (grey colored font) to Integer/Float (blue colored font) wherever possible.
InfluxDB v2.0
Trying out InfluxDB v2.0, it seems that the built-in Data Explorer doesn’t display the “Infinity” for such values. However, unlike Chronograf, there is no horizontal scrolling, so I cannot read anything from the Table because all the columns are resized to fit the page.
Also, in InfluxDB v2.0, if I want to display all my columns (combination of tags and fields), I would have to use the pivot() function in Flux to reorganize the data, which requires a bit more data transformation compared to Chronograf, where ALL fields will be displayed as its own column by default.
So far, I am leaning towards InfluxDB v2.0 because in the worst scenario, I can always just download the queried Table as a CSV every time and view it on excel (with horizontal scrolling).
Does anyone know which one would better suit the visualization of logs containing both text and numbers?
The image is split into thirds (since I can only upload 1 image as a new user)
- InfluxDB v2.0 Data Explorer with too many columns, making it unreadable. Can you add a horizontal scroll feature like in Chronograf?
- Flux pivot() function used for displaying each field key as its own column in the Table. Can we do this with Query Builder?
- Possible approach for TICK stack is to store everything in a generic “Message” column, but then filtering will be a bit harder (users will need to know the syntax for regex expressions). Is there any way to customize the current Log Viewer feature?
