InfluxQL does not offer ways to alter the structure of InfluxDB metadata (measurements, series, fields, tags, …). This has been a known limitation for a very long time, see for example the following open feature requests:
As a workaround to these limitations, we, at ABC Arbitrage developed infix, a command line tool designed to apply rules to tsm and wal files. infix support rules like DropMeasurement, RenameMeasurement, RenameField, UpdateFieldType, …
infix will scan your data and wal directories and apply a set of rules directlty to tsm and wal files, rewriting them.
We’d like to get feedback from the community. Feel free to download and try the tool. If you think a rule is missing, please fill an issue or open a Pull Request if you want to contribute your own rule.
Will definitely try it in a sandbox next time I’ll have the need.
Way better than having to import / export data in line protocol. The only drawback is the service interruption.
Thanks @octal, looks very cool and definitely a highly useful tool for working with Influx!
I hope you don’t mind if I ask a couple of questions:
As I understand it, the tool can currently rename tag keys, but not tag values. Is doing this also on your radar? I’m not clear on the complexity of the latter, but based on this comment I’m guessing that’s a different ball game.
Quite often we find ourselves in a situation where we’d like to update field types between float and int. For example, if the first value for a field that should be a float was (accidentally) inserted as int type, from that point onwards it’s impossible to insert float values*. If I use your tool to change the type of that field from int to float, does the underlying historical data also get recast (e.g. 5 => 5.0)? What happens if recasting is not possible, e.g. I’m trying to change a string field that contains “hello” to a float?
Yes, the tool supports renaming tag values as well. See the rule UpdateTagValue. However, I did not plug it in the configuration system yet (it was added after). You’re right to point it out, I’ll add configuration for the rule and update the README accordingly. It’s missing tests as well
Yes, we also had the same issue. You have to truncate shards to start a new shard with a new field type (but I’m not sure if it’s possible with OSS). If you use the tool to update the type of your field, it will also cast the historical data. Basically, if you’re configuring the rule to cast from int to float, any value of type int will be cast to float. a bool value won’t get cast, for example. If you want to apply the rule to a single shard, you can use the -shard command-line parameter with the shard id you want to update. If recasting is not possible, for example if you try to cast non-integer string to an int, it will return back an error and stop there, leaving the shard with inconsistent value types I guess. Let me know if you have a better idea (maybe dropping the value ?)
Thanks @octal , that sounds really great! In particular that you’ve managed to crack the tag value renaming question. Will give it a try.
Also cool that the values get recast when the field type changes. I don’t have any better ideas than what you’re suggesting in cases where casting isn’t possible. For us in 99% of cases it’s the int → float change I mentioned.
One more question if I may: the README says
Optional: rebuild the TSI index
If you configured infix to drop or rename measurements or series, make sure to rebuild your TSI index if you are using the tsi1 index type.
So I’m assuming that a tag renaming will always involves a TSI rebuild (since we do use tsi1 index type), but a field renaming or type change for example doesn’t require that?
So I’m assuming that a tag renaming will always involves a TSI rebuild (since we do use tsi1 index type), but a field renaming or type change for example doesn’t require that?
Yes, TSI only indexes measurements and tags. Fields are indexed on a per-measurement per-shard basis in a special fields.idx file (unless I’m missing something). infix will automatically update the fields.idx when needed (when renaming or updating a field). However, if you rename a measurement or tag, you’ll have to rebuild the index.
Hello, it looks like a great tool! I have tried it with some of the examples and is impresive!
Just one thing I am really missing… I am in need of using the UpdateTagValue rule, @octal are you linking it to the configuration and updating the README in the near future? @svet if you managed to crack it… could you share it or pull request it?
@octal is infix able to drop series that do not have tags? It looks like infix requires a tags rule for the series drop command. Influx does not require series to have tags. I have some fields I would like to drop but cant seem to figure out what I should put for the where section of the tags rule.
