InfluxDB v2 Changing Fields Datatype

InfluxDB 2
, , , ,
1

I am working on a tool to help convert datatypes in InfluxDB v2 but am running into an issue that I’m hoping someone from InfluxDB or the community is able to solve.

The tool begins by reading, converting, and saving the original Field to a .lp file on the client. I then delete the original Field using InfluxDB’s C# API. We then try to re-write the original Field with the same measurement/field name but am running into an issue where the query engine acknowledges that the original Field has been deleted, but the write engine doesn’t. It creates a type conflict since InfluxDB see’s floats when I’m trying to write int’s.

I’m 100% aware InfluxDB Fields are not designed to be changed like this.

With that said, I am wondering 1. What exactly causes this? Fields.idx file? What exactly is holding onto the deleted Field/type? and 2. is there any manual override or tricks to get the InfluxDB server to compact or process this change on command?

The requirements I have + InfluxDB’s constraints make this near impossible. Wondering if there is a solution that isn’t obvious.

2

How are you deleting the original field? What does that mean? Are you deleting the point (tags, fields, timestamp, with a DELETE?

3

Hi David. Sorry for not being more clear in the original post.

  predicate.AppendFormat("_measurement=\"{0}\"", influxTag.Measurement);

  foreach (KeyValuePair<string, string> tag in influxTag.Tags)
  {
      predicate.AppendFormat(" AND {0}=\"{1}\"", tag.Key, tag.Value);
  }

  mClient.GetDeleteApi().Delete(startTimeUTC.UtcTime, endTimeUTC.UtcTime, predicate.ToString(), mConfig.Bucket, mConfig.Organization);`

This is exactly how I am using the C# API to delete “Fields”. I am essentially deleting the entire series, measurement + tags + fields. It is bound by the start/end time, which I ensure includes every datapoint inside of InfluxDB for the specific series.

4

I believe fields are only deleted from the fields.idx when their containing measurement is deleted. To change a field type, try the following:

For each point with the old field, write the point again with the same tag set and timestamp, but a new field of correct type, and the zero value (empty string, 0, 0.0, whatever) for the old field.

Direct your queries to use the new field; the old field will remain, but because it is universally now the zero value for its type, it will compress very well to take minimal space.

Remember, the overwriting point doesn’t have to contain any fields other than the new one and the old one; InfluxDB will merge other field values with the same tag and time into a single point.

Alternately, you could rewrite the whole measurement into a new measurement, but if you have lots of fields per point, that may be a lot of IO.

5

I am unfortunately unable to rename the field due to a schema/framework we have setup to support InfluxDB in a few applications.

I suppose figuring out a way to support multiple fields, the way you explained, would be the cleanest solution. That unfortunately would be a large change and impact tons of queries and other “rules” we have setup for our system.

I appreciate the tips and will look further into supporting multiple fields as that’s likely the direction we need to go with InfluxDB v2.