We have to soon perform a migration from PI DB to Influx DB for a project. We were wondering whether there is a tool already available that could ease the process.
If not - which approach would you recommend?
Opt A: Export data in chunks to the Influx Line protocol and later import data from these files
Opt. B: Use REST API and code to iterate data
Are both having the same reliability or we have to handle by a separate check that our data are actually in?
Data Migration is supposed to be a one-time job.
Greetings
@Ivo_Andreev Hello! I think for a migration using the first method (Opt A) will be the most straightforward. You would use influx -import to write the data to Influx. I would be interested in hearing about your experience. Especially how you are pulling the data out of the PI DB and transforming it. I know there are other users who would benefit from you sharing your experience.
Greeting @jackzampolin.
I can share no sooner than the time when we have done it. Otherwise it would not be serious. Currently the intention is reading by using .NET code. But the question was whether to directly stuff in InfluxDB or prepare line files. The good thing in OptA is that we will have these files which we could use any time later - like export.
@Ivo_Andreev As I said above I think that writing to files first (Opt A) would be best. This gives you more room for errors and makes retrying easier. Good luck and if you have any problems I’ll see if we can help out.
Are you talking about OSIsoft PI? If so, you might consider using piconfig to extract text files… directly on the PI Server node. Be very careful about timestamps… best to do everything in UTC.
I forgot to comment on that, but we solved the case somewhere November 2017 and the best solution for us was to use the line protocol. Just be aware of things like time zone and daylight saving time when reading. Keep to UTC.
@Ivo_Andreev… I’m also looking to migrate OSIsoft PI data to an InfluxDB.
QUESTION: I’m curious how you “modelled” the PI digital states (both custom and System digital state sets) in the InfluxDB schema?
Here’s some background…PI Digital states are used to enumerate PI point values into string data and they allow you to associate PI points with a discrete status or state. For example, you can configure a PI Tag to use a custom “equipment status” digital state set that would include only the following enumerated PI point values: Running, Tripped, Idling, Stopped, and Powered Off. Also, there’s a set of default System Digital states that are used by data interfaces that write data to the PI Archive including I/O Timeout, No Data, Under Range, Over Range, Pt Created, I/O Scan Off, Shutdown, Bad Input, etc.
For the digital states, you could store the digital state numerical value and string in two columns.
For the migration, one option is to export from PI to csv. You might want to fill in values at regular intervals because the compression can make the data very sparse.
If you use a client such as the go client, during the import, you need to know the data type of the PI point to cast it to the correct data type. You could write an import script In go. You could then make and use a dictionary that returns the data type for a tag name and cast to the data type during import.
If you store states states with two values such as ON/OFF in PI, you might want to avoid storing them as booleans as they don’t support aggregation functions. You might want to store them as integers such as zero and one.
Greetings, not that there are news, but we managed to do that 3y ago. It was not Influx 2.0 and the important thing is that Influx 2.0 is 30 times slower with Flux, although much more flexible. The challenge in the migration was mostly the creation of the influx line protocol. Now, with 2.0, the import is much easier I think, but line protocol was a pain - we spent few days of trial and error, then visualizing side by side the imported and old data until we managed to make a clean import.
The thing you will miss most from PI would be the data historian and fine grain access control, but we found some solutions - i.e. like a separate measurement for telemetry with context to get closer to the historian features.
Our company offers a commercial product that showcases and leverages the following features and techniques. Contact me if you are interested.
• Writing data to Influx with data compression routines for tens-of-thousands of data points at one-second scan rates.
• Displaying data in trends by using optimized windowing and visualization of the signals (with min/max within each window) to chart months of data with rapid and scalable performance.
• Presents and leverages discrete state text descriptions.
• Displays can graphically mimic industrial processes. The displays incorporate graphical artwork which can dynamically change colors and images based on data point values.
• Searchable metadata to find displays and point properties (e.g. descriptions, zero, span, engineering units, etc.) across a fleetwide implementation of 1 million+ data points and 1000+ dashboard displays.
Also, we provided services to migrate 3+ years worth of 1-second data for 10k+ data points while implementing compression routines to reduce the number of timeseries events stored in Influx. On average we were compressing the data by 90% while keep the RMS error <1%.
Im in the same dilemma to choose between InfluxDB and PI. Would you say that influxdb is mature enough to take over what PI was able to do? Will the cost of revamping the entire workforce skillsets be worth the cost savings?
