Migration from PI DB to Influx DB


#1

Hi all,

We have to soon perform a migration from PI DB to Influx DB for a project. We were wondering whether there is a tool already available that could ease the process.
If not - which approach would you recommend?
Opt A: Export data in chunks to the Influx Line protocol and later import data from these files
Opt. B: Use REST API and code to iterate data
Are both having the same reliability or we have to handle by a separate check that our data are actually in?
Data Migration is supposed to be a one-time job.
Greetings

Thanks


#2

@Ivo_Andreev Hello! I think for a migration using the first method (Opt A) will be the most straightforward. You would use influx -import to write the data to Influx. I would be interested in hearing about your experience. Especially how you are pulling the data out of the PI DB and transforming it. I know there are other users who would benefit from you sharing your experience.


#3

Greeting @jackzampolin.
I can share no sooner than the time when we have done it. Otherwise it would not be serious. Currently the intention is reading by using .NET code. But the question was whether to directly stuff in InfluxDB or prepare line files. The good thing in OptA is that we will have these files which we could use any time later - like export.


#4

@Ivo_Andreev As I said above I think that writing to files first (Opt A) would be best. This gives you more room for errors and makes retrying easier. Good luck and if you have any problems I’ll see if we can help out.


#5

Are you talking about OSIsoft PI? If so, you might consider using piconfig to extract text files… directly on the PI Server node. Be very careful about timestamps… best to do everything in UTC.


#6

FYI: we will migrate a GE historian to influxdb before summer 2018 (500+ measurements, polled every 1 to 10 seconds, running for 7 years).


#7

I forgot to comment on that, but we solved the case somewhere November 2017 and the best solution for us was to use the line protocol. Just be aware of things like time zone and daylight saving time when reading. Keep to UTC.


#8

@Ivo_Andreev… I’m also looking to migrate OSIsoft PI data to an InfluxDB.

QUESTION: I’m curious how you “modelled” the PI digital states (both custom and System digital state sets) in the InfluxDB schema?

Here’s some background…PI Digital states are used to enumerate PI point values into string data and they allow you to associate PI points with a discrete status or state. For example, you can configure a PI Tag to use a custom “equipment status” digital state set that would include only the following enumerated PI point values: Running, Tripped, Idling, Stopped, and Powered Off. Also, there’s a set of default System Digital states that are used by data interfaces that write data to the PI Archive including I/O Timeout, No Data, Under Range, Over Range, Pt Created, I/O Scan Off, Shutdown, Bad Input, etc.


#9

For the digital states, you could store the digital state numerical value and string in two columns.

For the migration, one option is to export from PI to csv. You might want to fill in values at regular intervals because the compression can make the data very sparse.

If you use a client such as the go client, during the import, you need to know the data type of the PI point to cast it to the correct data type. You could write an import script In go. You could then make and use a dictionary that returns the data type for a tag name and cast to the data type during import.

If you store states states with two values such as ON/OFF in PI, you might want to avoid storing them as booleans as they don’t support aggregation functions. You might want to store them as integers such as zero and one.