Hi all,
just an update on the Belgium data recovery from Gary:
Quick progress update on the Belgium cluster data recover efforts:
The engineering team is trying to mine restoration data from two different avenues; the first avenue is nearing completion - but we don’t know how complete/accurate that data will be. This is a restoration of data that was in the process of being deleted so it is incomplete data by nature. Those that opened tickets with support to alert them of lost data may see support reach out to them soon with the presentation of this data to see if it is helpful. The 2nd avenue is the one where the hope is to get back the last 90 days of data; this is more promising for being complete (within that 90 days). This one is a much more complicated process to piece together data from individual entries in a backup of Kafka queue data (part of the data processing stream for incoming line protocol in InfluxDB Cloud 2.0). That is super slow going and based on the current rate, may take multiple additional days - but work continues.
As mentioned previously - There has been some success in recovering some tasks, notebooks, etc. Please file a support ticket requesting what we have and support will share with you what we have been able to salvage.
I got a single email on the 23d of Feb and no communication there after.
The process to migrate is totally fine but the engine change is not clear to those who are not aware of the technology change and what that means to the end user.
More communication and documentation would be good and a way to access our data after the date would have been prudent.
Not super happy my greenhouse stopped reporting data all of a sudden but not the end of the world. However as you’ve seen above a lot more important data is lost.
My Questions
Why did we have to create new accounts?
Why were we not able to be transferred to a new region automatically?
Lost creds, config as the account is closed down, again could have been migrated?
Why did we have to create new accounts?
I was able to use my existing account to create a new subscription. I only got the option to select AWS at different location (no Azure/GCP).
Hi @Austin_Barrington,
We are currently working on a TSM to 3.0 document. There have been quite a few changes with regards to the underlining storage engine. If you do not wish to upgrade yet we can provide a TSM account. Please make sure you contact support.
New accounts were created due to a bug within the redirect feature which was solved on the 11th.
With regards to why we couldn’t just move you:
Hi @Jerry1333,
Unfortunately, it’s currently not a simple process for us to merge thousands of users into new clusters. We helped manually move many of our customers who asked to move across to a new cluster. We also have to be careful legally for certain regions such as Belgium as this can be in direct violation of GDPR. Again no amount of excuses or explanations will now solve the damage caused. We are trying our best to help anyone whos been affected to continue with us if they wish to.
Again I am ever so sorry for the data loss. Let me know if you have any further questions.
Hi @aiwa888aa,
Has one of our team reached out to you yet via support? I have made sure internally that the team know about your significant expenditure. Hopefully, we can support you going forward and make amends for the situation
Fortunately, some of our clients have their own backups of the data, so we can request them to restore the data for us. For those who do not have backups, we are currently in discussions regarding possible solutions or compensations.
Hi all,
Just a quick update on the data recovery front:
Another update: Good progress on data recovery today. Our tech support team has started to reach out to GCP/Belgium customers affected to give them instructions for retrieving their data. If you haven’t heard from them already - it is just because they haven’t gotten to you yet or they don’t have a support ticket opened for you. If you are a GCP/Belgium customer who has lost your data - please make sure you have a support ticket opened so our support team knows to reach out to you.
How can company like this just turn off service without absolutely making sure the backups are still in (as happened for australia) is beyond me. Its 101 of running any service, if you discontinue something and turn off servers, you wait for a month before any deleting even comes into consideration… well… Glad to hear at least belgium region customers are covered, hope you are all up and running now.
We were considering moving to InfluxDB Cloud in the next few months, however this incident has severely shaken our confidence. This is something that we can no longer consider, quite apart from data sovereignty issues that we would face with the removal of the Australian region.
The lack of very very common, basic process here is shocking and raises serious concerns about the cloud operation team at InfluxDB. As Alesi and others have mentioned, this is 101 stuff.
