Hi @RoyalBlock,
If you need any support in data recovery if your using the Belgium region let me know.
Sydney unfortunately. Any updates on accessing data that was in that region?
Unfortunately @RoyalBlock at this time it does look like the data is lost for Sydney. We are still looking into it but I am 90% sure this is sadly the case.
OK, this is reasonable explanation, thank You.
I have the same problem.
No notifications or anything received; only monthly invoices, where nothing was mentioned. I subscribed to a paid solution to retain the data forever in a safe way; so I am very disappointed. No need to worry about your data safety, backup, performance, etc is one of the main reasons for a cloud subscription.
I would like to recover my data from GCP Europe-west. I have contacted support and received an answer “We are currently investigating if data can be retrieved from your account. We will keep you updated as these efforts continue.”
I am also missing current data, because my IOT sensors are still trying to write to gcp-europe-west.
What is the best solution to proceed? I would like to have my historical data back, but also connect my IOT devices again. Do I need to wait for the data recovery?
Hi @webstab,
Again firstly I am ever so sorry for your loss of trust. We are still investigating the cause of why some users didn’t receive notifications. Moving forward let me help you with the next steps:
-
Data recovery is still in process. If you where part of the Belgium customer you will receive the last 90 days of your data. Support will email you directly once this process is ready.
-
To continue using InfluxDB Cloud you have two options. You can either sign up for a new InfluxDB Serverless Account via the account creation screen: InfluxDB Cloud . Note this will sign you up to a 3.0 account. If you are not ready to move to a 3.0 account and wish to remain on TSM let me know and I can look into that for you.
If you have a chance to jump into the community slack that also helps as we can work through any sensitive data topics via DM.
Thanks,
Jay
FYI, it was not possible for us to re-create an account with the same email.
We created a secondary account using address expansion (our main email is info@example.com, and we created an account with the address info+influx@example.com). And that works.
Hi @Julienfastre, when did you create your new account? I believe we corrected the issue this morning. My apologies you where not notified. We had posted this in slack.
My colleague created the new account yesterday morning. It was one of the first measure we take to avoid data loss with the current activity.
Happy to read that you fixed that: we will change the email address later (or not, because it perfectly works like that).
I am going to write to the support to warn them that we are only one customer behind two different accounts.
1 Like
Hi @WingsofSuspensionLif, I am really sorry we let you down. Would you mind still logging a ticket with support? We are also looking into how we refund some of the damage to the users critically affected by data loss.
Hi all,
just an update on the Belgium data recovery from Gary:
Quick progress update on the Belgium cluster data recover efforts:
The engineering team is trying to mine restoration data from two different avenues; the first avenue is nearing completion - but we don’t know how complete/accurate that data will be. This is a restoration of data that was in the process of being deleted so it is incomplete data by nature. Those that opened tickets with support to alert them of lost data may see support reach out to them soon with the presentation of this data to see if it is helpful. The 2nd avenue is the one where the hope is to get back the last 90 days of data; this is more promising for being complete (within that 90 days). This one is a much more complicated process to piece together data from individual entries in a backup of Kafka queue data (part of the data processing stream for incoming line protocol in InfluxDB Cloud 2.0). That is super slow going and based on the current rate, may take multiple additional days - but work continues.
As mentioned previously - There has been some success in recovering some tasks, notebooks, etc. Please file a support ticket requesting what we have and support will share with you what we have been able to salvage.
Will keep you up to date as i know more.
We use AWS ap-southeast-2 (Sydney) and last month pay the service about 3,000USD.
There is nothing left now.
1 Like
Hello, Just to add my experience here.
I got a single email on the 23d of Feb and no communication there after.
The process to migrate is totally fine but the engine change is not clear to those who are not aware of the technology change and what that means to the end user.
More communication and documentation would be good and a way to access our data after the date would have been prudent.
Not super happy my greenhouse stopped reporting data all of a sudden but not the end of the world. However as you’ve seen above a lot more important data is lost.
My Questions
Why did we have to create new accounts?
Why were we not able to be transferred to a new region automatically?
Lost creds, config as the account is closed down, again could have been migrated?
Why did we have to create new accounts?
I was able to use my existing account to create a new subscription. I only got the option to select AWS at different location (no Azure/GCP).
Hi @Austin_Barrington,
We are currently working on a TSM to 3.0 document. There have been quite a few changes with regards to the underlining storage engine. If you do not wish to upgrade yet we can provide a TSM account. Please make sure you contact support.
New accounts were created due to a bug within the redirect feature which was solved on the 11th.
With regards to why we couldn’t just move you:
Hi @Jerry1333,
Unfortunately, it’s currently not a simple process for us to merge thousands of users into new clusters. We helped manually move many of our customers who asked to move across to a new cluster. We also have to be careful legally for certain regions such as Belgium as this can be in direct violation of GDPR. Again no amount of excuses or explanations will now solve the damage caused. We are trying our best to help anyone whos been affected to continue with us if they wish to.
Again I am ever so sorry for the data loss. Let me know if you have any further questions.
Hi @aiwa888aa,
Has one of our team reached out to you yet via support? I have made sure internally that the team know about your significant expenditure. Hopefully, we can support you going forward and make amends for the situation
Hi @Jay_Clifford
Yes, we have already made contact.
Fortunately, some of our clients have their own backups of the data, so we can request them to restore the data for us. For those who do not have backups, we are currently in discussions regarding possible solutions or compensations.
1 Like
Hi all,
Just a quick update on the data recovery front:
Another update: Good progress on data recovery today. Our tech support team has started to reach out to GCP/Belgium customers affected to give them instructions for retrieving their data. If you haven’t heard from them already - it is just because they haven’t gotten to you yet or they don’t have a support ticket opened for you. If you are a GCP/Belgium customer who has lost your data - please make sure you have a support ticket opened so our support team knows to reach out to you.
How can company like this just turn off service without absolutely making sure the backups are still in (as happened for australia) is beyond me. Its 101 of running any service, if you discontinue something and turn off servers, you wait for a month before any deleting even comes into consideration… well… Glad to hear at least belgium region customers are covered, hope you are all up and running now.
2 Likes
We were considering moving to InfluxDB Cloud in the next few months, however this incident has severely shaken our confidence. This is something that we can no longer consider, quite apart from data sovereignty issues that we would face with the removal of the Australian region.
The lack of very very common, basic process here is shocking and raises serious concerns about the cloud operation team at InfluxDB. As Alesi and others have mentioned, this is 101 stuff.
1 Like
