- Impact
- 290
Customers of web hosting outfit Gandi.net have been left less than impressed by its handling of a data-destroying storage crash.
The France-based hosting provider on Thursday disclosed that it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the data, however, failed, and there were no snapshots available to recover.
"The storage unit became unavailable, prompting an interruption in service for all PaaS and IaaS services using the disk associated with that unit," the Gandi team said. "The data import on the emergency machine was not possible due to a corruption of the meta-data that we are not aware of the cause of."
The biz went on to say it was conducting a full postmortem of the incident that would yield further details on what exactly went wrong. Its techies are still trying to recover the lost data, and thus far have had no luck.
While the loss of data without a viable backup is bad enough, the response of a business that bills itself as a "no bullshit" outfit is what has some customers up in arms.
Speaking to an aggrieved punter on Twitter, Gandi's web marketing and comms person Julie Pelloille offered the following answer:
Hi Andrea. It is confirmed we have lost data and we are terribly sorry for that. However, please note that what happenend could happen to any web host.
We will provide you with cloud credits enough in order to be able to start a new infra in another datacenter. -- Julie
https://twitter.com/gandibar/status/1215237915770527744
The conversation descended into farce as Pelloille tried to defend the firm's position that its no BS strategy "does not exclude flaws". The response from the customer was as expected, saying: "It's not a flaw, it (sic) YOUR design".
Pelloille tried to lighten the mood by using a memorable moment from fantasy epic Game of Thrones.
This failed natch. The Gandi staffer later apologised, saying they "truly regret[ted] posting it."
A quick browse through the company's documentation shows users able to create back-up copies of volumes on a Gandi Cloud server via Snapshots on a scheduled basis. Some customers had taken this as meaning that Gandi was taking care of those snapshots while they dealt with the likes of databases themselves.
Not so, according to boss Stephan Ramoin, who tweeted: "Andrea, sorry about that and the incident. If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge."
Needless to say, punters were upset with the response and the host's "it could happen to anyone" defense.
"Sure, we can all make mistakes," wrote one user, "but don't blame YOUR OWN major fuckup and not sticking to your promises to any other company!"
We couldn't put it any better ourselves.
Following publication of the article, Gandi made contact to say that around 300 customers were impacted by the incident.
A spokesperson told us:
"We now have some hope that we may recover the data but as we can't confirm it at the moment, customers who needed or need an immediate recovery should use their own backups, as was our initial recommendation."
And how to prevent this from happening again?
"We will certainly give a lot of thought to this question when we complete a full post mortem of the incident, but at this moment our teams are all still focused on restoring customer data.
"In particular, we'll be looking at what improvements can be made to our recovery time, our documentation, and our communications."
https://www.theregister.co.uk/AMP/2020/01/09/gandi_loses_customer_data
gandi.net incident page
https://news.gandi.net/en/2020/01/major-incident-on-our-hosting-infrastructure-in-luxembourg/
The France-based hosting provider on Thursday disclosed that it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the data, however, failed, and there were no snapshots available to recover.
"The storage unit became unavailable, prompting an interruption in service for all PaaS and IaaS services using the disk associated with that unit," the Gandi team said. "The data import on the emergency machine was not possible due to a corruption of the meta-data that we are not aware of the cause of."
The biz went on to say it was conducting a full postmortem of the incident that would yield further details on what exactly went wrong. Its techies are still trying to recover the lost data, and thus far have had no luck.
While the loss of data without a viable backup is bad enough, the response of a business that bills itself as a "no bullshit" outfit is what has some customers up in arms.
Speaking to an aggrieved punter on Twitter, Gandi's web marketing and comms person Julie Pelloille offered the following answer:
Hi Andrea. It is confirmed we have lost data and we are terribly sorry for that. However, please note that what happenend could happen to any web host.
We will provide you with cloud credits enough in order to be able to start a new infra in another datacenter. -- Julie
https://twitter.com/gandibar/status/1215237915770527744
The conversation descended into farce as Pelloille tried to defend the firm's position that its no BS strategy "does not exclude flaws". The response from the customer was as expected, saying: "It's not a flaw, it (sic) YOUR design".
Pelloille tried to lighten the mood by using a memorable moment from fantasy epic Game of Thrones.
This failed natch. The Gandi staffer later apologised, saying they "truly regret[ted] posting it."
A quick browse through the company's documentation shows users able to create back-up copies of volumes on a Gandi Cloud server via Snapshots on a scheduled basis. Some customers had taken this as meaning that Gandi was taking care of those snapshots while they dealt with the likes of databases themselves.
Not so, according to boss Stephan Ramoin, who tweeted: "Andrea, sorry about that and the incident. If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge."
Needless to say, punters were upset with the response and the host's "it could happen to anyone" defense.
"Sure, we can all make mistakes," wrote one user, "but don't blame YOUR OWN major fuckup and not sticking to your promises to any other company!"
We couldn't put it any better ourselves.
Following publication of the article, Gandi made contact to say that around 300 customers were impacted by the incident.
A spokesperson told us:
"We now have some hope that we may recover the data but as we can't confirm it at the moment, customers who needed or need an immediate recovery should use their own backups, as was our initial recommendation."
And how to prevent this from happening again?
"We will certainly give a lot of thought to this question when we complete a full post mortem of the incident, but at this moment our teams are all still focused on restoring customer data.
"In particular, we'll be looking at what improvements can be made to our recovery time, our documentation, and our communications."
https://www.theregister.co.uk/AMP/2020/01/09/gandi_loses_customer_data
gandi.net incident page
https://news.gandi.net/en/2020/01/major-incident-on-our-hosting-infrastructure-in-luxembourg/
