NameSilo

Major incident at gandi.net, some customer data wiped out in storage server meltdown

Labeled as news in Domain Industry News started by aliweb, Jan 11, 2020.

Replies:
15
Views:
881

  1. aliweb

    aliweb Established Member ★★★★★★★★★★

    Posts:
    359
    Likes Received:
    76
    Customers of web hosting outfit Gandi.net have been left less than impressed by its handling of a data-destroying storage crash.

    The France-based hosting provider on Thursday disclosed that it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the data, however, failed, and there were no snapshots available to recover.

    "The storage unit became unavailable, prompting an interruption in service for all PaaS and IaaS services using the disk associated with that unit," the Gandi team said. "The data import on the emergency machine was not possible due to a corruption of the meta-data that we are not aware of the cause of."

    The biz went on to say it was conducting a full postmortem of the incident that would yield further details on what exactly went wrong. Its techies are still trying to recover the lost data, and thus far have had no luck.

    While the loss of data without a viable backup is bad enough, the response of a business that bills itself as a "no bullsh*t" outfit is what has some customers up in arms.

    Speaking to an aggrieved punter on Twitter, Gandi's web marketing and comms person Julie Pelloille offered the following answer:

    Hi Andrea. It is confirmed we have lost data and we are terribly sorry for that. However, please note that what happenend could happen to any web host.
    We will provide you with cloud credits enough in order to be able to start a new infra in another datacenter. -- Julie


    https://twitter.com/gandibar/status/1215237915770527744

    The conversation descended into farce as Pelloille tried to defend the firm's position that its no BS strategy "does not exclude flaws". The response from the customer was as expected, saying: "It's not a flaw, it (sic) YOUR design".

    Pelloille tried to lighten the mood by using a memorable moment from fantasy epic Game of Thrones.
    This failed natch. The Gandi staffer later apologised, saying they "truly regret[ted] posting it."

    A quick browse through the company's documentation shows users able to create back-up copies of volumes on a Gandi Cloud server via Snapshots on a scheduled basis. Some customers had taken this as meaning that Gandi was taking care of those snapshots while they dealt with the likes of databases themselves.

    Not so, according to boss Stephan Ramoin, who tweeted: "Andrea, sorry about that and the incident. If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge."

    Needless to say, punters were upset with the response and the host's "it could happen to anyone" defense.

    "Sure, we can all make mistakes," wrote one user, "but don't blame YOUR OWN major f*ckup and not sticking to your promises to any other company!"

    We couldn't put it any better ourselves.
    Following publication of the article, Gandi made contact to say that around 300 customers were impacted by the incident.

    A spokesperson told us:

    "We now have some hope that we may recover the data but as we can't confirm it at the moment, customers who needed or need an immediate recovery should use their own backups, as was our initial recommendation."

    And how to prevent this from happening again?

    "We will certainly give a lot of thought to this question when we complete a full post mortem of the incident, but at this moment our teams are all still focused on restoring customer data.

    "In particular, we'll be looking at what improvements can be made to our recovery time, our documentation, and our communications."

    https://www.theregister.co.uk/AMP/2020/01/09/gandi_loses_customer_data

    gandi.net incident page
    https://news.gandi.net/en/2020/01/major-incident-on-our-hosting-infrastructure-in-luxembourg/
     
    The views expressed on this page by users and staff are their own, not those of NamePros.
  2. golan

    golan GolanMedia.com Gold Account VIP

    Posts:
    4,175
    Likes Received:
    4,090
    I used to have quite many domains with them, a few years ago. Now only one, purchased because of some sale. Their interface is too complicated, seems they desperately want to seem some kind of mega geek hipsters but in fact only oversmart themselves, it's very unfriendly to domainers.
     
  3. tonyk2000

    tonyk2000 Top Contributor VIP ★★★★★★★★★★

    Posts:
    1,713
    Likes Received:
    2,494
    For domainers - definitely no. Retail pricing and lack of specific tools. Endusers frequently use them as a registrar though, no major issues with the registrar end, the things are working as expected. I've seen and used their interface etc - all good. Moreover, EU jurisdiction is a plus nowadays, at least for some customers. They have 753,819 .com domains as per the last ICANN report - definitely not the smallest. Their hosting platform - where the issues occured - is another thing, looks like geeks developed it for other geeks :)
     
  4. stub

    stub DNStore.com PRO VIP ★★★★★★★★★★

    Posts:
    24,383
    Likes Received:
    9,651
    Isn't it standard practice NOT to have your registration and hosting with the same company. Because if one of your domains get suspended for some problem with your hosting, you can just move the hosting to another company. Hard to do if both your registration or hosting are with the same company, and your hosting locks down both your domain and your hosting. I'm thinking of the instances where this has happened at GoDaddy, for example.

    I never liked Gandi's offering. Never seemed professional enough for me. It seems to have shown to be true. Although I would NEVER have my registration and hosting with the same company. I much prefer to choose an excellent registrar and an excellent hosting separately.

    I suspect, there will be at least 300 customers who will move their hosting. But when this kind of problem happens and it's handled badly, I would move my account elsewhere, even if I wasn't affected. Once bitten, twice shy. They don't deserve my business. No second chances.
     
    Last edited: Jan 11, 2020
  5. MadAboutDomains

    MadAboutDomains Top Contributor VIP ★★★★★★★★★★

    Posts:
    3,097
    Likes Received:
    1,008
  6. Gube

    Gube NameInvestors.com - Epik.com Epik.com Staff Gold Account

    Posts:
    501
    Likes Received:
    695
    Wow.. Here is the reason why having a proper backup is needed
    + Choosing an host which offers data replication
    (Or handle replication by yourself, clone your instance in other datacenters, Having everything at the same place is not a good idea..)
     
  7. JB Lions

    JB Lions Top Contributor VIP

    Posts:
    15,088
    Likes Received:
    20,228
    Exactly, that was drilled into me early on. Don't reg your domains and host at the same place.

    This is a crazy response from them:
    Which one of us do you want naked? I have to warn I don’t have Cersei’s abs ;) - Julie
    @gandi_net
     
  8. stub

    stub DNStore.com PRO VIP ★★★★★★★★★★

    Posts:
    24,383
    Likes Received:
    9,651
    @JB Lions - I don't have any skin in this game. I'd say everyone. Including those who thought they had backups, but didn't have :)
     
    Last edited: Jan 12, 2020
  9. Embrand

    Embrand Top Member ICA Member Gold Account VIP ★★★★★★★★★★

    Posts:
    1,421
    Likes Received:
    1,112
    Yes, I agree their interface is complicated. I used them a while ago and had to use two different interfaces because one is for individual customers and the other for corporate customers. Some extensions were only offered to corporate customers, so I had to use both.

    The only good thing about them is that they support many ccTLDs.
     
  10. Deathstarr

    Deathstarr Top Member VIP ★★★★★★★★★★

    Posts:
    1,091
    Likes Received:
    109
    I would be pretty upset if that happened. I do keep backups but not as current as I should.
     
  11. sharfab

    sharfab Established Member

    Posts:
    473
    Likes Received:
    883
    "The moral to the story" is, don't register and host at the same company, and keep your own backups...:xf.wink:
     
  12. Recons.Com

    Recons.Com Top Contributor VIP

    Posts:
    6,371
    Likes Received:
    13,028
    I would never rely on my host to have backups for me.

    Also most of those 300 clients are making noise to get some credits and maybe something else.

    Probability of needing backup at exactly that time is very small, yet they all act like they lost something vital. Just go create new ones.
     
    Last edited: Jan 12, 2020
  13. Deathstarr

    Deathstarr Top Member VIP ★★★★★★★★★★

    Posts:
    1,091
    Likes Received:
    109
    it said they would get some credits, but after that they should find another place to host.
     
  14. Bob Hawkes

    Bob Hawkes formerly MetBob NameTalent Gold Account VIP Trusted Blogger

    Posts:
    5,680
    Likes Received:
    14,746
    Thanks for sharing the story @aliweb. It clearly is important for the 300 impacted hosting service customers, but some of the replies seem to be talking about Gandi as a place to register domains, which as I understand it were not impacted.

    I think @Recons.Com hit it precisely with this comment. While some hosting providers have better arrangements than others, you should never rely on any backup that is at your host for anything critical.
    It also of course illustrates, no matter how you handle backups, that they be truly independent device and location wise.

    Bob
     
    Last edited: Jan 12, 2020
  15. Datagang

    Datagang New Member

    Posts:
    5
    Likes Received:
    3
    Good customer service doesn't include the word "but."

    Wrong response: "Yes, unfortunately we lost some hosting data, but you should have had your own backup."

    The right response: "Yes, unfortunately we lost some hosting data. We know how important this data is to you. At this point we're assessing the damage to see what can be retrieved and when e can do that. If you happen to have your own backup we can restore that immediately. If not, we'll keep working to retrieve as much as possible. Again, we are so sorry for this situation and will do our best to make it right."
     
  16. stub

    stub DNStore.com PRO VIP ★★★★★★★★★★

    Posts:
    24,383
    Likes Received:
    9,651
    Its a lot longer and lets you down more slowly. I like it much better. But I still hate the occurrence of the incident. But I have no skin in the game :)
     
    Last edited: Jan 19, 2020

Want to reply or ask your own question?

It only takes a minute to sign up – and it's free!
Topics / Tags:
NameWorth
  1. NamePros uses cookies and similar technologies. By using this site, you are agreeing to our privacy policy, terms, and use of cookies.
    Dismiss Notice
Loading...