I Couldn't access Undeveloped.com

Working now. Just managed to access the site a minute ago.

undeveloped is working now

https://undeveloped.com/
working fine here now.

It is working now.

Hi All,

We’re back online since 8:05 (CEST). We had a unique and quite remarkable issue on our load balancer last night which took down Undeveloped.com.

Our server monitors didn’t catch up the issue, which is extremely strange and unacceptable. We're still investigating how it's possible that we weren't notified.

At night, at least two of our senior developers get notified on their phones when the site speed decreases or we go down. Last night, this failsafe never kicked in causing the issue to take longer than we're comfortable with.

We've identified the core cause now. As you know we're rolling out SSL on all parked domains with us and last night the certificate generator (which is connected to our load balancer since we are not allowed to generate certificates for non-active domains) all of sudden started sputtering and eventually took down the entire load balancer. This is absolutely not normal behavior so we're researching what the core cause has been and fixing it so we can continue completing generating the remaining certificates.

We have extensive safeguards in place for what happened last night, but the safeguards failed for some reason. We're looking into this as we speak and implementing a fix to prevent this situation from happening again.

Our sincere apologies for the inconvenience and thank you for your patience.

Kind regards,

Reza

Quick follow-up update:

The initiator of the issue was our hosting provider AWS (Amazon). They had a problem with their EC2 VPC network which impacted our service and caused the strange issue at our load balancer hosted with them:

https://phd.aws.amazon.com/phd/home#/event-log?Start time=2018-07-18T10:05:45 to 2018-07-19T10:05:45&End time=2018-07-18T10:05:45 to 2018-07-19T10:05:45&Status=Closed / Completed&eventID=arn:aws:health:eu-central-1::event/AWS_EC2_VPC_NETWORK_HEALTH_INTRA_AZ_ISSUE_2018_07_19_00_43_00_3066&eventTab=details&layout=vertical

The issue began at Thu, 19 Jul 2018 00:43:00 GMT (exactly when our load balancer started showing issues).



Likely because Amazon has had their own server issues due to Prime Day, they couldn't fix the issue vigorously for their clients since our servers did go online again after their quick fix but our load balancer didn't.