Todays Unscheduled Outage (1 February 2009)

[Kudrow]

At approximately 1.45PM AEST, power to the Melbourne datacentre failed. Preliminary reports suggest this is due to a generator failure.

At approximately 3PM AEST, limited power was restored, however this power failed again at approximately 4.50PM AEST.

OA was restored at 6.34PM AEST however the site will continue to run slow until ISPs being to route traffic back through the datacentre and until we run an integrity check on OA to ensure the database is OK.

More information will be posted when it comes to hand.

[Kudrow]

An update:

We are writing to you in regards to an extended unscheduled outage that caused all websites and email to go offline.

At around 2:00PM on Sunday, the Primus Data Centre in Melbourne suffered a building-wide power outage. This outage took many large ISP's and hosting networks offline for several hours until power was restored to the building.

While we were not able to do anything to assist with the outage itself, steps were taken to ensure that when power was restored all services were brought back online as quickly as possible.

Technicians were dispatched to the data centre within minutes of being notified of the outage and were on-site within 30 minutes.
After 2 hours power was partially restored however as a precaution all servers were powered down to avoid filesystem errors from unclean shutdowns.

When word came that power had been fully restored our technicians restored servers one by one and checked for any console errors before moving on to the next. Within an hour, all shared hosting and VPS nodes were back online with only a handful of minor problems being reported that have since been resolved.

Throughout the outage, we maintained communication with customers by a telephone voice recording and via third party forums. We also had our own forums back online as quickly as possible to ensure customers were being kept up to date with the latest information.

Customers using our premium email service had their mail being captured on our Sydney backup service which was delivered once the main server was back online. Customers using standard cPanel hosting and email should see their messages coming through in the next 24 hours as the senders mail server resends queued messages.

A full report will be issued tomorrow once we know from our suppliers the details of what happened today. You will find this report on the forums.

We sincerely apologise for this unscheduled outage and look forwarding to moving on from it.