Response from the Hosting company about recent down-time.
We just wanted to keep you all in the loop about yesterdays disaster and bring everyone upto speed on the implications of this over the comming days, the side effects and some other important service information.
Okay so what the hell happened
Just before midnight GMT we suffered a catastropic disk array failure, this isnt usually an issue, the enviroment should be able to carry on without issues, but the failure highlighted a weekness in our vendors setup that exposed the secondary arrays to so much additional work, they simply quit.
Although this counts as a major incident i am pleased to note the downtime was kept to a minamum, the cloud self restored around 80% of the total Virtual Machines, we had to bring the rest up by hand, during the night the cloud performed faultlessly there has been some excessive I/O wait as the arrays were rebuilt and data restored the data. We did however loose around 60 GB of data to corruption, NO WALLETS WERE LOST OR HARMED or anything like that, for the most part the corruption was limmited to our internal machines.
What are we going to do about it
Well were pulling out of OVH as soon as its practiacally possible to do so, we are currently working with several big companys to provide a better more reliable and more resilient platform.
Yes we missed our 60 minute window, all active customer will recieve a store credit over the weekend to help make up for the issues
IF I COULD JUST ASK EVERYONE TO BEAR WITH US WEVE BEEN FLAT OUT GETTING THE PLATFORM AND CUSTOMERS RESTORED TO SERVICE, WE HAVE A MASSIVE AND I MEAN MASSIVE TICKET BACKLOG RIGHT NOW PLEASE REFRAIN FROM TICKETS UNTILL SUNDAY UNLESS IT IS VERY URGENT, IE SERVERS OFFLINE, NOT RESPONDING ETC