Complete data loss of Meigs game world

Update:Relaunch of Meigs is now scheduled for April 19th, early afternoon, Central European Time. This way we can give Kaitak some more time to pick up it's back-log and more timezones get a practical chance of getting onto the server early.

We are incredibly sorry to let you know that our newest game world - Meigs - has experienced a full data loss in yesterday’s downtime. Please find more information and statements concerning the further proceedings below.

What has happened?

In the evening of April 17th, a hardware outage in one of our provider’s data centers had rendered one of our machines - the one which hosts the two game worlds Kaitak and Meigs - inaccessible. Following a standard procedure, the server was rebooted to see whether it would come back up again. For reasons which are unknown to us, this reboot must have caught the server in an unfortunate moment and thereby corrupted the database files of the game worlds.

Are there backups?

Apparently, the hardware failure also corrupted the back-up process which was in progress for Meigs while the outage occurred. We tried to rescue data from the server’s SSDs but so far have only been successful in recovering Kaitak’s data. Long story short: We have a complete data loss and are unable to restore the Meigs game world.

What will happen to Meigs?

Being just about 3 weeks old, Meigs will be restarted just like a regular new game world as soon as possible. The settings will be the same as for the original launch and we will take additional measures against the enormous IPO cheating that was plaguing the server in its third week.

What happens to the credits on the game world?

The credit purchases are collected in a central account management tool, so your credit balance remains unaffected.

Will there be any compensations?

Yes, there will be appropriate compensations. Most importantly, all credits that have been spent on the Meigs game world will be refunded. As such, playing on Meigs will have been essentially free of charge. On top of this we will add 30% additional free credits. This way players which were active for a longer period of time will receive a higher compensation than those who just snuck a peek.

What will be done to prevent this in the future?

Even though this was a classic sequence of unfortunate events we will definitely have to rethink and restructure our processes to keep something like this from happening. We will have to ensure that the failure of a single server cannot cause a complete data loss like it did in this case and we will take measures to spread our backups across more machines and different data centers.

Current Status

Kaitak is running but has a long backlog. It also runs on regular hard disks right now instead of the faster SSDs it had before, so it will be picking up slower than usual. This will probably be fixed tomorrow, inducing another short downtime. Meigs will be restarted as soon as we have sorted out a compensation plan and all technical issues have been resolved.

Final words

A combination of technical issues in the data center and human error on our side have caused this disaster and we are aware that the loss of the affected player’s airlines hurts a lot. We are extremely sorry about all of this, but unfortunately we will have to accept the situation as it is. As such we can only beg your pardon and hope that you will stick with us either way. Thank you very much for your understanding and loyalty. We promise to work hard on avoiding such a problem in the future.