Ok this has been a rough month for us...here is what happened:

Our service provider was upgrading our connection which required turning off our server and moving it physically.  With short notice, we were unable to back everything up before this move was being made.  Once they plugged everything in after the move, the system would not boot up...hard drive failure : (.  This was pretty much worse case scenario because our back up was not complete. Thankfully we were able to back up the entire database, so really the only thing we lost where some photos and videos, everyone’s profiles themselves where fine and all of your info such as forum posts, friends lists were all preserved. We're not trying to say that losing photos or videos is ‘ok’, however it could have been a lot worse for the site. We have looked at getting the data off the dead hard drive but it simply isn’t an option for a few reasons...mainly because the drive is dead : ) AND manual data recovery from a dead drive costs about 0-0 bucks, which we don't have :).  So instead, we had a burial out in Seattle (a.k.a throwing it in the ocean and cursing at it flying through the air).  Even in the worse situations...we have to maintain some sense of humor :P.
 
We took a few days to recoup and see what the best thing for us to do given the situation at hand. Seeing as the last full and true back up was done right after Christmas, we decided to revert back to that date for photos and videos.  This means we lost about two and a half weeks of photos / videos.  We also decided to take a different approach with our server as to what we had before. We had to rebuild everything from the ground up, because the other drive is gone, so we truly started from scratch. This has allowed us to tailor the server for EXACTLY what we want it to do and will be able to optimize it accordingly.

Having learned our lesson with Disaster Recovery, we now have some plans in place.  This was part of the reason we took a couple weeks to get things right.  We ended up setting up a VPS as a secondary server which will be a clone of the live server and database.  This will serve as an emergency fully-configured version of the site with minimal loss of data which we can restart with more capacity while we rebuild.  We also have a plain backup of the site, database, and configuration which gets pushed nightly to Amazon S3.  So that gives us two fall-back plans, so we're not down for 2-3 weeks again.

While it all sounds simple there will be a few things we have to work out and unfortunately we cannot test that particular thing unless the main server goes down so we will never truly know exactly what will happen until, or unless, it happens. We cannot tell you how unfortunate and how unexpected this whole situation is and that we are doing everything we can to help prevent situations like this from happening in the future.

If you've been following our tweet-like updates, you'll know that the site is going to be slow at first.  Our hosting provider did not have the ability to re-invest in the dead hard drive for the time being, so we were moved to another server with less RAM...but RAIDED hard drives.  This also means that we have less of a chance of data loss due to drive failue.  He did bring us from the original 4 GB in the new server to 8 GB and then we just purchased an additional 8 GB of RAM which is being shipped to our datacenter mid-week (Feb 11th-ish).  This means that the site WILL be a little slow in the beginning as we optimize things AND then it will be slow again after we get our RAM upgrade, so we can reoptimize for twice as much RAM. 

The problem is...I can't optimize the site without traffic, so even if you have the site open...even if it is too slow to browse, you will be doing your part to help us optimize. 

Anyways...I think that is enough for now.  I hope you are all psyched to see the site back and we'll be posting additional updates as things arise.

Regards,

~Chris