Three days into one of my businesses, in the middle of the marketing push while we were signing on lots of (paying) users, my business partner inadvertently deleted all the user passwords while messing around in the SQL console. We had no backup (we'd just launched 3 days earlier and automated backups were still on my todo list). I was in the office at my old job still. On the phone, I talked him through how to reset all the passwords again (to something different per user) and how to send a mail to all existing user with the new password. Brilliant spin: we presented it as a security upgrade. Along the way, I got him to do a database dump just in case we screwed something up while trying to fix things.
Two days later, I came back home from work and started working on the startup. By a stroke of luck, I took a db dump just before I started. A few hours later, I deployed some of my changes. Then, a little bit later, I noticed something strange: the total number of users had gone down over the last few hours. How bizarre.
Turns out my business partner had done the db dump in the one folder that got auto-executed every time I did an SQL deployment. So I'd just overwritten the db with a 2 day old version. I took a dump of the db again, restored the backup from 2 hours earlier, then manually added all the missing users back in.
No one noticed a thing - or said anything about it, anyway.
Morale: even the most stupid, huge screw-ups can be successfully covered up if you keep a cool head!
I always say, the difference between a master and an apprentice is not that the master doesn't make any mistakes, but that he recovers from them more effectively.
Two days later, I came back home from work and started working on the startup. By a stroke of luck, I took a db dump just before I started. A few hours later, I deployed some of my changes. Then, a little bit later, I noticed something strange: the total number of users had gone down over the last few hours. How bizarre.
Turns out my business partner had done the db dump in the one folder that got auto-executed every time I did an SQL deployment. So I'd just overwritten the db with a 2 day old version. I took a dump of the db again, restored the backup from 2 hours earlier, then manually added all the missing users back in.
No one noticed a thing - or said anything about it, anyway.
Morale: even the most stupid, huge screw-ups can be successfully covered up if you keep a cool head!