Scotty Wakefield blogs about how our Team Foundation Server went down over the weekend. Most likely due to a Windows Update on the SQL Server, but we’re still not sure. And no, before you ask, we don’t have Automatic updates on (who in their right mind would on a production server?)
But in Australia we’ve just had the Commonwealth Games and someone in their wisdom (the same kind of person who would have Automatic updates on me thinks) decided to extend daylight saving by a week (I’m not joking – I can see O/S readers shaking their heads).
In possibly one of the worst Microsoft patches to have been released, there was an update sent to handle this on servers, client PCs and some applications (eg Outlook). We needed to get the update onto some of our servers and so we had to install the update. I don’t know for sure, but I’m thinking it likely the update (or tag alongs that came with it) clobbered our TFS SQL Server instance. We had a bunch of other problems throughout our servers too – some had their time adjusted, others didn’t – all in all a bit of a debacle.
Two days downtime really hurts. Fortunately Scotty managed to keep the dev team working even though TFS was out (and source control, project management and client portal with it). He had it all up before close of business today, but was looking pretty frazzled after two days of frustration.