Tiring week, good weekend.

So, as the plane carrying our hero landed in Chicago from the Seattle trip, I turned on my phone. In time to get voice mail from one of my co-workers telling me that one of my clients' servers had crashed. Hard. Nothing I could have done from where I was, but I felt responsible anyway, even though it was a hardware failure. That occured on the Wednesday before Thanksgiving. Two people in their office smelled smoke in the server room, but didn't call us to see if we could find out what was going on. Turned out it was one of the drives in the RAID array. Unfortunately, it looks like the drive died, but kept spinning and producing heat. Because on Monday around noon, they were having some odd issues and rebooted the server. Which then reported having no OS. Since a second drive in the array had then seized up as well.

My co-worker spent a bunch of time at the site on Monday, then I went over and met him on Tuesday morning. I worried about the situation all night. But I also had a pre-scheduled appointment at another client site on Tuesday morning, and couldn't reschedule it. So I took care of that first. And got there to find that there were problems with the array. My co-worker had replaced the failed drives, but in the process, Adaptec (who makes the RAID controller) told him to upgrade the controller's BIOS. Unfortunately, it turns out that the newer version of the RAID BIOS doesn't work with the BIOS on the server's motherboard, causing problems when trying to install the OS. We eventually figured that out. Then I got to spend the rest of the day finding out that when your server is a Active Directory domain controller, you shouldn't re-add it to the domain with the same name. Unfortunately, we later found out that when you're rebuilding an Exchange server (which this server ALSO was), you're supposed to rebuild it with the same name, to make restoring the Exchange directories work properly.

I spent about 16 hours there on Tuesday and caught the last train out of downtown. That train gets me home at just after 2 in the morning. I had a client call at 8:30 the next morning. It was not pleasant. Fortunately, my boss let me go home early that day, and I spent much of the afternoon asleep, catching up.  My co-worker was there all day Wednesday and Thursday trying to get Exchange reinstalled and working, due to poor documentation from Microsoft. By the end of the week, we were both pretty burned out. It was up and running on Friday, but there are still issues. The backup is now failing, with errors indicating that it's overheating during the backup.

Also on Tuesday night, we found out that the houseguests that we were expecting to arrive on Thursday night were in fact arriving on Wednesday night. Alyse spent a bunch of the night cleaning, but we were still cleaning on Wednesday night when they got here. But they were fine with the mess. Jeff and Jesse were here for the weekend because they were showing their wares at a show called "One of a Kind" at the Merchandise Mart downtown. J&J make harps, and they are gorgeous harps. They're pretty damn cool people, too. They stayed with us Wednesday through Saturday night, then headed home after the show closed on Sunday night. But Friday night we sat around playing games with a couple of other people, then Saturday night, the four of us went to dinner with my father, who's a huge musical instrument buff, and he and Jeff and Jesse talked about instruments all through our (yummy) sushi dinner. A nice end to a day where I spent much of it asleep.

Sunday morning alymid worked on a project with her mother, and once that was done, we picked up free tickets to the One of a Kind show and went downtown to see what was there. We saw some ok stuff, and some really cool stuff, and spent too much money, but came away happy.

All in all, a very good weekend. I had a good time.