Monday, February 1, 2010

How Open Source saved the day. Nuthin' on databases though

This post has no content related to databases, liek MySQL, tuning or anything like that. It is a precursor to such a posting though. And for those of you who have been around a bit and was reading Byte magazine in the good old days, featuring Jerry Pournelles "Computing at Chaos Manor" column, you might recognize a Chaos Manor style pattern here, along the lines of:
- A new printer is bought.
- It doesn't work on the laptop.
- The main computer is hooked up to the printer instead.
- Allowing network access fails as the gateway between the laptop and the main computer doesn't work.
- Gateway is reconfigured, but now the sons computer has no network access.
- While fixing the sons computer network access, the kitchen sink suddenly gets clogged.
- When the kitchen sink is fixed, the laptop suddenly burst into flames.
etc. etc.

Now, for my version of this, which is actually a true story.

It started with me upgrading main main Windows box. I do a lot of photo editing on this box, so there is a lot of disks in it. Was was chaning an old RAID-set of 2 250 Gb disks in RAID-1 (mirror) configuration to 2 1Tb disks, again in RAID-1. This box has had some hardware instability issies before, but it has always fixed itself. This times, after the upgrading the disks, the Mobo was dead.

This was not so bad, as I had wanted to upgrade the configuration of this box for some time now. So I went shopping. I now wanted 64-bit and some more memory, so I ended up buying a new Asus 1156 Mobo, an Intel i7 860 CPU, 8 Gb of DDR3 RAM and an SSD drive for programs and booting.

I immediate learned a lesson: With DDR2 memories, although these were shaky when DDR2 was new, current, most DDR2 within spec will fit. Not so with DDR3 though, they are in the same state as DDR2 when new, which means: Only get DDR3 memories as certied for the Mobo you used. Doh! I had to change from the planned (and bought) OCZ to a new set of Corsair memories. Luckily, memory isn't that expensive anymore.

Secondly, don't beleive everything you hear about SSD disks. Although in theory they are the same as just any plain SATA disk, that is not how Windows see things. My SSD was an 80 Gb Intel S25. If I had known in advance what I was up to, I would have done things differently, and I wouldn't have gone with an Intel drive, probably.

In the event, Windows 7 would let me install on the Intel drive. I called Microsoft support and after a few days they told me that I maybe needed a driver for the SATA controller. That was all bogus of course, but I expected nothing else. The SSD needed a firmware upgrade though, as the particular version in my copy was known to have a bug that would trash a Windows 7 installation (Thanx you Intel and Microsoft for that), but even after that, Windows 7 wouldn't install. The machine was a bit shaky.

I finally figured out how to install Windows 7 on that SSD anyway. I first let a Windows XP install disk do the partitioning and formatting, and interupted the Windows XP install after that, and then I let Windows 7 install on that partition. If you want to try this trick yourself, note that you should not touch either the partitioning or the formatting for this to work.

OK, Windows 7 install, all i well then? No, the thing was very unreliable. I found out then about the DDR3 issue mentioned above. Having gotten myself a new set of DDR3 memories, and which were supported for my Asus Mobo, I was now rocking and rolling? Well, sort of.

In the process of upgrading, I wanted to clean out some stuff in the box, so I had the old bootdisk removed, as this was substituted for the SSD anyway, and another disk that was my "work disk", containing all sorts of programs, my own source code for stuff, some archived data and stuff like that, was also removed from the box.

The machine worked fine now, the Adaptec 1420 SATA-II RAID controlller was alive, and I started building the new 1Tb array. The work-disk was placed in a eSATA / USB external enclosure and I was to access that from the box this way instead of having the disk in the box itself and eventually copy all that stuff to a RAIDed internal or external disk, as this stuff was actually rather important.

When I attached the external enclosure to the box though, there was an issue. The disk did not shouw up in explorer, and in Disk Management I was told this disk was "Dynamic" and the the status was "Invalid". So I had read up on this. A Dynamic disk is the "new" was of setting up disks in Windows, the "old" way is using "Basic" disks. Now, that is how I would like it to be, but not so Microsoft. Dynamic disks has the advantage of being possible to software RAID in Microsoft. In an attempt at not allowing us mere mortals to protect out disk data by RAIDing though, Microsoft decided to keep Dynamic disks for the non-Home users only. Why? If you ask me, for no good reason, except to make life difficult for us Windows users. Now I had a disk that was "dynamic", as it was partitioned on a Win2K box ages ago. Windows XP which is suppoed to be better than Win2K really isn't in this respect, so my other Windows box, a Win XP Laptop, could not work in this disk either, as it was "Status: Invalid" and didn't like this dynamic disk, despite this was a Win XP Pro.

Now Windows has two ways of handling disks, Basic and Dynamic, and although Dynamic is much better, I really recommend against it, as it cannot be used all over, so you cannot easily move one disk from one box to another. And it doesn't stop there of course, sometimes Microsoft does things to incredibly complex for no good reason at all.

A Basic disk can be upgraded to a Dynamic disk, no problems. But a Dynamic disk cannot be converted to a Basic disk, or rather, it can, but all data will be lost. Again, for no good reason, and this I know as this is only about formating the partition, nuthin' else. A dynamic disk contains some metadata about the disk, I guess this is for when you work with disk groups, volumes, software RAID etc. So converting to a Basic disk would require you to just get rid of that. Nope, Windows will not do that for you.

In my case, the Dynamic disk was also flagged in Windows 7 as "Status: Invalid". Googling on this I found numerous issues with this, and only a few replies. In Disk Management help in Windows, this status isn't even mentioned, thank you Microsoft. Although Windows 7 can import a Dynamic disk, it cannot import one with Status Invalid, there is nothing it can do it it actually. In the process of getting access to my disk, and I was getting desperate, I checked out Dynamic Disk Converter. This is a software kit that would do what Microsoft Windows 7 cannot, which is fix the partition table of a Dynamic disk and convert it to a basic disk, without touching the data on the disk. Why Windows 7 cannout do this is beyond me, it's just a partition table in heavens name, why do you insist on overwriting all the data just to change that! And why can't all Windows versions use the same kind of "good" disk setup? Why? But OK, this 3rd party software then can fix this. Turns out it can't. The software sux, and it cannot do it's work with any kind of Dynamic disk, specifically not a disk with "Status: Invalid", in this case it will instead do something much more interesting: Crash, after a completely unintelligible error message. Thank you again Microsoft!

So no disk yet. Now I was really upset with Microsoft and Windows 7 and all stupid people involved with "Dynamic" disks. I will keep all my disks Basic on all systems in the future, and Microsoft has just forced this on me, after Microsoft spending lots of R&D dollars on developing something that is so much better than the Basic disk, but delivering it in such a way that it is just useless and incredibly annoying.

So what did I do? Well, what was left was an Open Source hack called TestDisk. I downloaded this on a Linux box and then I attached the "Status: Invalid" Dynamic USB disk on a USB port on this box. Look here, Microsoft: This is a disk created on an OS provided by you. I want to use it on a box with an OS provided by you. ANd before you ask, I am a paying customer. I have payed for all these OSes, and I have used it in accordance with the licence. And you guys still suck. And you guys cannot handle the disks that I created with your software on your operating systems. Linux and TestDisk had no problems at all! None. The disk was that, as was all the data! I used TestDisk to backup all the data on the disk, easily, all I had to fix was to make sure I had enough disk on the Linux box for the backup. TestDisk found the drive, figured out the partitioning, just like that, and found all the data on the disk. Also, TestDisk allowed me to backup the the files on the disk, to be safe. After this, I had TestDisk rewrite the partition table.

Now, for the scary part: Remounting the disk in my Windows 7 box. And it worked like a charm: The disk was found, it was "Basic" and all the data was intact.

All this is not to say that I dislike Windows per se, or even that I think Microsoft does everything wrong. But here, Microsoft flunked, big time: Come on, let all versions of Windows use all kinds of disk, any way the like. Maybe Microsoft forgot about it, or didn't know, but we, ordinary users, often have more than 1 computer. And we sometimes move disks and data between them. And surprisingly, we sometimes upgrade the OS, even though it seems MS doesn't want us to.

If you want to use the Dynamic to Basic Disk trick with TestDisk, the instructions I mainly followed are here:

Who just has to fix the eSATA port to be happy now