Much of the beauty of dm-multipath in Red Hat is the chance to combine the redundancy and stability of traditional IP networking with the performance of fibre channel connectivity, culminating in a network connection that can withstand the failure of so many points it almost makes the head spin. This can be accomplished by the use of iSCSI and a TOE network interface, a dual port (minimum) FC-HBA, a properly configured dm-multipath, and a storage system that can export block storage across both mediums (IP network and FC..which is most systems, unless you are living in the year 2000).

Here is the basic premise:

1) Fibre Channel connections are your primary block transport. Set the initiator group on the storage unit as normal, export the blocks, and connect LUN's as normal.

2) Set up an iSCSI initiator to the storage unit and export the same LUN out the target of your storage system. Initiate the connection on both network ports out of the iSCSI TOE card.

3) Set up DM-Multipath. Out of the box, DM-Multipath will create a pseudo-device based on the SCSI_ID of each attached LUN and create a round-robin connection to each LUN (in count, you should have 6 total devices - 2 FC primary paths, 2 FC secondary paths, and 2 iSCSI IP network paths).

4) Set the 4 FC paths (primary and secondary) up accordingly with policies that let's the primary path be used during service time, the secondary path be a potential failover path in the event that either of the 2 primary paths are down, and a final policy that sets the IP network connections to the iSCSI target as dual tertiary paths that will be the final path of last resort.

Hang in there with me! I know that this sounds complicated, but over the next few posts, I'll be explaining each step in detail in a way that is easy for even a beginner to understand. I know that much of this sounds like a drunk man trying to explain why he is wearing a pink tutu in a park at 11am, but trust me when I say it is not that difficult.

At the end of this series, you can build a storage network that can withstand a charging rhino running from Godzilla (ok, maybe not a rhino, but certainly a koala bear wreaking havoc munching on FC cables or networks...if you have koala bears in your data center..after all..who doesn't?).

Until then, happy file and block exporting!

I hate it when I have a disk that needs to be removed while the system is still active. If you are sick of having to reboot to remove devices, look no further! Brainstormes U to the rescue!

From the CLI (where all Linux and Unix administration REALLY happens) execute the following:

echo 1 > /block/sys/<devicename>/device/delete

So if you had a disk /dev/sdb that was either a LUN you want to delete, or an attached device (SATA,SAS,IDE,yo mamma) by issuing the command

echo 1 > /block/sys/sdb/device/delete

You will have successfully removed it from the kernel and can unplug, deattach, throw out on its butt, the device in question, all without a reboot.

Ok, I'll take it to the next step and reverse the process:

To add a device on any bus (SATA,SAS,FC-HBA,iSCSI) the reverse is true...with some modifications:

echo "- - -" >/sys/class/scsi_host/host$NUMBER/scan

Where host$NUMBER is the number of the bus you want to scan. (The "- - -" means to look at every channel, every target, and every lun on that host). After you finish, check dmesg for your new device and BAM! You've successfully hotplugged a new LUN,SATA disk, SAS disk, USB disk

Peace!

Booting from SAN has so many benefits, it is sure to become the standard for server engineering. By attaching storage address space to an HBA, the disks can be removed from the internal system and controlled by a tier I storage unit providing many benefits local disk cannot. For instance, should hardware fail on an existing Boot-From-SAN system, moving the disks to a working server is just a matter of re-pointing the WWID to a standby system and booting the address space on the working hardware; no muss, no fuss. From this feature comes one of the kewlest features I employ as often as I can: address space cloning, and thus entire server cloning. By employing storage virtualization (which is a standard feature on pretty much ALL storage devices nowadays...kind of like a CD-ROM or twisty ties) system admins can simply create a clone of storage address space, assign it to an HBA, grip it and rip it, and you are off to the races; a full server provisioning in under 10 minutes...unless you have an hour to waste watching a status bar move from left to right...

The steps are pretty generic amongst all major (and pretty much all minor) vendor storage units:

1) Set you FC-HBA to boot from SAN with a Boot BIOS
2) Attach some storage address space to the FC-HBA's WWID
3) Set the BIOS to boot from the address space assigned
4) Install as normal

To perform the cloning process:

1) Create a snapshot (most vendors have some type of cloning process that does this by default)
2) Clone a new volume from the snapshot
3) Attach cloned volume to the HBA WWID
4) Boot that bad boy

These are pretty high level steps, but really it is as simple as the steps listed above. Of course you have to concern yourself with spindle limitations, bandwidth, and all the other variables required in storage engineering, but is certainly not a process that should be feared because of the stigma on storage virtualization. It is a skill that should be in every worthwhile systems/storage engineer's bag.