Linux RHEL 5 configuring multipathing with DM-Multipath

In my example, I am connected to an IBM DS5020 from 2 IBM blade servers.  The blade chasis is equipped with 2 Brocade 4GB SAN switch modules.  Each blade server has two access paths, provided the server is installed with 2 Qlogic HBA modules.

Currently, I have configured 2 Logical Drives to be presented to each blade server.  Both have access to ASM (LUN0), and independently blade A has access to ORA1, blade B access to ORA2.  By default, multipathd isn’t started. 

The procedure below will walk through the configuration of multipathing.

Here’s the output of fdisk to show what the disks are seen by the system — notice the 4 paths to 50/100GB devices:
# fdisk -l

Disk /dev/sda: 145.9 GB, 145999527936 bytes
255 heads, 63 sectors/track, 17750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         122      979933+  83  Linux
/dev/sda2             123        1166     8385930   83  Linux
/dev/sda3            1167        1688     4192965   82  Linux swap / Solaris
/dev/sda4            1689       17750   129018015    5  Extended
/dev/sda5            1689        6910    41945683+  83  Linux
/dev/sda6            6911       10043    25165791   83  Linux
/dev/sda7           10044       11610    12586896   83  Linux
/dev/sda8           11611       13177    12586896   83  Linux
/dev/sda9           13178       14221     8385898+  83  Linux
/dev/sda10          14222       15265     8385898+  83  Linux
/dev/sda11          15266       16309     8385898+  83  Linux
/dev/sda12          16310       17353     8385898+  83  Linux

Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn’t contain a valid partition table

Disk /dev/sdc: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 53.6 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdd doesn’t contain a valid partition table

Disk /dev/sde: 53.6 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sde doesn’t contain a valid partition table

Disk /dev/sdf: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdf doesn’t contain a valid partition table

Disk /dev/sdg: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdg doesn’t contain a valid partition table

Disk /dev/sdh: 53.6 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdh doesn’t contain a valid partition table

Disk /dev/sdi: 53.6 GB, 53687091200 bytes
64 heads, 32 sectors/track, 51200 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdi doesn’t contain a valid partition table

Setting up DM-Multipath
Edit /etc/multipath.conf and comment out the ‘blacklist’ section
#blacklist {
#        devnode “*”
#}
Save the file and execute the following commands:
modprobe dm-multipath
service multipathd start
multipath -v3  # will print out all multipathed paths
chkconfig multipathd on

Since the value of “user_friendly_name” is set to “yes” in multipath.conf, the multipath devices will be created as /dev/mapper/mpathn.
# ls -l /dev/mapper/
control  mpath1   mpath2

From above, we see that /dev/sda is the internal disk and should be ignored from the configuration.   Edit /etc/multipath.conf again and add the following stanza:
blacklist {
        wwid 3600508e000000000b9cfe9bbf3879804
}

Notice, the WWID is used and not /dev/sda since the devnode type can change during bootup and this would be dangerous.  To get the WWID associated with the device names, run:
multipath -l
Here’s a sample output:
VOTE1 (360060160e49e1600d488b94e3e39e011) dm-12 DGC,RAID 10
size=1.0G features=’1 queue_if_no_path’ hwhandler=’1 emc’ wp=rw
|-+- policy=’round-robin 0′ prio=-1 status=active
| `- 3:0:1:6 sdz  65:144 active undef running
|-+- policy=’round-robin 0′ prio=-1 status=enabled
| `- 5:0:1:6 sdal 66:80  active undef running
|-+- policy=’round-robin 0′ prio=-1 status=enabled
| `- 5:0:0:6 sdo  8:224  active undef running
`-+- policy=’round-robin 0′ prio=-1 status=enabled
  `- 3:0:0:6 sdn  8:208  active undef running

Now run:
service multipathd reload
multipath -F
multipath -v3  # Look for “3600508e000000000b9cfe9bbf3879804: blacklisted” in output

Note, if your device does not show up, it may be because the default supported devices does not include your storage array.  Additional ones can be added by following viewing /usr/share/doc/device-mapper-multipathd-0.4.7/multipath.conf.annotated.

Some common DM-Multipath situations

# You need to resize an online multipath device
First, resize your physical device.
Then find the paths to your LUN:
# multipath -l
mpath2 (360080e500018299c000003444cb493b5) dm-1 IBM,1814      FAStT
[size=50G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:1 sdd 8:48  [active][undef]
 \_ 2:0:0:1 sde 8:64  [active][undef]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:1 sdh 8:112 [active][undef]
 \_ 2:0:1:1 sdi 8:128 [active][undef]
mpath1 (360080e50001820ec000003244caca742) dm-0 IBM,1814      FAStT
[size=100G][features=1 queue_if_no_path][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:0 sdb 8:16  [active][undef]
 \_ 2:0:0:0 sdc 8:32  [active][undef]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:1:0 sdf 8:80  [active][undef]bdf
 \_ 2:0:1:0 sdg 8:96  [active][undef]

Next, resize you paths by causing your SCSI driver to rescan:
# echo 1 > /sys/block/device_name/device/rescan

Now, resize your multipath device:
# multipathd -k’resize map mpath0′

Finally, resize the filesystem (assuming no LVM or DOS partitions are used):
# resize2fs /dev/mapper/mpath0