Multipath

There is more than one way to skin a cat

© 2010-2015 Dennis Leeuw dleeuw at made-it dot com
License: GPLv2 or later

Index

    1. Introduction
    2. Symmetric vs. Asymmetric
    3. Getting to know the system
      1. Unique Device Naming
      2. Blacklisting all devices
      3. Start and stop multipath daemon
      4. Preconfigured device support
    4. How does multipathd know about our devices
    5. The configuration file
      1. defaults
      2. blacklist and blacklist_exceptions
    6. The multipath command
    7. The multipath daemon
    8. Supporting unsupported devices
    9. The Device Mapper maps
    10. A device list

Introduction

To be able to support failover connections to a device one needs multiple paths to that device. If we have a iSCSI disk connected to an ethernet network and we want to make sure that we can reach the disk from a host even when we disconnect one ethernet cable we need two network cables running from the host to the disk and we need a piece of software that makes sure when one link failes the other one is used. This last piece of software is called multipath on GNU/Linux systems. multipath makes use of the device mapper as discussed in a previous article.
User Spacedmsetupmultipath
libdevmapper
KernelDevice Mapper

To make a full redundant system we need two switches and two paths from every point to every point:

When a HBA driver loads or when the iSCSI subsystem is started, the SCSI midlayer initiates a scan of its bus and detects all assigned storage LUNs through every available path, this means that our Host, in the on the left hand side provided image, is able to see a single disk in the Hardware Enclosure through four different paths. With multipath enabled this means that 4 different disks will be registered by the system:
Block devicePathModeMultipath device
/dev/sdbGreen, Blueactive/dev/dm-1
/dev/sdcGreen, Rozepassive
/dev/sddGrey, Purpleactive
/dev/sdeGrey, Redpassive

This means that /proc/partitions contains four more disks.

This over comes the following failures:

The DM Multipath system consists of a kernel module called dm-multipath.ko or just multipath.ko, the multipath commandline tool to configure multipath devices and multipathd the watchdog that monitors the paths and takes action when paths fail or come back. The multipathd is configured through the /etc/multipath.conf configuration file.

Userspace Applications
ioctl interfacefilesystem interface 
control interfaceblock interface
core device-mapper
mapping / target interface
linearmirrorsnapshotmultipath
 path selectorshardware handlers

Symmetric vs. Asymmetric

There are two different ways in which multipaths can be used. The first option, symmetric, is to use every path to the device to provide for higher through put, the other option is to use one path for access and the other for hot-standby.

Asymmetric arrays devide paths to the same LUN in active/passive groups

Asymmetric arrays assign a LUN to a one of the two storage processors in the system. Only when a problem occurs the LUN is assigned to the secondary processor. This process is known as LUN trespassing. LUN trespassing is achieved by sending a device-dependent trespassing command to the storage array.

The choice for symmetric of asymmetric is dependend on the the provider of the storage array(?).

Getting to know the system

In an ideal world multipathd would start and finds everything on its own and handles everything automatically, but this is not an ideal world. So sometimes we need to help multipathd a bit by providing settings in the /etc/multipath.conf file. Experience has tought me to do as little as possible and have the daemon figure out as much on its own as is possible.

Unique Device Naming

Per default the multipath system will create a dm-X device (where X is a number) as a single device that reroutes I/O to the underlying disks. If you set the user_friendly_names option in the /etc/multipath.conf file an alias is created in /dev/mapper/. The /dev/dm-X device should never be used since it is as volatile as the other device names. The name is handed out on a first come first serve manner and can thus change after a reboot. The /dev/mapper/ device is bound to the WWID of the multipath device and is stable over reboots. Use this name to configure and setup your multipath system.

If you set the user_friendly_names option to no the link in /dev/mapper/ is the WWID of the multipath-device. This is the default.

If you set the user_friendly_names option to yes a /dev/mapper/mpath-N link is created, where N is a number. The mapping between the WWID and the user friendly name is maintained in the /var/lib/multipath/bindings file. If you want your bindings to be identical on all your hosts distribute this file to all your systems. You can even set the bindings_file option in /etc/multipath.conf if you want this file to be placed in a different place then the default one e.g. /etc/multipath.d/bindings.

Our advise is to use the default, since it is the easiest and most uniform way of handling multipath devices.

We start our configuration file with some defaults. The provided values are the default values for these options, so nothing fancy. This section is only provided to give you an idea of how it works. For more details use man 5 multipath.conf:

defaults {
        udev_dir		/dev
        user_friendly_names	no
}

Blacklisting all devices

To prevent any device from being recognized by the multipath daemon we are going to tell the system that all devices are blacklisted. This is easiest done by the following configuration in your /etc/multipath.conf:

defaults {
        udev_dir		/dev
        user_friendly_names	no
}

blacklist {
	wwid "*"
}

blacklist_exceptions {
}
Make sure nothing else is present in the file.

Start and stop multipath daemon

To start the multipathd use on Red Hat based systems:

service multipathd start
and on Debian based systems:
service multipath-tools start

Now we check to see if the multipath daemon is alive:

echo "help" | multipathd -k
the "help" command does not exist, but it will tell you which commands are available anyway. The first line that the daemon returns also tells you the version of the multipath-tools you are running.

To stop the multipath daemon use the before mentioned service command and replace start by stop.

Preconfigured device support

Your multipathd comes already with a lot of supported devices without you knowing about it. And that will change with using the following command:

echo "show config" | multipathd -k
This will provide you with a long list of already supported devices and also a complete overview of all the different sections that can occur in the multipath.conf configuration file. The available sections are:

How does multipathd know about our devices

As you might have seen in the output of the "show config" command, your multipathd knows about some systems. We are doing our tests with a Dell MD3200i system, so we will use that as an example. The configuration shows a configuration for the Dell MD32xxi that looks something like this:

	device {
		vendor "DELL"
		product "MD32xx|MD32xxi"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		checker rdac
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		prio rdac
		failback immediate
		rr_weight uniform
		no_path_retry 15
		rr_min_io 1000
	}

So apparantly the system can detect that this device is present, but how does it do that? Well, multipathd is started after OpenISCSI which means that the iscsid has already made sure the iSCSI devices are available to the system as block devices. If we do some investigation we find the following information in our system:

# cat /sys/block/sdb/device/vendor 
DELL
# cat /sys/block/sdb/device/model 
MD32xxi
That is exactly what is matched with the vendor and product lines from our device definition. Except that our product line has an or (|) and matches thus for both the MD32xxi (iSCSI) and the MD32xx (locally attached) devices.

Since we have 4 paths to our device we have 4 devices:

# lsscsi  | grep MD32xxi
[9:0:0:0]    disk    DELL     MD32xxi          0784  /dev/sdc
[10:0:0:0]   disk    DELL     MD32xxi          0784  /dev/sdb
[11:0:0:0]   disk    DELL     MD32xxi          0784  /dev/sdd
[12:0:0:0]   disk    DELL     MD32xxi          0784  /dev/sde

The trick for multipathd is to figure out which device is actually the same device, but using a different path. The easiest way to figure this out is by comparing the WWID, the world wide unique identifier. The getuid_callout line tells how we should get that information, where %n is replace by the device name. Luckily for us the different distributions use different commands to figure this out. So for Debian-based systems use:

getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
and for Red Hat-based systems use:
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
Running one of the commands for each iSCSI device shows us exactly what is happening:
# /sbin/scsi_id -g -u -s /block/sdb
36842b2b0006b95b20000047f4d533f9c
# /sbin/scsi_id -g -u -s /block/sdc
36842b2b0006b95b20000047f4d533f9c
# /sbin/scsi_id -g -u -s /block/sdd
36842b2b0006b95b20000047f4d533f9c
# /sbin/scsi_id -g -u -s /block/sde
36842b2b0006b95b20000047f4d533f9c
Every device reports the same WWID, which means that each device is actually one and the same device but reached by a different path.

And the smart thing multipath does is tying this knowledge to a single device. Using the commandline tool multipath one can list how multipathd configured the system for us:

# multipath -l
mpath0 (36842b2b0006b95b20000047f4d533f9c) dm-2 DELL,MD32xxi
[size=16T][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=0][active]
 \_ 10:0:0:0 sdb 8:16  [active][ready]
 \_ 11:0:0:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 9:0:0:0  sdc 8:32  [active][ghost]
 \_ 12:0:0:0 sde 8:64  [active][ghost]
Multipath deals with two files: dm-2, our device mapper file, and mpath0, our user friendly name. The last name is maintained in a simple database file stored as /var/lib/multipath/bindings which looks like this:
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpath0 36842b2b0006b95b20000047f4d533f9c
This explains how the system figured out how all the 4 devices are actually one and the same device.

The configuration file

defaults

In the defaults section you can set defaults for all settings used in the devices and the multipaths sections, plus a couple of generic items that can only be set in the defaults section. Configuration options that we would like to mention here are:

udev_dir
The directory where udev creates the device files. Make sure this one is set to /dev.
user_friendly_names
The default is no which means that the WWID is used as the alias for the multipath. If set to yes multipath will create a persistent alias in the form of a /dev/mapper/mpathN based on the WWID. This mapping is maintained in /var/lib/multipath/bindings.
verbosity
Default is 2. Higher values increase the verbosity level. Valid levels are between 0 and 6.

Example:

defaults {
        udev_dir                /dev
        user_friendly_names     yes
}

blacklist and blacklist_exceptions

There are 3 valid options to use in both the blacklist and blacklist_exceptions sections:

devnode
A regular expression that describes the device
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
wwid
The World Wide IDentifier that describes the device
wwid 46832b2b3796b95b2aad0007885150e5f1
device
A device like description as a section that contains the vendor and product lines to describe the device
device {
	vendor "IBM"
	product "S/390.*"
}

Example:

blacklist {
	wwid "*"
}

blacklist_exceptions {
	# Make an exception for our device(s)
	wwid "46832b2b3796b95b2aad0007885150e5f1"
}

The multipath command

With multipath -l you can get an overview of which disks belong to which multipath device:

mpath3 (36a4badb00018e1c50000051c4c450a6b) dm-3 DELL,MD3000i
[size=4.5T][features=0][hwhandler=1 rdac][rw]
\_ round-robin 0 [prio=200][active]
 \_ 8:0:0:2   sdd 8:48   [active][ready]
 \_ 11:0:0:2  sde 65:0   [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 9:0:0:2   sdf 8:80   [active][ghost]
 \_ 10:0:0:2  sdg 8:240  [active][ghost]

The multipath daemon

With multipathd -k one can directly talk to the multipath daemon. It opens an interactive console from which you can escape by pressing CTRL-D

On the console prompt type help to get an overview of the available commands. The command that might be the most useful is reconfigure, which makes multipathd reread the /etc/multipath.conf configuration file.

One can also talk to multipathd directly from the commandline to e.g. look at the available paths. On newer versions you could use the "show status" command (see below), but that is not availble in older versions, what always works is:

# echo "show paths" | multipathd -k
A system with multipaths present should report something like this:
hcil     dev dev_t pri dm_st   chk_st  next_check      
10:0:0:0 sdb 8:16  100 [active][ready] X......... 2/20 
11:0:0:0 sdd 8:48  100 [active][ready] XX........ 5/20 
9:0:0:0  sdc 8:32  0   [active][ghost] XXX....... 7/20 
12:0:0:0 sde 8:64  0   [active][ghost] XXXXXXXX.. 17/20
If you get no output there are no paths. As mentioned before the newer versions (0.4.9) can present you with a status.
# echo "show status" | multipathd -k
The output should tell you if there is a multipath present:
multipathd> show status
path checker states:

paths: 0
busy: False
As you can see the system I ran this on has no device with multipaths.

Supporting unsupported devices

To find out which devices are built-in to your multipath system (Red Hat), use

echo "show config" | multipathd -k
Debian based system do not seem to have default built-in configurations. If someone knows more about this please contact me.

We discussed the devices section in the multipath.conf file before and promised to come back to it. Now is the time. It happens that new devices come to market, which are not yet supported by the installed version of multipath. To overcome this hurdle we can define our own device in the devices section. Before we can do this we need to collect some information from the our system:

The Device Mapper maps

A multipath map is defined as: start length multipath #features [feature1 ... featureN] #dsm [dsm1 ... dsmN] #pathgroups next_pathgroup pathgroupargs1 ... pathgroupargsN

start
Starting block in the virtual device
length
Length of this segment
#features
Number of multipath features. 0 means there are no features and thus the next part in the line is the #handlerargs. The available features ([feature1 ... featureN]) are:
queue_if_no_path
If there are no paths available to the device, queue the I/O operations
#dsm
Number of DSM (hardware handler) arguments, followed by the arguments ([dsm1 ... dsmN]). A DSM (hardware handler) is a module that performs hardware-specific operations when paths are switched.
#pathgroups
The number of path groups, further specified in the pathgroupargsN
next_pathgroup
The next pathgroup to try when the current one fails
pathgroupargs
The pathgroupargs is defined as: pathselector #selectorargs #paths #pathargs device1 #ioreqs1 ... deviceN #ioreqsN
pathselector
Algorithm to use to determine what path to use for the next action in this path group
#selectorargs
Should always be 0
#paths
Number of paths in the path group
#pathargs
Number of path arguments, should always be 1 (only ioreqs is allowed)
deviceN
Block device in a major:minor notation
#ioreqsN
Number of I/O requests sent to this route before switching to the next.

Example, the output of the table command is wrapped for readability, of course it is presented as one long line:

[prompt]# dmsetup ls --tree -o device
data-part1 (253:10)
 |_mpath3 (253:3)
    |- (8:240)
    |- (8:80)
    |- (65:0)
    |_ (8:48)
[prompt]# dmsetup table /dev/mapper/mpath3 
0 9663676416 multipath 0 1 rdac 2 1 round-robin 0 2 1 8:48 1000
						      65:0 1000
                                    round-robin 0 2 1 8:80 1000
						      8:240 1000
[prompt]# ls -l /dev | grep -E '^b.+[[:space:]]8,.*[[:space:]]240[[:space:]]'
brw-r-----  1 root disk   8,   240 Feb 22 18:39 sdp

A device list

devices {
	device {
		vendor "APPLE*"
		product "Xserve RAID "
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "3PARdata"
		product "VV"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "DEC"
		product "HSG80"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker hp_sw
		features "1 queue_if_no_path"
		hardware_handler "1 hp-sw"
		prio_callout "/sbin/mpath_prio_hp_sw /dev/%n"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "COMPAQ"
		product "(MSA|HSV)1.0.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker hp_sw
		features "1 queue_if_no_path"
		hardware_handler "1 hp-sw"
		prio_callout "/sbin/mpath_prio_hp_sw /dev/%n"
		rr_weight uniform
		no_path_retry 12
		rr_min_io 100
	}
	device {
		vendor "(COMPAQ|HP)"
		product "HSV1[01]1|HSV2[01]0|HSV3[046]0|HSV4[05]0"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "(COMPAQ|HP)"
		product "MSA VOLUME"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "MSA2[02]12fc|MSA2012i"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "MSA2012sa|MSA2312sa|MSA2324sa"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -n -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "MSA2312fc|MSA2324fc|MSA2312i|MSA2324i|MSA2000s VOLUME"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "P2000 G3 FC"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "HSVX700"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "1 alua"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "HSVX740"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "HP"
		product "A6189A"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "HP"
		product "LOGICAL VOLUME.*"
		path_grouping_policy multibus
		getuid_callout "/sbin/cciss_id /dev/%n"
		path_selector "round-robin 0"
		path_checker cciss_tur
		features "0"
		hardware_handler "0"
		rr_weight uniform
		no_path_retry 12
		rr_min_io 1000
	}
	device {
		vendor "HP"
		product "P2000 G3*|P2000G3 FC/iSCSI"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 100
	}
	device {
		vendor "DDN"
		product "SAN DataDirector"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "EMC"
		product "SYMMETRIX"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -ppre-spc3-83 -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		rr_weight uniform
		no_path_retry 6
		rr_min_io 1000
	}
	device {
		vendor "DGC"
		product ".*"
		product_blacklist LUNZ
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker emc_clariion
		features "1 queue_if_no_path"
		hardware_handler "1 emc"
		prio_callout "/sbin/mpath_prio_emc /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 60
		rr_min_io 1000
	}
	device {
		vendor "FSC"
		product "CentricStor"
		path_grouping_policy group_by_serial
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "(HITACHI|HP)"
		product "OPEN-.*"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry 18
		rr_min_io 1000
	}
	device {
		vendor "HITACHI"
		product "DF.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_hds_modular /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "EMC"
		product "Invista"
		product_blacklist LUNZ
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		rr_weight uniform
		no_path_retry 5
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "ProFibre 4000R"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1722-600"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "1 queue_if_no_path"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 300
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1724"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "1 queue_if_no_path"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 300
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1726"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "1 queue_if_no_path"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 300
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1742"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1814"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1745|1746"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 15
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1815"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1818"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "3526"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "3542"
		path_grouping_policy group_by_serial
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "2105(800|F20)"
		path_grouping_policy group_by_serial
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "1 queue_if_no_path"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "1750500"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "1 queue_if_no_path"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "2107900"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "1 queue_if_no_path"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "2145"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "1 queue_if_no_path"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "S/390 DASD ECKD"
		product_blacklist S/390.*
		path_grouping_policy multibus
		getuid_callout "/sbin/dasd_id /dev/%n"
		path_selector "round-robin 0"
		path_checker directio
		features "1 queue_if_no_path"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "S/390 DASD FBA"
		product_blacklist S/390.*
		path_grouping_policy multibus
		getuid_callout "/sbin/dasd_id /dev/%n"
		path_selector "round-robin 0"
		path_checker directio
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "NETAPP"
		product "LUN.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker directio
		features "1 queue_if_no_path"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_ontap /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 128
	}
	device {
		vendor "IBM"
		product "Nseries.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "1 queue_if_no_path"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_ontap /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 128
	}
	device {
		vendor "Pillar"
		product "Axiom [35]00"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "IBM"
		product "3303      NVDISK"
		path_grouping_policy failover
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry 60
		rr_min_io 1000
	}
	device {
		vendor "AIX"
		product "VDASD"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker directio
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry 60
		rr_min_io 1000
	}
	device {
		vendor "AIX"
		product "NVDISK"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "1 alua"
		prio_callout "/sbin/mpath_prio_alua /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 60
		rr_min_io 1000
	}
	device {
		vendor "SGI"
		product "TP9[13]00"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "SGI"
		product "TP9[45]00"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "SGI"
		product "IS.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "STK"
		product "OPENstorage D280"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "STK"
		product "FLEXLINE 380"
		product_blacklist Universal Xport
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "SUN"
		product "(StorEdge 3510|T4)"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "PIVOT3"
		product "RAIGE VOLUME"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -p 0x80 -g -u -d /dev/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "1 queue_if_no_path"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 100
	}
	device {
		vendor "SUN"
		product "CSM200_R"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "SUN"
		product "LCSM100_F"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "0"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "(LSI|ENGENIO)"
		product "INF.*"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 15
		rr_min_io 1000
	}
	device {
		vendor "DELL"
		product "MD3000|MD3000i"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 15
		rr_min_io 1000
	}
	device {
		vendor "DELL"
		product "MD32xx|MD32xxi"
		path_grouping_policy group_by_prio
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker rdac
		features "2 pg_init_retries 50"
		hardware_handler "1 rdac"
		prio_callout "/sbin/mpath_prio_rdac /dev/%n"
		failback immediate
		rr_weight uniform
		no_path_retry 15
		rr_min_io 1000
	}
	device {
		vendor "COMPELNT"
		product "Compellent Vol"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker tur
		features "0"
		hardware_handler "0"
		failback immediate
		rr_weight uniform
		no_path_retry queue
		rr_min_io 1000
	}
	device {
		vendor "GNBD"
		product "GNBD"
		path_grouping_policy multibus
		getuid_callout "/sbin/gnbd_import -q -U /block/%n"
		path_selector "round-robin 0"
		path_checker directio
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
	device {
		vendor "EUROLOGC"
		product "FC2502"
		path_grouping_policy multibus
		getuid_callout "/sbin/scsi_id -p 0x80 -g -u -s /block/%n"
		path_selector "round-robin 0"
		path_checker readsector0
		features "0"
		hardware_handler "0"
		rr_weight uniform
		rr_min_io 1000
	}
}