 |
Henry Newman
This month, I’ll explain how and why to tune volume managers. In my previous columns, I’ve examined how application I/O using C Library I/O has a great impact on how much work the system has to do. In my most recent column, I specifically covered how applications I/O from system calls impact the system. The extra work caused by applications can be reduced or eliminated given how the volume managers are configured. By the same token, a poor volume can add to I/O overhead in the system and significantly reduce system performance. In both cases, the volume manager is a significant part of the equation. So this month, I will discuss aspects of tuning the system for volume managers.
System Tuning
One of the keys to tuning a system is gaining an understanding of the I/O path through the system. Many operating systems, volume managers, and file systems have limitations on the size of the I/O requests that can be made to the system. On Solaris, for example, by default, the largest physical request that can be made is 128 KB. This can be changed by a modification to /etc/system to add:
set maxphys= "value in decimal, octal or hexadecimal"
To set maxphys to 8 MB, you add to /etc/system:
set maxphys=8388608
or
set maxphys=0x800000
In Solaris, each of the device drivers relating to SCSI I/O sd (SCSI Device Driver), ssd (Fibre Channel Device Driver), and st (Tape Device Driver) supports an upper limit of 1 MB for a transfer size, even though maxphys from /etc/system is set to a value larger than 1 MB. Each of these device drivers must be changed if you want to make I/O requests over 1 MB for devices used by that driver.
Therefore, the largest value that can be transferred from an application to or from the system will be 1 MB without changes to the device driver — even if you have made changes in /etc/system. Changes to the driver are more difficult than a changes to /etc/system. Mistakes in the configuration of device drivers can prevent you from seeing the devices. You must make changes to each driver in Solaris by changing the configuration files in /kernel/drv/"driver name".conf. Where the driver name is sd, ssd, or st, you need to add the following for each driver type that you would like to change (sd, ssd, or st).
"driver name"_max_xfer_size="value"
For example to change the sd driver to allow the largest transfer to be 16,777,216 (16MB)
sd_max_xfer_size=0x1000000;
For Solaris 8, this change is placed as the last line listing the SCSI targets, but for Solaris 9 it must be the first line in the file. It is extremely important for the case to be correct. You cannot use capital letters or a capital X for the hexadecimal representation. Using capital letters will prevent you from using all devices for that driver. Also, the line must end with a semicolon.
Equally important to note is that some Fibre Channel HBAs do not support requests greater than 8 MB. This is usually based on the DMA size allowed within the Fibre Channel chipset. It is important to test any changes to ensure that they work as expected. Making requests larger than the HBA allows can cause the channel to hang and no I/O can be done via that channel. I know that’s not how it should work, but that’s how it does work. After the file system has been configured and the tuning changes have been made, you can use the dd(1M) and sar(1) commands to check the average size of an I/O request. This is a final test to ensure that the system is correctly configured. Run the following two commands at the same time to verify the system performance:
dd if=/dev/zero of=path_of_filesystem/any_file bs=8192K count=64 & sar -d 1 5
The dd(1M) command performs I/O with a block size (bs) of 8192 kilobytes (8,388,308 bytes), copying 64 records, or blocks. The sar -d command shows the resulting disk activity, as shown in the following example:
# sar -d 1 5
SunOS gandalf 5.8 Generic_108528-07 sun4u 04/16/02
16:59:36 device %busy avque r+w/s blks/s avwait avserv
16:59:37 sd2 0 0.0 4 131076 0.0 0.0
sd3 0 0.0 0 0 0.0 0.0
16:59:38 sd2 24 1.8 3 98304 57.9 27.3
sd3 0 0.0 0 0 0.0 0.0
16:59:39 sd2 1 0.0 4 131184 0.0 6.7
sd3 0 0.0 0 0 0.0 0.0
16:59:40 sd2 0 0.0 5 163846 0.0 0.0
sd3 0 0.0 0 0 0.0 0.0
16:59:41 sd2 0 0.0 3 98406 0.0 0.0
sd3 0 0.0 0 0 0.0 0.0
Average sd2 5 0.4 3.8 124562 53.1 25.6
sd3 0 0.0 0 0 0.0 0.0
#
To determine the average size of an I/O request, divide the number of blocks transferred per second (blks/s) by the number of read/write operations per second (r+w/s). The resulting value should be approximately 16384 (8192 * 2). (In this example, the average I/O request is about 16390.)
If the average I/O request size is much smaller than this amount, there might be a kernel error causing a problem. Look for errors in /var/adm/messages, especially anything relating to maxphys and “xx”_max_xfer_size.
Tape tuning can be very problematic, so before you change the st driver to make large requests for tape, you must do the following:
Other operating systems, such as SGI IRIX, have an equivalent value to maxphys, and it is called maxdmasz. IRIX does not have the driver limitations that Solaris has. AIX has limitations based on the file system, but not based on the kernel. Linux is limited by the file system implementation and not by the kernel.
Volume Manager Configuration
The way to obtain the most optimal performance is to match the application I/O, volume manager stripe width, file system allocations, and the RAID allocation (which I will be addressing over the next few columns). This is almost impossible in all but very specific, controlled, application environments. Therefore, tuning for most applications environments becomes a tradeoff for high performance for some applications and lower performance for others.
Volume Manager Settings
For file systems that require a volume manager, these are the two most important areas to consider:
Setting Stripe Values
The most important volume manager setting is the stripe size for the volume. Stripe values should be set based on the underlying hardware architecture. Thus, you must understand:
Setting the stripe value to a large number could improve performance if files are large, but if files are small, this could hurt performance. Take the following example:
Number of devices in the stripe group = 4
Average File size = 32KB
If you make your stripe value 2 MB, you would place on average 64 files on each device before writing on the next device. This might be a good thing if you accessed them sequentially. However, if you accessed them randomly in groups of 32, you could have performance problems because all 32 files would be on a single device. Having them on different devices could improve performance. Often, the longest time doing I/O to any RAID and/or disk is not the data transfer but the seek and latency time. (This, too, will be discussed in great detail in a later column.)
Additionally, some volume managers, such as Veritas VxVM, have tunables that limit the size of requests to the volume manager. In VxVM, the value is called vxio:vol_maxio. By default, the largest request that can be made to the volume manager without the system breaking up the I/O requests is 256 KB. To change this value for Solaris, add the following to /etc/system:
set vxio:vol_maxio= 32768 # sets the largest request to a volume
# before breaking up to 16 MB
The largest value that can be set is 65536 or 32 MB. Even if you set maxphys, /kernel/drv/sd.conf, and vxio:vol_maxio to 32 MB, the largest request that can be made to a SCSI driver currently is 32 MB - 32 KB - 1 byte. A standards committee is looking at changing this and the largest LUN supported, which is now 2 TB. Some vendors (like IBM AIX) have rewritten drivers to support these large LUNs.
A number of tradeoffs must to be considered when setting volume manager stripe settings:
Number of Devices
The number of devices within the volume is important to understand. Considering that most vendors, as well as the standard, only support 2 TB LUNs, and many only support 2TB file systems, the number of underlying devices might only be 11 180-GB disk drives. It is still important to understand how many devices will be in the stripe group. For example, having 10 devices with a 32-KB stripe element means that a full stripe of data is 10*32 KB. Making that stripe size 2 MB instead of 32 KB means that a full stripe is now 20 MB. Having applications that can write full stripes or, even better, multiple stripes of data to allow file system readahead will significantly improve I/O performance. And, if available RAID readahead algorithms come into play, that’s even better. Be aware that there is a downside, as I will discuss.
Size of I/O Requests
Now for the downside. If you are making small I/O request and want to ensure that you write a full stripe, your I/O performance will drop dramatically as the size of the request drops. Let’s say your application requests 256-KB I/Os (Oracle can be tuned to make requests this size), and you have 8 devices and a 32-KB stripe element so you can write full stripes of data. The performance both in wall time and CPU overhead of 32-KB I/O as compared with 256-KB I/O is hugely different. A number of factors play into this, including the types of devices, the transport medium (SCSI or FC), and the file system, but you might easily see 50% reduction in megabytes per second and 20% increase in system overhead. So, as you can see, it is difficult to provide generalizable tuning suggestions. Also, to achieve good I/O performance, you must understand the whole I/O path.
Device Fragmentation
Using small stripe elements can cause the underlying device to become fragmented if files are not preallocated within the file system. The tradeoff is that for large I/O requests, you are not getting the advantage of multiple devices reading or writing at the same time. In this day and age, a single Fibre Channel RAID device provides sufficient performance for almost all applications. So, one of the things I have been suggesting to customers is to make very large stripe values, instead of making smaller stripe values to allow multiple devices to participate in delivering the I/O. For example, many databases use a 2-GB file size and, for those sites, I have suggested using 2-GB stripe values. That way, each database file is round-robined to a different device.
Conclusions
Although there are few system tuning parameters and few volume manager tunables, understanding how they work is critical to the efficient use of the hardware and thereby to reduced expenditures on new hardware. Understanding what the applications are doing to your system is the key to setting up the volume stripe size. Equally important are the file system setting and the RAID tunables, which will be covered in the next few months.
Henry Newman has worked in the IT industry for more than 20 years. Originally at Cray Research and now with a consulting organization, he has provided expertise in systems architecture and performance analysis to customers in government, scientific research, and industry around the world. His focus is high-performance computing, storage and networking for UNIX systems, and he previously authored a monthly column about storage for Server/Workstation Expert magazine. He may be reached at: hsn@hsnewman.com.
|
|