|Placing the Cluster Root File Systems Under LSM Control|
|Tru64 UNIX Version 5.1A, January 2002|
© 2002 Compaq Computer Corporation
This Best Practice describes how you can place the clusterwide root
file systems under Logical Storage Manager (LSM) control by migrating them
to LSM volumes with the
Prior to Tru64TM UNIX Version 5.1A, you could not use LSM volumes to mirror the root file system; you had to use hardware RAID.
Using LSM volumes for these file systems lets you:
Provide redundancy through LSM volume mirroring
Improve performance through striping
Configure one or more spare disks to automatically replace a failing disk in a mirrored volume
See the Tru64 UNIX Best Practices Web page for more information about Best Practices documentation:
Is This Best Practice Right for You?
Not all Best Practices apply to all configurations, so you must be sure that this Best Practice is appropriate for your system and circumstances. To use this Best Practice, you must meet the requirements described in the following table:
|Operating System||Tru64 UNIX Version 5.1A, and TruClusterTM Server Version 5.1A|
|Logical Storage Manager||
LSM must be installed and initialized on the cluster. (See Background and Reference Information.)
The Advanced File System (AdvFS) Utilities license.
Logical Storage Manager license. To use mirroring and striping, the LSM license must be installed.
|Disks and Accessibility||
Within the rootdg disk group, there must be one or more suitable LSM simple or sliced disks as targets for the migration, because the volumes must belong to rootdg. You cannot migrate domains to nopriv disks. (See Background and Reference Information.)
If you configure spare disks, there should be one spare disk for each disk used by a mirrored volume. The spare disks should be the same size as the disks they could replace.
Ideally the disks should be on a shared bus, so that all cluster members have access to the disks whether an individual cluster member is up or down.
|Impact on Availability||
None. No shutdown or unmounting is required.
|Additional Requirements||You must be logged in as superuser and have some knowledge of LSM and its commands.|
If you do not meet the previous requirements, see
Before You Begin
Before you migrate a cluster file system to an LSM volume, you must
understand some background information and develop a strategy for migration
and disk usage.
Background and Reference Information
See the Logical Storage Manager manual for information on installing the LSM license, installing and initializing LSM on a cluster, and descriptions of LSM disk types (sliced, simple, and nopriv) and the rootdg disk group.
See the Cluster Administration and Cluster Hardware Configuration manuals for information on cluster configurations, including recommendations for shared storage and avoiding a single point of failure.
volmigratecommand operates on AdvFS domain names. Within this Best Practice, the cluster file systems are referred to by the default AdvFS domain names
command is a shell script that calls
several other commands to:
Create an LSM volume for the domain on the LSM disk or disks that you specify.
Add the LSM volume to the domain being migrated, with the
Migrate the data from the original disk partition to the LSM volume.
Remove the original disk partition from the domain with the
command, and set the disk label partition table
entry for that partition to
For detailed information on the
including the options for striping and mirroring, recommendations, and restrictions,
Disk Space Considerations
If you have limited available disks but want the benefits of mirroring, you can place the original disk or disks under LSM control after migrating the domains and then mirror the volumes to the disk or disks.
When you place a disk under LSM control, its usable space is reduced by 4096 blocks (2 MB) for the LSM private metadata.
If a domain uses the
cpartition of a disk (the entire disk), you must migrate it to a volume 2 MB smaller than the domain if you want to mirror the volume to the same disk. If several domains are on the same disk, and if the disk is not at least 2 MB larger than the total size of the domains, you must migrate one domain to a volume 2 MB smaller than the domain. Decide which domain to reduce based on disk usage and expected growth.
By default, the volume created by the
command is the same size as the AdvFS domain.
you create a volume smaller than the domain, within the restrictions described
If necessary, both LSM and AdvFS allow you to add space to an LSM volume
or AdvFS domain later.
To provide higher availability for mirrored volumes, you can configure one or more spare disks that LSM can use to automatically relocate data from a failing disk. Ideally, configure one spare disk for every disk used in a mirrored volume in the disk group. Each spare disk should be as large as the largest disk in use by a mirrored volume in the disk group.
daemon notifies you of a disk failure
(and automatic replacement).
At that time, you can add another disk to the
disk group and configure it as a new spare.
You can check the number of spare disks in each disk group by entering
column of the output, all available spare disks show a status of
Preparing to Use This Best Practice
To prepare for applying this Best Practice, decide whether you want to reuse the original disk or disk partition to mirror the volumes.
If you plan to reuse the original disk, migrate each domain to an unmirrored volume (optionally specifying the volume size, as discussed in Disk Space Considerations), place the original disk under LSM control, and then mirror the volumes to that disk.
Normally, you should not use the same disk to support multiple volumes, because this increases the number of volumes at risk if the disk fails. Mirroring the volumes and configuring hot-spare disks reduces this risk. However if you have available disks, consider migrating each domain to its own disk, using the original disk to mirror just one volume and using other disks to mirror the other volumes.
If you do not plan to reuse the original disk, migrate each domain to a mirrored volume (optional) and use the original disk for other purposes (perhaps as a hot-spare disk).
If you want the volume to be striped, sufficient disks must be available at the time of migration. (Because striping requires multiple disks, you cannot create a striped volume on one disk.)
Each mirror or stripe column in a volume must be on different disks.
To create a striped and mirrored volume, you need multiple disks (nstripes
For example, to create a volume with two mirrors, each of which
is striped across four disks, requires eight disks.
Applying the Best Practice
Before you migrate a domain to an LSM volume, be sure to read Before You Begin.
There are three steps for placing the cluster file systems under LSM control:
Migrate the domains.
Mirror the domain volumes to the original disk (optional).
Configure hot-spare disks for the rootdg disk group (optional).
Step 1: Migrate the Domains
The general syntax for the
/usr/sbin/volmigrate [-g diskgroup] [-m num_mirrors] [-s num_columns] \ [-N volume_name] [-l sectors] domain_name disk_media_name ...
For this Best Practice, you do not need to specify a disk group. The default disk group is rootdg, which is a requirement for these volumes.
You can run the
command from any cluster
Depending on the size of the domain, the command might take several
minutes to complete.
Unless the command displays an error message, the migration
If you plan to reuse the original disk or disk partition for the mirror, migrate the domain to an unmirrored volume and then mirror the volume in a separate step.
You can choose names for the volumes.
By default, the volume name is
the domain name with the
suffix , as in
No two LSM volumes within the same disk group can have the same
To migrate the domains:
Display the attributes (specifically, sizes and names) of the file system domains:
# cd /etc/fdmns # showfdmn *
Id Date Created LogPgs Version Domain Name 3bcc9383.000bd63b Tue Oct 16 16:07:31 2001 512 4 cluster_root Vol 512-Blks Free % Used Cmode Rblks Wblks Vol Name 1L 557936 329360 41% on 256 256 /dev/disk/dsk6a Id Date Created LogPgs Version Domain Name 3bcc9384.000abbf1 Tue Oct 16 16:07:32 2001 512 4 cluster_usr Vol 512-Blks Free % Used Cmode Rblks Wblks Vol Name 1L 1684224 116672 93% on 256 256 /dev/disk/dsk6g Id Date Created LogPgs Version Domain Name 3bcc9385.00076d14 Tue Oct 16 16:07:33 2001 512 4 cluster_var Vol 512-Blks Free % Used Cmode Rblks Wblks Vol Name 1L 1667024 1056912 37% on 256 256 /dev/disk/dsk6h
The cluster root domain is 557936 blocks in size, and each block on the Tru64 UNIX system is 512 bytes. This means the cluster root domain is approximately 272 MB. The total for all three domains is 3909184 blocks. If you plan to migrate all three domains to the same LSM disk, make sure the disk has that much free space (step 3).
Display the disks in the rootdg disk group to find simple
or sliced disks in the
# voldisk -g rootdg list
DEVICE TYPE DISK GROUP STATUS dsk0 sliced dsk0 rootdg online dsk1 sliced dsk1 rootdg online dsk2 sliced dsk2 rootdg online dsk3 sliced dsk3 rootdg online dsk4 sliced dsk4 rootdg online dsk5 sliced dsk5 rootdg online dsk6 sliced dsk6 rootdg online dsk7 sliced dsk7 rootdg online dsk8 sliced dsk8 rootdg online dsk9 sliced dsk9 rootdg online dsk10 sliced dsk10 rootdg online dsk11 sliced dsk11 rootdg online dsk12 sliced dsk12 rootdg online dsk13 sliced dsk13 rootdg online dsk14 sliced dsk14 rootdg online dsk15 sliced dsk15 rootdg online dsk16 sliced dsk16 rootdg online dsk17 sliced dsk17 rootdg online dsk18 sliced dsk18 rootdg online dsk19 sliced dsk19 rootdg online dsk20 sliced dsk20 rootdg online dsk21 sliced dsk21 rootdg online dsk22 sliced dsk22 rootdg online dsk23 sliced dsk23 rootdg online dsk24 sliced dsk24 rootdg online
Display the free space available in the rootdg disk group:
# voldg -g rootdg free
DISK DEVICE TAG OFFSET LENGTH FLAGS dsk0 dsk0 dsk0 1048576 7327392 - dsk1 dsk1 dsk1 1048576 3057792 - dsk2 dsk2 dsk2 0 8375968 - dsk3 dsk3 dsk3 0 8375968 - dsk4 dsk4 dsk4 0 4106368 - dsk5 dsk5 dsk5 0 8375968 - dsk6 dsk6 dsk6 0 8375968 - dsk7 dsk7 dsk7 0 8375968 - dsk8 dsk8 dsk8 0 4106368 - dsk9 dsk9 dsk9 0 8375968 - dsk10 dsk10 dsk10 0 8375968 - dsk11 dsk11 dsk11 0 4106368 - dsk12 dsk12 dsk12 0 4106368 - dsk13 dsk13 dsk13 0 4106368 - dsk14 dsk14 dsk14 0 4106368 - dsk15 dsk15 dsk15 0 8375968 - dsk16 dsk16 dsk16 6912 17762500 - dsk17 dsk17 dsk17 6912 17762500 - dsk18 dsk18 dsk18 6977 4099391 - dsk19 dsk19 dsk19 6912 8369056 - dsk20 dsk20 dsk20 6912 8369056 - dsk21 dsk21 dsk21 6912 8369056 - dsk22 dsk22 dsk22 0 8375968 - dsk23 dsk23 dsk23 0 8375968 - dsk24 dsk24 dsk24 0 17769412 -
Choose sliced or simple disks with enough free space to create a volume with the characteristics you want (such as mirrored or striped).
If possible, choose disks with an offset of 0.
Avoid using disks with
an offset, because these are in use by another volume.
(To confirm this, enter
command and examine the output for the disks
Confirm that each disk you want to use is accessible by all cluster members:
# hwmgr -view devices -cluster
HWID: Device Name Mfg Model Hostname Location -------------------------------------------------------------------------------- 3: /dev/dmapi/dmapi ernest 3: /dev/dmapi/dmapi joey 4: scp ernest 5: kevm ernest 62: /dev/disk/floppy0c 3.5in floppy ernest fdi0-unit-0 77: /dev/disk/dsk0c COMPAQ BB018122B7 ernest bus-0-targ-0-lun-0 78: /dev/disk/dsk1c COMPAQ BB018122B7 ernest bus-0-targ-1-lun-0 79: /dev/disk/dsk2c DEC RZ28M (C) DEC ernest bus-2-targ-1-lun-0 79: /dev/disk/dsk2c DEC RZ28M (C) DEC joey bus-2-targ-1-lun-0 80: /dev/disk/dsk3c DEC RZ28L-AS (C) DEC ernest bus-2-targ-2-lun-0 80: /dev/disk/dsk3c DEC RZ28L-AS (C) DEC joey bus-2-targ-2-lun-0 81: /dev/disk/dsk4c DEC RZ29B (C) DEC ernest bus-2-targ-3-lun-0 81: /dev/disk/dsk4c DEC RZ29B (C) DEC joey bus-2-targ-3-lun-0 82: /dev/disk/dsk5c DEC RZ28D (C) DEC ernest bus-2-targ-4-lun-0 82: /dev/disk/dsk5c DEC RZ28D (C) DEC joey bus-2-targ-4-lun-0 83: /dev/disk/dsk6c DEC RZ28L-AS (C) DEC ernest bus-2-targ-5-lun-0 83: /dev/disk/dsk6c DEC RZ28L-AS (C) DEC joey bus-2-targ-5-lun-0 84: /dev/disk/dsk7c DEC RZ1CF-CF (C) DEC ernest bus-2-targ-8-lun-0 84: /dev/disk/dsk7c DEC RZ1CF-CF (C) DEC joey bus-2-targ-8-lun-0 85: /dev/disk/dsk8c DEC RZ1CB-CS (C) DEC ernest bus-2-targ-9-lun-0 85: /dev/disk/dsk8c DEC RZ1CB-CS (C) DEC joey bus-2-targ-9-lun-0 86: /dev/disk/dsk9c DEC RZ1CF-CF (C) DEC ernest bus-2-targ-10-lun-0 86: /dev/disk/dsk9c DEC RZ1CF-CF (C) DEC joey bus-2-targ-10-lun-0 87: /dev/disk/dsk10c DEC RZ1CF-CF (C) DEC ernest bus-2-targ-11-lun-0 87: /dev/disk/dsk10c DEC RZ1CF-CF (C) DEC joey bus-2-targ-11-lun-0 88: /dev/disk/dsk11c DEC RZ1CF-CF (C) DEC ernest bus-2-targ-12-lun-0 88: /dev/disk/dsk11c DEC RZ1CF-CF (C) DEC joey bus-2-targ-12-lun-0 89: /dev/disk/dsk12c DEC RZ1CF-CF (C) DEC ernest bus-2-targ-13-lun-0 89: /dev/disk/dsk12c DEC RZ1CF-CF (C) DEC joey bus-2-targ-13-lun-0 90: /dev/disk/cdrom0c COMPAQ CRD-8402B ernest bus-3-targ-0-lun-0 91: random ernest 92: urandom ernest 95: /dev/scp_scsi joey 96: /dev/kevm joey 147: /dev/disk/floppy1c 3.5in floppy joey fdi0-unit-0 162: /dev/disk/dsk13c COMPAQ BB018122B7 joey bus-0-targ-0-lun-0 163: /dev/disk/dsk14c COMPAQ BB018122B7 joey bus-0-targ-1-lun-0 164: /dev/disk/cdrom1c COMPAQ CRD-8402B joey bus-3-targ-0-lun-0 165: /dev/random joey 166: /dev/urandom joey
Migrate the domains.
The options you specify for the migration depend on whether you want to mirror the volume in this step or wait and mirror the volume to the original disk.
To migrate a domain to an LSM volume with the default properties (concatenated, no mirror), whether or not you plan to use the original disk for the mirror later:
# volmigrate domain disk ...
# volmigrate cluster_root dsk4
To migrate a domain to an LSM volume of a specific size (for example, smaller than the domain by 2 MB):
# volmigrate -l sectors domain disk ...
For example, to migrate the
is 1667024 sectors (approximately 814 MB), to a volume of 1662928 sectors
# volmigrate -l 1662928 cluster_var dsk4
To migrate a domain to a mirrored volume on two disks:
# volmigrate -m 2 domain disk disk
# volmigrate -m 2 cluster_usr dsk4 dsk8
To migrate a domain to a striped, unmirrored volume on four disks:
# volmigrate -s 4 cluster_root disk1 disk2 disk3 disk4
Remember that you will need four disks to mirror the volume later (for example, the original disk plus three others).
To migrate the domain to a striped, mirrored volume on six disks (each mirror will be striped over three disks):
# volmigrate -m 2 -s 3 cluster_root \ dsk4 dsk8 dsk11 dsk12 dsk13 dsk14
Step 2: Mirroring the Volumes to the Original Disk (Optional)
The attributes you specify for a volume with the
command (such as the length and whether the volume is striped)
are applied to the mirror when you mirror that volume.
If you create a striped volume on multiple disks, you need to specify the same number of additional disks to mirror the volume. For example, if the volume is striped over four disks, you need four additional disks to create the mirror; one of which can be the original disk.
To mirror a volume to the original disk:
Confirm that all partitions on the original disk are unused (in this example, dsk6):
# disklabel -r dsk6
Add the disk to LSM:
# voldisksetup -i dsk6
Add the disk to the rootdg disk group. You do not need to specify a disk group because the rootdg disk group is assumed:
# voldg adddisk dsk6
Verify that there is enough space in the public region of the LSM disk to mirror the volume:
# voldisk list dsk6 | grep public public: slice=6 offset=16 len=8375968
Verify the cluster volume names:
# volprint -vt | grep cluster
v cluster_rootvol fsgen ENABLED ACTIVE 557936 SELECT - v cluster_usrvol fsgen ENABLED ACTIVE 1684224 SELECT - v cluster_varvol fsgen ENABLED ACTIVE 1667024 SELECT -
You use these names to mirror the volumes.
Mirror the volumes. You must mirror each volume separately.
If the volume is currently on only one disk, enter:
# volassist mirror volume disk
# volassist mirror cluster_rootvol dsk6 # volassist mirror cluster_usrvol dsk6 # volassist mirror cluster_varvol dsk6
If the volume is striped over several disks, you must specify the same number of disks the volume is striped over.
For example, to mirror a volume that is striped over four disks:
# volassist mirror cluster_rootvol dsk6 dsk11 dsk12 dsk13
Step 3: Configure Hot-Spare Disks (Optional)
You can configure spare disks to automatically replace a failing disk in mirrored volumes.
Configure enough spare disks to replace more than one failed disk at any given time. Each spare disk should be as large as the largest disk in use by a volume in the disk group.
To configure one or more spare disks for the rootdg disk group:
# voledit set spare=on disk ...
For example, in the output from step 3 in Step 1: Migrate the Domains, disks dsk2, dsk3, and dsk5 (among others) have at least twice the amount of free space as the disks used in this Best Practice for creating the domain volumes. So, disks dsk2, dsk3, and dsk5 are suitable to configure as hot spares, as follows:
# voledit set spare=on dsk2 dsk3 dsk5
command runs in verbose mode
by default, it alerts you to problems as it executes.
If errors occur, the command that experienced the error displays
its own error messages or codes, and
any changes made, if possible.
If no error messages appear, the migration process completed successfully.
You can additionally verify the results of the migration process by doing the following:
Display the properties of the LSM volumes by entering:
In the output, verify that there are volumes named
(the default names) or the names you assigned to the volumes, with
the attributes you specified.
Verify the spare disks you configured by entering:
Information similar to the following is displayed.
Available spare disks
have a status of
DEVICE TYPE DISK GROUP STATUS
dsk2 sliced dsk2 rootdg online spare dsk3 sliced dsk3 rootdg online spare dsk5 sliced dsk5 rootdg online spare
Verify that the spare disks have enough free space to replace any disk in the mirrored volumes:
#voldg -g rootdg spare
Information similar to the following is displayed, showing the size of the spare disks:
DISK DEVICE TAG OFFSET LENGTH FLAGS dsk2 dsk2 dsk2 0 8375968 s dsk3 dsk3 dsk3 0 8375968 s dsk5 dsk5 dsk5 0 8375968 s
If this Best Practice was not successful, see
for information about identifying and solving problems.
command or one of the commands
that it calls returns an error, refer to the indicated command's reference
page for more information about corrective steps.
The following list presents some of the error messages you might encounter and the action needed to solve each problem.
volmigrate: must specify an AdvFS domain name.
Problem: The domain name is not valid.
Solution: Verify that you entered the domain name correctly. Also, make sure you specify the domain name before the disk or disks on the command line.
volmigrate: the cluster root volume must be in rootdg.
Problem: The specified disk is not in the rootdg disk group or is not an LSM disk.
Confirm that the disk
is under LSM control and belongs to the rootdg disk group.
volmigrate: must specify LSM disk media names to use for volume.
Problem: The specified disk media name is incorrect.
Verify the disk media name
You must use the LSM disk's
media name, which might differ from the name assigned to the disk by the operating
system (the disk access name, or device name).
In the output of the
command, the disk media name is in the
lsm:volassist: ERROR: Cannot allocate space for length block volume
Problem: The specified disk or disks do not have enough free space to create a volume for the domain.
Use a larger LSM disk,
specify additional LSM disks, or use the
option to specify a smaller volume
If you tried to use the original disk to mirror one or more volumes but there was not enough free space, you can still mirror the volume to any other LSM simple or sliced disk that has enough space and use the original disk for other purposes, such as a spare disk.
volmigrate: length specified must be at least xxxxxx, or use the -f option to override 10% spare requirement. Specified size must be at least yyyyyy.
Problem: The specified volume size does not meet the minimum requirement of the in-use area of the domain (yyyyyy) plus 10 percent extra space for file system overhead.
If the volume size you
specified is at least the size of the in-use area of the domain but provides
less than 10 percent extra room, use the
You cannot specify a volume size smaller than the in-use area
of the domain.
This Best Practice is the recommended method for placing the cluster root domains under LSM control.
If LSM is not installed on your system, see the Installation Guide for information on installing software subsets and rebuilding the kernel.
The only alternative to using LSM to create a mirror of the clusterwide
file systems is to use hardware RAID.
See your hardware RAID documentation
or Customer Support representative for more information.
Comments and Questions
We value your comments and questions on the information in this document. Please mail your comments to us at this address:
Compaq, the Compaq logo, Tru64, and TruCluster are trademarks of Compaq Information Technologies Group, L.P. in the U.S. and/or other countries. UNIX is a trademark of The Open Group in the United States and other countries. All other product names mentioned herein may be trademarks of their respective companies.
Confidential computer software. Valid license from Compaq required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.
Compaq shall not be liable for technical or editorial errors or omissions contained herein. The information is provided "as is" without warranty of any kind and is subject to change without notice. The warranties for Compaq products are set forth in the express limited warranty statements accompanying such products. Nothing herein should be construed as constituting an additional warranty.