Placing the Cluster Root File Systems Under LSM Control
Tru64 UNIX Version 5.1A, January 2002

© 2002 Compaq Computer Corporation

This Best Practice describes how you can place the clusterwide root (/), /usr, and /var file systems under Logical Storage Manager (LSM) control by migrating them to LSM volumes with the volmigrate command.

Prior to Tru64TM UNIX Version 5.1A, you could not use LSM volumes to mirror the root file system; you had to use hardware RAID.

Using LSM volumes for these file systems lets you:

See the Tru64 UNIX Best Practices Web page for more information about Best Practices documentation:

http://www.tru64unix.compaq.com/docs/best_practices/

Is This Best Practice Right for You?

Not all Best Practices apply to all configurations, so you must be sure that this Best Practice is appropriate for your system and circumstances. To use this Best Practice, you must meet the requirements described in the following table:

Requirement Description
Operating System Tru64 UNIX Version 5.1A, and TruClusterTM Server Version 5.1A
Logical Storage Manager

LSM must be installed and initialized on the cluster. (See Background and Reference Information.)

Licenses

The Advanced File System (AdvFS) Utilities license. The LSM volmigrate command uses the AdvFS addvol and rmvol commands, which are available only with the AdvFS Utilities license.

Logical Storage Manager license. To use mirroring and striping, the LSM license must be installed.

Disks and Accessibility

Within the rootdg disk group, there must be one or more suitable LSM simple or sliced disks as targets for the migration, because the volumes must belong to rootdg. You cannot migrate domains to nopriv disks. (See Background and Reference Information.)

If you configure spare disks, there should be one spare disk for each disk used by a mirrored volume. The spare disks should be the same size as the disks they could replace.

Ideally the disks should be on a shared bus, so that all cluster members have access to the disks whether an individual cluster member is up or down.

Impact on Availability

None. No shutdown or unmounting is required.

Additional Requirements You must be logged in as superuser and have some knowledge of LSM and its commands.

If you do not meet the previous requirements, see Alternative Practices.

Before You Begin

Before you migrate a cluster file system to an LSM volume, you must understand some background information and develop a strategy for migration and disk usage.

Background and Reference Information

See the Logical Storage Manager manual for information on installing the LSM license, installing and initializing LSM on a cluster, and descriptions of LSM disk types (sliced, simple, and nopriv) and the rootdg disk group.

See the Cluster Administration and Cluster Hardware Configuration manuals for information on cluster configurations, including recommendations for shared storage and avoiding a single point of failure.

Note

The volmigrate command operates on AdvFS domain names. Within this Best Practice, the cluster file systems are referred to by the default AdvFS domain names cluster_root, cluster_usr, and cluster_var.

The volmigrate command is a shell script that calls several other commands to:

For detailed information on the volmigrate command, including the options for striping and mirroring, recommendations, and restrictions, see the volmigrate(8) reference page.

Disk Space Considerations

If you have limited available disks but want the benefits of mirroring, you can place the original disk or disks under LSM control after migrating the domains and then mirror the volumes to the disk or disks.

Note

When you place a disk under LSM control, its usable space is reduced by 4096 blocks (2 MB) for the LSM private metadata.

If a domain uses the c partition of a disk (the entire disk), you must migrate it to a volume 2 MB smaller than the domain if you want to mirror the volume to the same disk. If several domains are on the same disk, and if the disk is not at least 2 MB larger than the total size of the domains, you must migrate one domain to a volume 2 MB smaller than the domain. Decide which domain to reduce based on disk usage and expected growth.

By default, the volume created by the volmigrate command is the same size as the AdvFS domain. The volmigrate -l length command lets you create a volume smaller than the domain, within the restrictions described in the volmigrate(8) reference page. In effect, this shrinks the domain. The volume size you specify must be at least 10 percent larger than the in-use portion of the domain. We recommend reducing the volume by only the 2 MB necessary.

If necessary, both LSM and AdvFS allow you to add space to an LSM volume or AdvFS domain later.

Availability Considerations

To provide higher availability for mirrored volumes, you can configure one or more spare disks that LSM can use to automatically relocate data from a failing disk. Ideally, configure one spare disk for every disk used in a mirrored volume in the disk group. Each spare disk should be as large as the largest disk in use by a mirrored volume in the disk group.

The LSM volwatch daemon notifies you of a disk failure (and automatic replacement). At that time, you can add another disk to the disk group and configure it as a new spare.

You can check the number of spare disks in each disk group by entering the voldisk list command. In the STATUS column of the output, all available spare disks show a status of online spare.

Preparing to Use This Best Practice

To prepare for applying this Best Practice, decide whether you want to reuse the original disk or disk partition to mirror the volumes.

If you want the volume to be striped, sufficient disks must be available at the time of migration. (Because striping requires multiple disks, you cannot create a striped volume on one disk.)

Each mirror or stripe column in a volume must be on different disks. To create a striped and mirrored volume, you need multiple disks (nstripes * nmirrors). For example, to create a volume with two mirrors, each of which is striped across four disks, requires eight disks.

Applying the Best Practice

Before you migrate a domain to an LSM volume, be sure to read Before You Begin.

There are three steps for placing the cluster file systems under LSM control:

  1. Migrate the domains.

  2. Mirror the domain volumes to the original disk (optional).

  3. Configure hot-spare disks for the rootdg disk group (optional).

Step 1: Migrate the Domains

The general syntax for the volmigrate command is:

/usr/sbin/volmigrate [-g diskgroup] [-m num_mirrors] [-s num_columns] \
[-N volume_name] [-l sectors] domain_name disk_media_name ...

For this Best Practice, you do not need to specify a disk group. The default disk group is rootdg, which is a requirement for these volumes.

You can run the volmigrate command from any cluster member. Depending on the size of the domain, the command might take several minutes to complete. Unless the command displays an error message, the migration is progressing.

If you plan to reuse the original disk or disk partition for the mirror, migrate the domain to an unmirrored volume and then mirror the volume in a separate step.

You can choose names for the volumes. By default, the volume name is the domain name with the vol suffix , as in cluster_rootvol. No two LSM volumes within the same disk group can have the same name.

To migrate the domains:

  1. Display the attributes (specifically, sizes and names) of the file system domains:

    # cd /etc/fdmns
    # showfdmn *
    

                   Id              Date Created  LogPgs  Version  Domain Name
    3bcc9383.000bd63b  Tue Oct 16 16:07:31 2001     512        4  cluster_root
     
      Vol   512-Blks        Free  % Used  Cmode  Rblks  Wblks  Vol Name
       1L     557936      329360     41%     on    256    256  /dev/disk/dsk6a
     
                   Id              Date Created  LogPgs  Version  Domain Name
    3bcc9384.000abbf1  Tue Oct 16 16:07:32 2001     512        4  cluster_usr
     
      Vol   512-Blks        Free  % Used  Cmode  Rblks  Wblks  Vol Name
       1L    1684224      116672     93%     on    256    256  /dev/disk/dsk6g
     
                   Id              Date Created  LogPgs  Version  Domain Name
    3bcc9385.00076d14  Tue Oct 16 16:07:33 2001     512        4  cluster_var
     
      Vol   512-Blks        Free  % Used  Cmode  Rblks  Wblks  Vol Name
       1L    1667024     1056912     37%     on    256    256  /dev/disk/dsk6h
     
    .
    .
    .

    The cluster root domain is 557936 blocks in size, and each block on the Tru64 UNIX system is 512 bytes. This means the cluster root domain is approximately 272 MB. The total for all three domains is 3909184 blocks. If you plan to migrate all three domains to the same LSM disk, make sure the disk has that much free space (step 3).

  2. Display the disks in the rootdg disk group to find simple or sliced disks in the TYPE column:

    # voldisk -g rootdg list
    

    DEVICE       TYPE      DISK         GROUP        STATUS
    dsk0         sliced    dsk0         rootdg       online
    dsk1         sliced    dsk1         rootdg       online
    dsk2         sliced    dsk2         rootdg       online
    dsk3         sliced    dsk3         rootdg       online
    dsk4         sliced    dsk4         rootdg       online
    dsk5         sliced    dsk5         rootdg       online
    dsk6         sliced    dsk6         rootdg       online
    dsk7         sliced    dsk7         rootdg       online
    dsk8         sliced    dsk8         rootdg       online
    dsk9         sliced    dsk9         rootdg       online
    dsk10        sliced    dsk10        rootdg       online
    dsk11        sliced    dsk11        rootdg       online
    dsk12        sliced    dsk12        rootdg       online
    dsk13        sliced    dsk13        rootdg       online
    dsk14        sliced    dsk14        rootdg       online
    dsk15        sliced    dsk15        rootdg       online
    dsk16        sliced    dsk16        rootdg       online
    dsk17        sliced    dsk17        rootdg       online
    dsk18        sliced    dsk18        rootdg       online
    dsk19        sliced    dsk19        rootdg       online
    dsk20        sliced    dsk20        rootdg       online
    dsk21        sliced    dsk21        rootdg       online
    dsk22        sliced    dsk22        rootdg       online
    dsk23        sliced    dsk23        rootdg       online
    dsk24        sliced    dsk24        rootdg       online
    

  3. Display the free space available in the rootdg disk group:

    # voldg -g rootdg free
    

    DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
    dsk0         dsk0         dsk0         1048576   7327392   -
    dsk1         dsk1         dsk1         1048576   3057792   -
    dsk2         dsk2         dsk2         0         8375968   -
    dsk3         dsk3         dsk3         0         8375968   -
    dsk4         dsk4         dsk4         0         4106368   -
    dsk5         dsk5         dsk5         0         8375968   -
    dsk6         dsk6         dsk6         0         8375968   -
    dsk7         dsk7         dsk7         0         8375968   -
    dsk8         dsk8         dsk8         0         4106368   -
    dsk9         dsk9         dsk9         0         8375968   -
    dsk10        dsk10        dsk10        0         8375968   -
    dsk11        dsk11        dsk11        0         4106368   -
    dsk12        dsk12        dsk12        0         4106368   -
    dsk13        dsk13        dsk13        0         4106368   -
    dsk14        dsk14        dsk14        0         4106368   -
    dsk15        dsk15        dsk15        0         8375968   -
    dsk16        dsk16        dsk16        6912      17762500  -
    dsk17        dsk17        dsk17        6912      17762500  -
    dsk18        dsk18        dsk18        6977      4099391   -
    dsk19        dsk19        dsk19        6912      8369056   -
    dsk20        dsk20        dsk20        6912      8369056   -
    dsk21        dsk21        dsk21        6912      8369056   -
    dsk22        dsk22        dsk22        0         8375968   -
    dsk23        dsk23        dsk23        0         8375968   -
    dsk24        dsk24        dsk24        0         17769412  -
    

  4. Choose sliced or simple disks with enough free space to create a volume with the characteristics you want (such as mirrored or striped).

    If possible, choose disks with an offset of 0. Avoid using disks with an offset, because these are in use by another volume. (To confirm this, enter the volprint command and examine the output for the disks in question.)

  5. Confirm that each disk you want to use is accessible by all cluster members:

    # hwmgr -view devices -cluster
    

     HWID: Device Name          Mfg      Model            Hostname   Location
    --------------------------------------------------------------------------------
        3: /dev/dmapi/dmapi                               ernest
        3: /dev/dmapi/dmapi                               joey
        4: scp                                            ernest
        5: kevm                                           ernest
       62: /dev/disk/floppy0c            3.5in floppy     ernest     fdi0-unit-0
       77: /dev/disk/dsk0c      COMPAQ   BB018122B7       ernest     bus-0-targ-0-lun-0
       78: /dev/disk/dsk1c      COMPAQ   BB018122B7       ernest     bus-0-targ-1-lun-0
       79: /dev/disk/dsk2c      DEC      RZ28M    (C) DEC ernest     bus-2-targ-1-lun-0
       79: /dev/disk/dsk2c      DEC      RZ28M    (C) DEC joey       bus-2-targ-1-lun-0
       80: /dev/disk/dsk3c      DEC      RZ28L-AS (C) DEC ernest     bus-2-targ-2-lun-0
       80: /dev/disk/dsk3c      DEC      RZ28L-AS (C) DEC joey       bus-2-targ-2-lun-0
       81: /dev/disk/dsk4c      DEC      RZ29B    (C) DEC ernest     bus-2-targ-3-lun-0
       81: /dev/disk/dsk4c      DEC      RZ29B    (C) DEC joey       bus-2-targ-3-lun-0
       82: /dev/disk/dsk5c      DEC      RZ28D    (C) DEC ernest     bus-2-targ-4-lun-0
       82: /dev/disk/dsk5c      DEC      RZ28D    (C) DEC joey       bus-2-targ-4-lun-0
       83: /dev/disk/dsk6c      DEC      RZ28L-AS (C) DEC ernest     bus-2-targ-5-lun-0
       83: /dev/disk/dsk6c      DEC      RZ28L-AS (C) DEC joey       bus-2-targ-5-lun-0
       84: /dev/disk/dsk7c      DEC      RZ1CF-CF (C) DEC ernest     bus-2-targ-8-lun-0
       84: /dev/disk/dsk7c      DEC      RZ1CF-CF (C) DEC joey       bus-2-targ-8-lun-0
       85: /dev/disk/dsk8c      DEC      RZ1CB-CS (C) DEC ernest     bus-2-targ-9-lun-0
       85: /dev/disk/dsk8c      DEC      RZ1CB-CS (C) DEC joey       bus-2-targ-9-lun-0
       86: /dev/disk/dsk9c      DEC      RZ1CF-CF (C) DEC ernest     bus-2-targ-10-lun-0
       86: /dev/disk/dsk9c      DEC      RZ1CF-CF (C) DEC joey       bus-2-targ-10-lun-0
       87: /dev/disk/dsk10c     DEC      RZ1CF-CF (C) DEC ernest     bus-2-targ-11-lun-0
       87: /dev/disk/dsk10c     DEC      RZ1CF-CF (C) DEC joey       bus-2-targ-11-lun-0
       88: /dev/disk/dsk11c     DEC      RZ1CF-CF (C) DEC ernest     bus-2-targ-12-lun-0
       88: /dev/disk/dsk11c     DEC      RZ1CF-CF (C) DEC joey       bus-2-targ-12-lun-0
       89: /dev/disk/dsk12c     DEC      RZ1CF-CF (C) DEC ernest     bus-2-targ-13-lun-0
       89: /dev/disk/dsk12c     DEC      RZ1CF-CF (C) DEC joey       bus-2-targ-13-lun-0
       90: /dev/disk/cdrom0c    COMPAQ   CRD-8402B        ernest     bus-3-targ-0-lun-0
       91: random                                         ernest
       92: urandom                                        ernest
       95: /dev/scp_scsi                                  joey
       96: /dev/kevm                                      joey
      147: /dev/disk/floppy1c            3.5in floppy     joey       fdi0-unit-0
      162: /dev/disk/dsk13c     COMPAQ   BB018122B7       joey       bus-0-targ-0-lun-0
      163: /dev/disk/dsk14c     COMPAQ   BB018122B7       joey       bus-0-targ-1-lun-0
      164: /dev/disk/cdrom1c    COMPAQ   CRD-8402B        joey       bus-3-targ-0-lun-0
      165: /dev/random                                    joey
      166: /dev/urandom                                   joey
    

  6. Migrate the domains.

    The options you specify for the migration depend on whether you want to mirror the volume in this step or wait and mirror the volume to the original disk.

Step 2: Mirroring the Volumes to the Original Disk (Optional)

The attributes you specify for a volume with the volmigrate command (such as the length and whether the volume is striped) are applied to the mirror when you mirror that volume.

If you create a striped volume on multiple disks, you need to specify the same number of additional disks to mirror the volume. For example, if the volume is striped over four disks, you need four additional disks to create the mirror; one of which can be the original disk.

To mirror a volume to the original disk:

  1. Confirm that all partitions on the original disk are unused (in this example, dsk6):

    # disklabel -r dsk6
    

  2. Add the disk to LSM:

    # voldisksetup -i dsk6
    

  3. Add the disk to the rootdg disk group. You do not need to specify a disk group because the rootdg disk group is assumed:

    # voldg adddisk dsk6
    

  4. Verify that there is enough space in the public region of the LSM disk to mirror the volume:

    # voldisk list dsk6 | grep public
    public:    slice=6 offset=16 len=8375968
    

  5. Verify the cluster volume names:

    # volprint -vt | grep cluster
    

    v  cluster_rootvol fsgen     ENABLED  ACTIVE   557936  SELECT    -
    v  cluster_usrvol  fsgen     ENABLED  ACTIVE   1684224 SELECT    -
    v  cluster_varvol  fsgen     ENABLED  ACTIVE   1667024 SELECT    -
    

    You use these names to mirror the volumes.

  6. Mirror the volumes. You must mirror each volume separately.

Step 3: Configure Hot-Spare Disks (Optional)

You can configure spare disks to automatically replace a failing disk in mirrored volumes.

Configure enough spare disks to replace more than one failed disk at any given time. Each spare disk should be as large as the largest disk in use by a volume in the disk group.

To configure one or more spare disks for the rootdg disk group:

# voledit set spare=on disk ...

For example, in the output from step 3 in Step 1: Migrate the Domains, disks dsk2, dsk3, and dsk5 (among others) have at least twice the amount of free space as the disks used in this Best Practice for creating the domain volumes. So, disks dsk2, dsk3, and dsk5 are suitable to configure as hot spares, as follows:

# voledit set spare=on dsk2 dsk3 dsk5

Verifying Success

Because the volmigrate command runs in verbose mode by default, it alerts you to problems as it executes.

You can additionally verify the results of the migration process by doing the following:

If this Best Practice was not successful, see Troubleshooting for information about identifying and solving problems.

Troubleshooting

If the volmigrate command or one of the commands that it calls returns an error, refer to the indicated command's reference page for more information about corrective steps.

The following list presents some of the error messages you might encounter and the action needed to solve each problem.

volmigrate: must specify an AdvFS domain name.

Problem: The domain name is not valid.

Solution: Verify that you entered the domain name correctly. Also, make sure you specify the domain name before the disk or disks on the command line.

volmigrate: the cluster root volume must be in rootdg.

Problem: The specified disk is not in the rootdg disk group or is not an LSM disk.

Solution: Enter the volprint or voldisk list command. Confirm that the disk is under LSM control and belongs to the rootdg disk group.

volmigrate: must specify LSM disk media names to use for volume.

Problem: The specified disk media name is incorrect.

Solution: Verify the disk media name with the voldisk list command. You must use the LSM disk's media name, which might differ from the name assigned to the disk by the operating system (the disk access name, or device name). In the output of the voldisk list command, the disk media name is in the DISK column.

lsm:volassist: ERROR: Cannot allocate space for length block volume

Problem: The specified disk or disks do not have enough free space to create a volume for the domain.

Solution: Use a larger LSM disk, specify additional LSM disks, or use the volmigrate -l length option to specify a smaller volume size. See the volmigrate(8) reference page for size limitations and restrictions.

If you tried to use the original disk to mirror one or more volumes but there was not enough free space, you can still mirror the volume to any other LSM simple or sliced disk that has enough space and use the original disk for other purposes, such as a spare disk.

volmigrate: length specified must be at least xxxxxx, or use the -f option to override 10% spare requirement. Specified size must be at least yyyyyy.

Problem: The specified volume size does not meet the minimum requirement of the in-use area of the domain (yyyyyy) plus 10 percent extra space for file system overhead.

Solution: If the volume size you specified is at least the size of the in-use area of the domain but provides less than 10 percent extra room, use the -f (force) option. You cannot specify a volume size smaller than the in-use area of the domain.

Alternative Practices

This Best Practice is the recommended method for placing the cluster root domains under LSM control.

The only alternative to using LSM to create a mirror of the clusterwide file systems is to use hardware RAID. See your hardware RAID documentation or Customer Support representative for more information.

Comments and Questions

We value your comments and questions on the information in this document. Please mail your comments to us at this address:

best_practices@zk3.dec.com

Legal Notice

Compaq, the Compaq logo, Tru64, and TruCluster are trademarks of Compaq Information Technologies Group, L.P. in the U.S. and/or other countries. UNIX is a trademark of The Open Group in the United States and other countries. All other product names mentioned herein may be trademarks of their respective companies.

Confidential computer software. Valid license from Compaq required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license.

Compaq shall not be liable for technical or editorial errors or omissions contained herein. The information is provided "as is" without warranty of any kind and is subject to change without notice. The warranties for Compaq products are set forth in the express limited warranty statements accompanying such products. Nothing herein should be construed as constituting an additional warranty.