4    Installing Fibre Channel Hardware and Configuring the emx Driver

The SWXHB-07 hub supports up to seven 1.6025 Gb/sec ports. The ports can be connected to the KGPSA-CA PCI-to-Fibre Channel host bus adapter or to an HSG80 array controller.

Unlike the DSGGA switch, the SWXHB-07 hub does not have any controls or even a power-on switch. Simply plug in the hub to power it on. The hub has a green power indicator on the front panel.

The SWXHB-07 hub has slots to accommodate up to seven plug-in interface converters. Each interface converter in turn supports two 1-Giga Bit Interface Converter (GBIC) modules. The GBIC module is the electrical-to-optical converter, and supports both 50-micron and 62.5-micron multimode fiber (MMF) using the standard SC connector. Only the 50-micron MMF optical cable is supported for the TruCluster Software Products.

The GBIC modules and MMF optical cables are not provided with the hub. If you need to obtain them, contact your authorized Compaq service representative.

4.2.1    Installing the Hub

Ensure that you place the hub within 500 meters of the member systems (with KGPSA-CA PCI-to-Fibre Channel adapter) and the HSG80 array controllers.

The SWXHB-07 hub can be placed on a flat, solid surface or, when configured in the DS-SWXHX-07 rack mount kit, part number 242795-B21, the hub can be mounted in a 48.7 cm (19-inch) rackmount installation. (One rack kit holds two hubs.) The hub is shipped with rubber feet to prevent marring the surface.

When you plan the hub location, ensure that you provide access to the GBIC connectors on the back of the hub. All cables plug into the back of the hub.

For an installation, at a minimum, you have to:

1. Place the hub on an acceptable surface or install it in a rack.

2. Install one or more GBIC modules. Gently push the GBIC module into an available port on the hub until you feel the GBIC module click into place. The GBIC module has a built-in guide key that prevents you from inserting it incorrectly. Do not use excessive force.

3. Connect the optical fiber cables. To do this, plug one end of an MMF cable into one of the GBIC modules installed in the hub. Attach an MMF cable for all active port connections. Unused ports or improperly seated GBIC modules remain in loop bypass and do not affect the operation of the loop.

4. Attach the other end of the MMF cable to either the KGPSA-CA adapter or to the HSG80.

5. Connect power to the hub using a properly grounded outlet. Look at the power indicator on the front of the hub to make sure that it powered on.

For more installation information, see the Fibre Channel Storage Hub 7 Installation Guide.

4.2.2    Determining the Hub Status

Because the SWXHB-07 hub is not a manageable unit, look at the LED indicators to make sure that the hub is operating correctly. The LED indicators will be particularly useful after you have connected the hub to the KGPSA-CA host adapters and the HSG80 controller. However, you can use the LEDs now to verify that the GBIC connectors are installed correctly.

At power on, with no optical cables attached, ensure that the green and amber LEDs are both on, indicating that the port is active but that the connection is invalid. The other possible LED states are as follows:

• Both off: Not active. Make sure that the GBIC is installed correctly.

• Solid green: Indicates presence and proper functionality of a GBIC.

• Green off: Indicates a fault condition (GBIC transmitter fault, improperly seated GBIC, no GBIC installed, or other failed device). The port is in bypass mode. This is the normal status for ports without GBICs installed.

• Solid amber: Indicates that a loss of signal or poor signal integrity has put the port in bypass mode. Make sure that a GBIC is installed, that a cable is attached to the GBIC, and that the other end of the cable is attached to a KGPSA or HSG80.

• Amber off (and green on): Indicates that the port and device are fully operational.

For more information on hub status, see the Fibre Channel Storage Hub 7 Installation Guide.

4.3    Setting Up the HSG80 Array Controller for a Fibre Channel Cluster

This section covers setting up the HSG80 controller for operation with Tru64 UNIX and the TruCluster products. For more information on installing the HSG80, see the HSG80 Array Controller ACS Version 8.4 Configuration and CLI Reference Guide or the HSG80 Array Controller ACS Version 8.5 CLI Reference Guide.

The steps described here apply to both fabric and arbitrated loop configurations. However, arbitrated loop requires specific settings for the port topology and AL_PA values. If this is an arbitrated loop configuration, follow the steps described here, taking note of the difference in the port topology setting. Then see Section 4.3.1 for additional information.

To set up an HSG80 for TruCluster operation, follow these steps:

1. If you have not already done so, install the HSG80 controllers into the RA8000 or ESA1200 storage arrays.

2. If you use an external cache battery (ECB), ensure that it is connected to the controller cache modules.

3. Install the fiber-optic cables between the switch and the HSG80.

4. Set the power verification and addressing (PVA) ID. Use PVA ID 0 for the enclosure that contains the HSG80 controller(s). Set the PVA ID to 2 and 3 on expansion enclosures (if present).

   Note

   Do not use PVA ID 1.

   With Port-Target-LUN (PTL) addressing, the PVA ID is used to determine the target ID of the devices on ports 1 through 6 (the LUN is always zero). Valid target ID numbers are 0 through 3 and 8 through 15. Target IDs 6 and 7 are reserved for the controller pair, and target IDs 4 and 5 are never used.

   The enclosure with PVA ID 0 will contain devices with target IDs 0 through 3; with PVA ID 2, target IDs 8 through 11; with PVA ID 3, target IDs 12 through 15. Setting a PVA ID of an enclosure to 1 sets target IDs to 4 through 7, generating a conflict with the target IDs of the controllers.

5. Remove the program card ESD cover and insert the controller's program card. Replace the ESD cover.

6. Install disks into storage shelves.

7. Connect a terminal to the maintenance port on one of the HSG80 controllers. You need a local connection to configure the controller for the first time. The maintenance port supports serial communication with the following default values:

   • 9600 bps

   • 8 data bits

   • 1 stop bit

   • No parity

8. Connect the RA8000 or ESA12000 to the power source and apply power.

   Note

   For the HSG80 to see the connection to the KGPSAs, the KGPSA host bus adapters must be cabled to the switch, with the system power applied before you turn power on to the RA8000/ESA12000.

9. If an uninterruptible power supply (UPS) is used instead of the external cache battery, enter the following command to prevent the controller from periodically monitoring the cache batteries after power is applied:

   > set this CACHE_UPS
    
   

   Note

   Setting the controller variable CACHE_UPS for one controller sets it for both controllers.

10. From the maintenance terminal, use the show this and show other commands to verify that controllers have the current firmware version. See the HSG80 Array Controller ACS Version 8.4 Maintenance and Service Guide for information on upgrading the firmware.

11. To ensure proper operation of the HSG80 with Tru64 UNIX and the TruCluster products, set the controller values as follows. Note that the port topology setting of fabric is specific to fabric configurations.

     set this scsi_version = scsi-2       [1]
     set this nocommand_console_lun       [2]
     set this port_1_profile = plda       [3]
     set this port_1_topology = fabric    [4]
     set this port_2_profile = plda       [3] 
     set this port_2_topology = fabric    [4]
     set other scsi_version = scsi-2      [1]
     set other nocommand_console_lun      [2]
     set other port_1_profile = plda      [3]
     set other port_1_topology = fabric   [4] 
     set other port_2_profile = plda      [3]
     set other port_2_topology = fabric   [4] 
     set failover copy = this_controller  [5]
     
    

    1. Specifies the host protocol to use. Also specifies how the Command Console LUN (CCL) is handled. When set to SCSI-2, the CCL is not fixed at a particular location, but floats depending on the configuration. For example, if D0 and D1 are defined on the controller (which map to LUN 0 and LUN 1), then the CCL device will appear at LUN 2. If D2 is then configured, the CCL device will appear at LUN 3. [Return to example]

    2. Disables the virtual LUN that is used with the command console. [Return to example]

    3. Defines the format used to present and interpret the LUN values in the Fibre Channel command information set. For the value of plda, the LUN number ranges from 0 through 255, and is contained in byte 1 of the 8-byte FCP_LUN field. [Return to example]

    4. For fabric, set the port topology to fabric. The default is LOOP_HARD. If this is an arbitrated loop configuration, see Section 4.3.1 for configuration differences. [Return to example]

    5. Specifies that the failover mode is transparent failover. With transparent failover, controller A (the upper controller) port 1 is active for units D0-D99. Controller A port 2 is passive for units D100-D199. Controller B port 1 is passive for units D0-D99 and controller B port 2 is active for units D100-D199.

       Keep in mind that, for Tru64 UNIX Version 4.0F, you can configure only units D0 through D7 and D100 through D107. [Return to example]

12. Execute the show connections command to determine the connection name, and then use the connection name to set the operating system and unit offset. In the following example, the connection name is !NEWCON05. The ! (exclamation mark) is part of the name.

    > show connections
    Connection                                                            Unit
    Name        Operating System   Controller   Port   Address   Status   Offset
    !NEWCON05   TRU64_UNIX         THIS         1      000001    OL this   0
                HOST_ID=1000-0000-C920-A6D9     ADAPTER_ID=1000-0000-C920-A6D9
    > set !NEWCON05 unit_offset = 0                  [1]
    > set !NEWCON05 operating_system = TRU64_UNIX    [2]
    > restart this                                   [3]
    > restart other                                  [3]
     
    

    1. Set the relative offset for LUN numbering to 0. [Return to example]

    2. Specify that the host environment that is connected to the Fibre Channel port is TRU64_UNIX. [Return to example]

    3. Restart both controllers to cause all changes to take effect. [Return to example]

    Use the show this and show other commands to verify the changes.

    Note

    You change the connection name with the HSG80 CLI RENAME command. For example, you change the connection name !NEWCON05 to indicate that it is the first KGPSA on host pepecilli as follows:

    HSG80> rename !NEWCON49 pep_pga_1
     
    

13. Set up the storage sets as required for the applications to be used.

4.3.1    Setting Up the HSG80 Array Controller for Arbitrated Loop

Section 4.3 describes settings that are common to both fabric and arbitrated loop configurations. This section describes settings that are unique to setting up the HSG80 controller for the arbitrated loop topology.

For more information on installing the HSG80 in an arbitrated loop topology, see the HSG80 Array Controller ACS Version 8.5 Configuration Guide.

To set up an HSG80 for TruCluster arbitrated loop operation, use the maintenance terminal to set the controller values as follows:

1. Set the PORT_x_TOPOLOGY value to LOOP_HARD. For example:

    set this port_1_topology = LOOP_HARD  
    set this port_2_topology = LOOP_HARD
    set other port_1_topology = LOOP_HARD   
    set other port_2_topology = LOOP_HARD   
    
   

   The PORT_x_TOPOLOGY value of LOOP_HARD enables arbitrated loop operation. Although the HSG80 controller also permits a topology setting of LOOP_SOFT, this is not supported in Tru64 UNIX.

2. Set PORT_x_AL_PA to unique values. PORT_x _AL_PA specifies the hexadecimal arbitrated loop physical address (AL_PA) for the HSG80 host ports.

   This is the preferred address, but the HSG80 controller can use whatever AL_PA it obtains during loop initialization. However, the address that you specify must be valid and must not be used by another port. If the controller is unable to obtain the address that you specify (for example, because two ports are configured for the same address), the controller cannot come up on the loop.

   In particular, if you do not set PORT_x_AL_PA, multiple ports might attempt to use the default address, thus causing a conflict.

   The valid AL_PA addresses are within the range of 0-EF (hexadecimal), but not all addresses within this range are valid; the default value is 69 (hexadecimal).

   The list of valid AL_PA addresses is as follows:

              0x01, 0x02, 0x04, 0x08, 0x0F, 0x10, 0x17, 0x18, 0x1B, 0x1D,
              0x1E, 0x1F, 0x23, 0x25, 0x26, 0x27, 0x29, 0x2A, 0x2B, 0x2C,
              0x2D, 0x2E, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x39, 0x3A,
              0x3C, 0x43, 0x45, 0x46, 0x47, 0x49, 0x4A, 0x4B, 0x4C, 0x4D,
              0x4E, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x59, 0x5A, 0x5C,
              0x63, 0x65, 0x66, 0x67, 0x69, 0x6A, 0x6B, 0x6C, 0x6D, 0x6E,
              0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x79, 0x7A, 0x7C, 0x80,
              0x81, 0x82, 0x84, 0x88, 0x8F, 0x90, 0x97, 0x98, 0x9B, 0x9D,
              0x9E, 0x9F, 0xA3, 0xA5, 0xA6, 0xA7, 0xA9, 0xAA, 0xAB, 0xAC,
              0xAD, 0xAE, 0xB1, 0xB2, 0xB3, 0xB4, 0xB5, 0xB6, 0xB9, 0xBA,
              0xBC, 0xC3, 0xC5, 0xC6, 0xC7, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD,
              0xCE, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD9, 0xDA, 0xDC,
              0xE0, 0xE1, 0xE2, 0xE4, 0xE8, 0xEF
    
   

   The convention in transparent failover mode is to use the same AL_PA address for Port 1 on both controllers and the same AL_PA address for Port 2 on both controllers. This convention allows the standby port on the alternate controller to have the same AL_PA address in the event of a failover. Because the ports are not active at the same time, the AL_PA addresses do not conflict.

   Do not make the AL_PA address that is assigned to Port 1 the same as that assigned to Port 2 because they are distinct devices on the Fibre Channel loop.

   The following example sets the PORT_x_AL_PA value for ports on two HSG80 controllers in transparent failover mode:

    set this PORT_1_AL_PA  = 01   
    set this PORT_2_AL_PA  = 02   
    set other PORT_1_AL_PA  = 01   
    set other PORT_2_AL_PA  = 02   
    
   

4.4    Obtaining the Worldwide Names of HSG80 Controllers

A worldwide name is a unique 64-bit number that the Institute of Electrical and Electronics Engineers (IEEE) assigns to a subsystem and the manufacturer sets prior to shipping. The worldwide name that is assigned to a subsystem never changes. Obtain and record the worldwide names of Fibre Channel components in case you need to verify their target ID mappings in the operating system.

Each HSG80 controller's worldwide name ends in 0, for example, 5000-1FE1-FF0C-EE00. The worldwide name (and checksum) of an HSG80 controller (also known as the node name) appears on a sticker placed above the controllers. You can also use the SHOW THIS_CONTROLLER Array Controller Software (ACS) command to display the worldwide name.

For HSG80 controllers, the controller port IDs are derived from the worldwide name. In a subsystem with two controllers in transparent failover mode, the controller port IDs increment as follows:

• Controller A and controller B, port 1--worldwide name + 1

• Controller A and controller B, port 2--worldwide name + 2

For example, using the worldwide name of 5000-1FE1-FF0C-EE00, the following port IDs are automatically assigned and shared between the ports as a REPORTED PORT_ID on each port:

• Controller A and controller B, port 1-- 5000-1FE1-FF0C-EE01

• Controller A and controller B, port 2-- 5000-1FE1-FF0C-EE02

4.5    Installing and Configuring the KGPSA PCI-to-Fibre Channel Adapter Module

When you install the KGPSA PCI-to-Fibre Channel adapter module, you have to ensure that it is configured symmetrically. The following sections discuss KGPSA installation and configuration.

For an arbitrated loop configuration, you need to configure the link type of the adapter for loop operation, as described in Section 4.5.3.

4.5.1    Obtaining the Worldwide Names of KGPSA Adapters

A worldwide name is a unique number that the Institute of Electrical and Electronics Engineers (IEEE) assigns to a subsystem and that the manufacturer sets prior to shipping. The worldwide name that is assigned to a subsystem never changes. Obtain and record the worldwide names of Fibre Channel components in case you need to verify their target ID mappings in the operating system.

Fibre Channel devices have both a node name and a port name worldwide name, both of which are 64-bit numbers. Most commands you use with Fibre Channel only show the port name.

There are multiple ways to obtain the KGPSA port name worldwide name:

• You can obtain the worldwide name from a label on the KGPSA module before you install it.

• You can use the show dev command as follows:

  P00>>> show dev
         
  .
  .
  .
  pga0.0.0.1.0       PGA0    WWN 1000-0000-c920-eda0
  pgb0.0.0.2.0       PGB0    WWN 1000-0000-c920-da01
   
  

• If the operating system is installed, the worldwide name of a KGPSA adapter is also displayed in the boot messages generated when the emx driver attaches to the adapter when the adapter's host system boots. Or, you can use the grep utility and obtain the worldwide name from the /var/adm/messages file as follows:

  # grep wwn /var/adm/messages
  F/W Rev 2.20X2(1.12): wwn 1000-0000-c920-eda0
  F/W Rev 2.20X2(1.12): wwn 1000-0000-c920-eda0
  F/W Rev 2.20X2(1.12): wwn 1000-0000-c920-eda0
         
  .
  .
  .
  
  

  Record the worldwide name of each KGPSA adapter for later use.

4.5.2    Installing the KGPSA PCI-to-Fibre Channel Adapter Module

Use the following steps as a guide to installing the KGPSA PCI-to-Fibre Channel adapter module. See either of the following manuals for more information:

• KGPSA-BC PCI-to-Optical Fibre Channel Host Bus Adapter User Guide

• 64-Bit PCI-to-Fibre Channel Host Bus Adapter User Guide

Caution

Static electricity can damage modules and electronic components. We recommend using a grounded antistatic wrist strap and a grounded work surface when you handle modules.

1. If necessary, install the mounting bracket on the KGPSA module. Place the mounting bracket tabs on the component side of the board. Insert the screws from the solder side of the board.

2. The KGPSA-BC should arrive with the gigabit link module (GLM) installed. If they are not, close the GLM ejector mechanism. Then, align the GLM alignment pins, alignment tabs, and connector pins with the holes, oval openings, and board socket. Press the GLM into place. The KGPSA-CA adapter has an embedded optical interface.

3. Install the KGPSA in an open 32-bit or 64-bit PCI slot.

4. Insert the fiber-optic cable SC connectors into the GLM of the KGPSA-BC adapter, or into the embedded optical interface of the KGPSA-CA adapter. The SC connectors are keyed to prevent their being plugged in incorrectly. Do not use unnecessary force. Remember to remove the transparent plastic covering on the extremities of the optical cable.

5. Connect the fiber-optic cables to the DSGGA switch or hub.

4.5.3    Configuring the KGPSA-CA Adapter for Loop Mode

If you do not want to use the KGPSA-CA adapter in loop mode, you can skip this section.

Before you can use the KGPSA adapter in loop mode, you must set the link type of the adapter to loop. You use the Alpha Systems Console World Wide ID Manager for Fibre Channel Devices (wwidmgr) to accomplish this task.

The version of the wwidmgr utility included with Version 5.6 and higher of the SRM console can set the KGPSA to run in arbitrated loop mode or in fabric mode. Specifically, the wwidmgr -set adapter command stores the selected topology into the nonvolatile random-access memory (NVRAM) storage on the KGPSA adapter. The adapter retains this setting even if the adapter is later moved to another system.

Although the Version 5.6 SRM console allows you to set the KGPSA link type to loop, the console does not otherwise directly support the Fibre Channel arbitrated loop topology. That is, after the KGPSA has been set to loop topology, console commands such as show device or quick assign will not display any host bus adapters and their attached devices. However, the KGPSA functions as expected.

Link Type

If a KGPSA in loop mode is connected to a Fibre Channel switch, the results are unpredictable. The same is true for a KGPSA in fabric mode that is connected to a loop. Therefore, determine the topology setting before you use the adapter.

The wwidmgr utility is documented in the Wwidmgr User's Manual, which is located in the DOC subdirectory of the Alpha Systems Firmware CD-ROM, Version 5.6 and later.

The steps required to set the link type are summarized here; see the Wwidmgr User's Manual for complete information and additional examples.

Assuming that you have the required Version 5.6 console firmware, use the wwidmgr utility to set the link type, as follows:

1. Display the adapter on the system to determine its configuration:

   POO>>> wwidmgr -show adapter
    
   item       adapter           WWN                Cur. Topo   Next Topo
   kgpsaa0.0.0.4.6 - Nvram read failed.
   [ 0]    kgpsaa0.0.0.4.6    1000-0000-c920-05ab    FABRIC     UNAVAIL
   [9999] All of the above.
    
    
   

   The warning message Nvram read failed indicates that the NVRAM on the KGPSA adapter has not been initialized and formatted. This is expected and is corrected when you set the adapter link type.

2. Set the link type on the adapter using the following values:

   • loop -- Sets the link type to loop (FC-AL)

   • fabric -- Sets the link type to fabric (point to point)

   You use the item number to indicate which adapter you want to change. For example, to configure adapter 0 (zero) for loop, use the following command:

   POO>>> wwidmgr -set adapter -item 0 -topo loop
    
   

   The item number 9999 refers to all adapters. If you have KGPSA adapters configured for both arbitrated loop and fabric topologies, selecting 9999 will set them all to loop mode.

3. Verify the adapter settings:

   POO>>> wwidmgr -show adapter
     item       adapter           WWN                Cur. Topo  Next Topo 
    [ 0]    kgpsaa0.0.0.4.6   1000-0000-c920-05ab     FABRIC     LOOP     
    
   

4. After making the change, reinitialize the console:

   POO>>> init
   

5. Boot the system. The emx driver (Version 1.12 or higher is required) displays a message at boot when it recognizes the console setting, and configures the link accordingly.

6. If this is a two-node TruCluster configuration, repeat this process for the other cluster member.

4.5.4    Configuring the KGPSA Adapter

Ensure that each KGPSA adapter is configured into the system symmetrically.

If you are adding Fibre Channel hardware to an existing ASE configuration, boot each member system to single-user mode after all Fibre Channel hardware has been installed, and the fiber-optic cables are connected between the switch or hub, KGPSA adapters, and HSG80 RAID controller. Run the ase_fix_config utility to ensure that the emx (KGPSA) adapters are configured symmetrically. For more information on the ase_fix_config utility, see the TruCluster Software Products Software Installation manual.

If this is an new installation and the software has not been installed, install the Fibre Channel hardware. When the TruCluster software is installed, the ase_fix_config utility is used to set up SCSI and Fibre Channel buses.

4.6    Configuring the emx Driver for Fibre Channel Operation

For the TruCluster Software Products for Version 1.6 to operate correctly, the SCSI and Fibre Channel adapters must be configured symmetrically.

The SCSI adapters have to be connected symmetrically, and the KGPSA Fibre Channel adapters (emxn) must also be connected symmetrically. For example, all emx0 adapters must be connected to the same switch, and all emx1 adapters must be connected to the same switch (which is not the same switch to which the emx0 adapters are connected). See Figure 1-5 for an example configuration.

The worldwide name-to-target ID mapping on each cluster member must be symmetrically assigned.

For more information on the emx driver, see emx(7).

In a typical shared parallel SCSI configuration, the SCSI target IDs are assigned by means of console-level commands, console firmware settings, jumpers, or other hardware implementations. In a Fibre Channel configuration, the Tru64 UNIX system uses a number of files to create and maintain a persistent mapping of target IDs to Fibre Channel device worldwide names. This mechanism may appear to be a bit out of the ordinary. However, the general concepts are virtually identical, and this is an important consideration when you install or replace Fibre Channel-based hardware.

4.6.1    Initial Worldwide Name-to-Target ID Mapping

After you install Fibre Channel hardware, the worldwide name-to-target ID mapping is set up, on a first-device-encountered basis, the first time that the member system is booted. The software does the mapping so you do not have to take any action. After the system is operational, the target ID mapping is written to the emx driver data and backup files, /etc/emx.db and /etc/emx.db.bak. The database allows the mapping to persist across power cycles and reboots. A text version of the database file is written to /etc/emx.info.

The mapping is done on a first-device-encountered basis. Therefore, if you boot all cluster member systems, each system might have a different target ID mapping and the contents of the /etc/emx.db and /etc/emx.db.bak files might be different between systems. To prevent this, the assignment of target ID mapping must be synchronized.

4.6.2    Synchronizing the Initial Assignment of Target IDs in a Cluster

To ensure that all cluster member systems have the same target ID mapping at initial boot, follow these steps. In the case of arbitrated loop configurations, a maximum of two member systems is supported. However, the same process applies.

1. Make sure that all member systems are powered up and fully connected to shared storage. Additionally, make sure that all HSG80 controllers have storage sets defined, are powered up, and properly connected to a Fibre Channel switch or hub.

2. Boot all cluster member systems but one to single-user mode as follows:

   >>> boot -fl s
    
   

   Note

   Booting all cluster members but one to single-user mode is critical because the member that will be booted to multi-user mode must see the KGPSA adapters of every other member system to create target IDs for them.

3. Boot the remaining cluster member system to multi-user mode. The initial target ID mapping takes place and is written to the emx drivers data and backup files, /etc/emx.db and /etc/emx.db.bak (binary files).

4. Create device special files as needed on the member system (see Section 4.8). They should have been created as part of the boot process. If they have not been created, use the scsimgr(8) utility to automatically create them or use the MAKEDEV(8) command to manually create them.

5. On each member system at single-user mode, mount the file systems, set the root file system writeable, and start the network by entering the following commands on each of those systems:

   # /sbin/lsmbstartup
   # /sbin/bcheckrc
   # /sbin/init.d/inet start
    
   

   Note

   You only have to execute /sbin/lsmbstartup if you are using the Logical Storage Manager (LSM).

6. Use the ftp utility to copy the /etc/emx.db and /etc/emx.db.bak files from the system at multi-user mode to each of the other systems to synchronize the target ID mapping.

7. Shut down to a halt each of the member systems that are still at single-user mode. Then reboot each of them to multi-user mode. (The shutdown is required.)

8. Create device special files as needed on the member systems (see Section 4.8). They should have been created as part of the boot process. If they have not been created, use the scsimgr(8) utility to automatically create them or use the MAKEDEV(8) command to manually create them.

Note

You can synchronize the target ID mapping across cluster members by copying the /etc/emx.db and /etc/emx.db.bak database files because:

• The TruCluster software has configured the emx adapters to the same SCSI bus number on all hosts (for example, emx0 is scsi3 and emx1 is scsi4).

• Each member system is cabled symmetrically to each shared storage bus (for example, the emx0 adapter on each member system is cabled to a common switch and emx1 adapter on each member is also cabled to a common switch).

The worldwide name-to-target ID mapping thus established persists across member and cluster reboots. You do not need to change it unless you add or replace KGPSA adapters, explicitly modify target IDs, or perform a wholescale swap of an RA8000 or ESA12000.

4.6.3    Remapping Target IDs After Adding or Replacing a Fibre Channel Device

When you modify the cluster Fibre Channel storage configuration by adding or replacing a Fibre Channel device, you potentially disrupt the worldwide name-to-target ID mapping that was established when the cluster was established, causing this mapping to go out of synchronization across cluster members.

This section describes the general procedure to reestablish a consistent mapping within a cluster. Section 5.3.3 discuss the specific tasks for adding or replacing Fibre Channel devices in a cluster.

Note

You must complete the procedures in Section 5.3.3 before you use the following procedure.

To modify the worldwide name-to-target ID mappings after you add or replace a Fibre Channel adapter, follow these steps:

1. Boot the member system to single-user mode:

   >>> boot -fl s
    
   

2. Mount the file systems, set the root file system writable, and start the network by entering the following commands:

   # /sbin/lsmbstartup
   # /sbin/bcheckrc
   # /sbin/init.d/inet start
    
   

   Note

   You only have to execute /sbin/lsmbstartup if you are using the Logical Storage Manager (LSM).

3. Make backup copies of the following emx driver files:

   • /etc/emx.db

   • /etc/emx.db.bak

   • /etc/emx.info.

4. Use the emxmgr utility to display the topology for the emx for which changes were made. The new KGPSA Fibre Channel adapter or RA8000 (ESA12000) appears in the place of the old KGPSA or RAID array in the display. The SCSI target ID may be wrong, and might even be out of the 0-7 range. The replaced device appears below the dotted line because it still has a SCSI ID assigned, but it is not present in the configuration. In the following example, a KGPSA host bus adapter has been replaced:

   # /usr/sbin/emxmgr -t emx1
    
   emx1 state information:                             [1]
     Link :  connection is UP     
             Point to Point
             Fabric attached
             FC DID 0x210413
     Link is SCSI bus 3 (e.g. scsi3)
             SCSI target id 3
             portname is 1000-0000-C921-18A4
             nodename is 1000-0000-C921-18A4
     N_Port at FC DID 0x210013 - SCSI tgt id 5 :
       portname 5000-1FE1-0001-8932
       nodename 5000-1FE1-0001-8930
       Present, Logged in, FCP Target, FCP Logged in, 
   
   .
   .
   .
   N_Port at FC DID 0x210513 - SCSI tgt id 6 :
       portname 1000-0000-C921-07F4
       nodename 2000-0000-C921-07F4
       Present, Logged in, FCP Initiator, FCP Target, FCP Logged in, 
     ----------------------
     The following N_Ports have SCSI target ids assigned but have
     not been seen in the topology...
       SCSI tgt id   7 : portname 1000-1FE1-C921-07C4  [2]
                         nodename 1000-1FE1-C921-07C4
    
   

   1. The new KGPSA adapter has the wrong SCSI ID. The port and node names of the new adapter are provided. You need the new port name and node name for the remapping. [Return to example]

   2. The old KGPSA adapter still has worldwide name-to-target ID mapping information in the kernel and database files, but it is not seen. You need to use the emxmgr utility to set the new KGPSA adapter to the correct SCSI ID (7) shown here. [Return to example]

5. Create a file that contains the worldwide name-to-target ID mapping information obtained from the emxmgr -t display. The file can contain entries in either of the following formats:

   emx1 7 1000-0000-C921-18A4 1000-0000-C921-18A4
    
   { 0, 7, 0x0010, 0x0000, 0x21C9, 0xA418, 0x0010, 0x0000, 0x21C9, 0xA418 }
    
   

6. Modify the worldwide name-to-target ID mapping for the new devices by executing the emxmgr utility using the file that you just created (for example, call the file /tmp/KGPSA-SCSI-3-remapping):

   # /usr/sbin/emxmgr -c /tmp/KGPSA-SCSI-3-remapping
    
     Continue with the change in id mappings (y/n) [n] ? y
    
     Change in Target id mappings has been performed
   #
    
   

   Note

   You can also run the emxmgr in interactive mode and use the file or type the entries when you are prompted.

7. Boot the system to multi-user mode.

   Note

   You must now distribute this new target ID mapping to the other cluster members by performing the following two steps on each remaining cluster member.

8. Use the ftp utility to copy the mapping file that you created (/tmp/KGPSA-SCSI-3-remapping) to another cluster member system.

9. Modify the worldwide name-to-target ID mapping for the new devices by executing the emxmgr utility using the file that you just copied:

   # /usr/sbin/emxmgr -c /tmp/KGPSA-SCSI-3-remapping
    
     Continue with the change in id mappings (y/n) [n] ? y
    
     Change in Target id mappings has been performed
   #
    
   

4.7    Verifying the Fibre Channel Device Configuration

After you have synchronized the target ID mapping across cluster members, verify that the Fibre Channel storage has been correctly configured. The following are some suggested methods:

• Use the emxmgr -t utility on each member system to display the Fibre Channel topology and compare the displays.

• Use the scu utility to examine various targets and LUNs to verify that the newly generated kernel has brought the Fibre Channel storage on line as expected. For example:

  # scu
  scu> scan edt
  scu> show edt
  Ctrl/D
   
  

• Use the uerf or dia utility to verify the device configuration by displaying the startup messages in reverse order (so that the most recent messages are displayed first):

  # dia -R -i osf=300 | more
   
  

• Use the dd command to generate some I/O activity by reading from the Fibre Channel disks. Visually verify that the correct disk or set of disks is active. For example:

  # dd if=/dev/rrzb49c of=/dev/null
   
  

4.8    Determining the Device Special File Name for a Disk

The actual device number of a particular disk will be determined both by:

• The SCSI target assigned by the method discussed in Section 4.6

• The bus number assignment as determined (in the traditional manner) by the system configuration file.

Using the cluster's SCSI bus assignments, the data that is recorded in the /etc/emx.info, and the list of Fibre Channel worldwide names that you compiled in Section 4.4, you can derive a device's special file name as follows:

(SCSI bus # for the emx instance) * 8 +
(target id # for the HSG80 controller on the emx instance), 
prefixed by the lun value
("" for lun 0, "b" = lun 1, "c" lun 2, etc).
 

Thus, if an HSG80 is at target 1 (as determined by /etc/emx_info) on scsi6 (as determined by the system configuration file, which could assign scsi6 to device emx0, which is the first KGPSA adapter), then its LUN 0 is /dev/rrz49 and its LUN 1 is /dev/rrzb49.

These device assignments are seldom changed throughout the life of the installation. Although you can change these device assignments with the emxmgr utility, consider such changes carefully, plan ahead of time, and do so deliberately.