This chapter provides an overview of Fibre Channel and Fibre
Channel configuration examples.
3.1 Overview
Fibre Channel supports multiple protocols over the same physical interface. Fibre Channel is primarily a protocol-independent transport medium; therefore, it is independent of the function for which you use it.
TruCluster Server uses the Fibre Channel Protocol (FCP) for SCSI to use Fibre Channel as the physical interface.
Fibre Channel, with its serial transmission method, overcomes the limitations of parallel SCSI by providing:
Support for multiple protocols
Better scalability
Improved reliability, serviceability, and availability
Fibre Channel uses an extremely high-transmit clock frequency to
achieve the high data rate.
Using optical fiber transmission
lines allows the high-frequency information to be sent up to 40
km (24.85 mi), the maximum distance between transmitter and
receiver.
Copper transmission lines may be used for shorter
distances.
3.2 Basic Fibre Channel Terminology
The following list describes the basic Fibre Channel terminology:
The Arbitrated Loop Physical Address (AL_PA) is used to address nodes on the Fibre Channel loop. When a node is ready to transmit data, it transmits Fibre Channel primitive signals that include its own identifying AL_PA.
A Fibre Channel topology in which frames are routed around a loop set up by the links between the nodes in the loop. All nodes in a loop share the bandwidth, and bandwidth degrades slightly as nodes and cables are added.
All data is transferred in a packet of information called a frame. A frame is limited to 2112 bytes. If the information consists of more than 2112 bytes, it is divided up into multiple frames.
The source and destination of a frame. A node may be a computer system, a redundant array of independent disks (RAID) array controller, or a disk device. Each node has a 64-bit unique node name (worldwide name) that is built into the node when it is manufactured.
Each node must have at least one Fibre Channel port from which to send or receive data. This node port is called an N_Port. Each port is assigned a 64-bit unique port name (worldwide name) when it is manufactured. An N_Port is connected directly to another N_Port in a point-to-point topology. An N_Port is connected to an F_Port in a fabric topology.
In an arbitrated loop topology, information is routed around a loop. A node port that can operate on the loop is called an NL_Port (node loop port). The information is repeated by each NL_Port until it reaches its destination. Each port has a 64-bit unique port name (worldwide name) that is built into the node when it is manufactured.
A switch, or multiple interconnected switches, that route frames between the originator node (transmitter) and destination node (receiver).
The ports within the fabric (fabric port). This port is called an F_port. Each F_port is assigned a 64-bit unique node name and a 64-bit unique port name when it is manufactured. Together, the node name and port name make up the worldwide name.
An F_Port containing the loop functionality is called an FL_Port.
The physical connection between an N_Port and another N_Port or an N_Port and an F_Port. A link consists of two connections, one to transmit information and one to receive information. The transmit connection on one node is the receive connection on the node at the other end of the link. A link may be optical fiber, coaxial cable, or shielded twisted pair.
An expansion port on a switch used to make a connection between two switches in the fabric.
Fibre Channel supports three different interconnect topologies:
Point-to-point (Section 3.3.1)
Fabric (Section 3.3.2)
Arbitrated loop (Section 3.3.3)
Note
Although you can interconnect an arbitrated loop with fabric, hybrid configurations are not supported at the present time, and therefore are not discussed in this manual.
The point-to-point topology is the simplest Fibre Channel topology. In a point-to-point topology, one N_Port is connected to another N_Port by a single link.
Because all frames transmitted by one N_Port are received by the other N_Port, and in the same order in which they were sent, frames require no routing.
Figure 3-1
shows an example point-to-point
topology.
Figure 3-1: Point-to-Point Topology
The fabric topology provides more connectivity than point-to-point topology. The fabric topology can connect up to 224 ports.
The fabric examines the destination address in the frame header and routes the frame to the destination node.
A fabric may consist of a single switch, or there may be several interconnected switches (up to three interconnected switches are supported). Each switch contains two or more fabric ports (F_Port) that are internally connected by the fabric switching function, which routes the frame from one F_Port to another F_Port within the switch. Communication between two switches is routed between two expansion ports (E_Ports).
When an N_Port is connected to an F_Port, the fabric is responsible for the assignment of the Fibre Channel address to the N_Port attached to the fabric. The fabric is also responsible for selecting the route a frame will take, within the fabric, to be delivered to the destination.
When the fabric consists of multiple switches, the fabric can determine an alternate route to ensure that a frame gets delivered to its destination.
Figure 3-2
shows an example fabric topology.
Figure 3-2: Fabric Topology
3.3.3 Arbitrated Loop Topology
In an arbitrated loop topology, frames are routed around a loop set up by the links between the nodes. The hub maintains loop continuity by bypassing a node when the node or its cabling fails, when the node is powered down, or when the node is removed for maintenance. The hub is transparent to the protocol. It does not consume any Fibre Channel arbitrated loop addresses so it is not addressable by a Fibre Channel arbitrated loop port.
The nodes arbitrate to gain control (become master) of the loop. After a node becomes master, the nodes select (by way of setting bits in a bitmask) their own Arbitrated Loop Physical Address (AL_PA). The AL_PA is used to address nodes on the loop. The AL_PA is dynamic and can change each time the loop is initialized, a node is added or removed, or at any other time that an event causes the membership of the loop to change. When a node is ready to transmit data, it transmits Fibre Channel primitive signals that include its own identifying AL_PA.
In the arbitrated loop topology, a node port is called an NL_Port (node loop port), and a fabric port is called an FL_Port (fabric loop port).
Figure 3-3
shows an example of an arbitrated
loop topology.
Figure 3-3: Arbitrated Loop Topology
3.4 Fibre Channel Topology Comparison
This section compares and contrasts the fabric and arbitrated loop topologies and describes why you might choose to use them.
When compared with the fabric (switched) topology, arbitrated loop is a lower cost, and lower performance, alternative. Arbitrated loop reduces Fibre Channel cost by substituting a lower-cost, often nonintelligent and unmanaged hub, for a more expensive switch. The hub operates by collapsing the physical loop into a logical star. The cables, associated connectors, and allowable cable lengths are similar to those of a fabric. Arbitrated loop supports a theoretical limit of 127 nodes in a loop. Arbitrated loop nodes are self-configuring and do not require Fibre Channel address switches.
Arbitrated loop provides reduced cost at the expense of bandwidth; all nodes in a loop share the bandwidth (100 MB/sec per loop), and bandwidth degrades slightly as nodes and cables are added. Nodes on the loop see all traffic on the loop, including traffic between other nodes. The hub can include port-bypass functions that manage movement of nodes on and off the loop. For example, if the port bypass logic detects a problem, the hub can remove that node from the loop without intervention. Data availability is then preserved by preventing the down time associated with node failures, cable disconnections, and network reconfigurations. However, traffic caused by node insertion and removal, errors, and so forth, can cause temporary disruption on the loop.
Although the fabric topology is more expensive, it provides both increased connectivity and higher performance; switches provide a full-duplex 100 (200) MB/sec point-to-point connection to the fabric. Switches also provide improved performance and scaling because nodes on the fabric see only data destined for themselves, and individual nodes are isolated from reconfiguration and error recovery of other nodes within the fabric. Switches can provide management information about the overall structure of the Fibre Channel fabric, which may not be the case for an arbitrated loop hub.
Table 3-1
presents a comparison between the fabric and
arbitrated loop topologies.
Table 3-1: Fibre Channel Fabric and Arbitrated Loop Comparison
| When to use Arbitrated Loop | When to use Fabric |
| In clusters of up to two members | In clusters of more than two members |
| In applications where low total solution cost and simplicity are key requirements | In multinode cluster configurations when possible temporary traffic disruption due to reconfiguration or repair is a concern |
| In applications where the shared bandwidth of an arbitrated loop configuration is not a limiting factor | In high bandwidth applications where a shared arbitrated loop topology is not adequate |
| In configurations where expansion and scaling are not anticipated | In cluster configurations where expansion is anticipated and requires performance scaling |
3.5 Example Fibre Channel Configurations Supported by TruCluster Server
This section provides diagrams of some of the configurations supported
by TruCluster Server Version 5.1.
Diagrams are provided for both
transparent failover mode and multiple-bus failover mode.
3.5.1 Fibre Channel Cluster Configurations for Transparent Failover Mode
With transparent failover mode:
The hosts do not know a failover has taken place (failover is transparent to the hosts).
The units are divided between an HSG80 port 1 and port 2.
If there are dual-redundant HSG80 controllers, controller A port 1 and controller B port 2 are normally active; controller A port 2 and controller B port 1 are normally passive.
If one controller fails, the other controller takes control and both its ports are active.
Figure 3-4
shows a typical Fibre Channel
cluster configuration using transparent failover mode.
Figure 3-4: Fibre Channel Single Switch Transparent Failover Configuration
In transparent failover, units D00 through D99 are accessed through port 1 of both controllers. Units D100 through D199 are accessed through port 2 of both HSG80 controllers.
You cannot achieve a no-single-point-of-failure (NSPOF) configuration using transparent failover. The host cannot initiate failover, and if you lose a host bus adapter, switch or hub, or a cable, you lose the units behind at least one port.
You can, however, add the hardware for a second bus (another KGPSA,
switch, and RA8000/ESA12000 with associated cabling) and use LSM to
mirror across the buses.
However, because you cannot use LSM to
mirror the cluster root (/) file system, member
boot partitions, the quorum disk, or swap partitions you cannot obtain
an NSPOF transparent failover configuration, even though you have
increased availability.
Figure 3-5
shows a two-node Fibre Channel
cluster with a single RA8000 or ESA12000 storage array with
dual-redundant HSG80 controllers and an DS-SWXHB-07 Fibre Channel hub.
Figure 3-5: Arbitrated Loop Configuration with One Storage Array
3.5.2 Fibre Channel Cluster Configurations for Multiple-Bus Failover Mode
With multiple-bus failover:
The host controls the failover by accessing units over a different path or causing the access to the unit to be through the other HSG80 controller.
An active controller causes a failover to the other controller if the controller recognizes the loss of the switch, hub, or cable to a controller port.
Each cluster member system has two or more (fabric only) KGPSA host bus adapters (multiple paths to the storage units).
Normally, all available units (D0 through D199) are available at all host ports. Only one HSG80 controller will be actively doing I/O for any particular storage unit.
However, both controllers can be forced active by preferring units to one
controller or the other (SET
unit
PREFERRED_PATH=THIS).
By balancing the preferred units, you
can obtain the best I/O performance using two controllers.
Note
If you have preferred units, and the HSG80 controllers restart because of an error condition or power failure, and one controller restarts before the other controller, the HSG80 controller restarting first will take all the units, whether they are preferred or not. When the other HSG80 controller starts, it will not have access to the preferred units, and will be inactive.
Therefore, you want to ensure that both HSG80 controllers start at the same time under all circumstances so that the controller sees its own preferred units.
Figure 3-6 and Figure 3-7 show two different recommended multiple-bus NSPOF cluster configurations. The only difference is the fiber-optic cable connection path between the switch and the HSG80 controller ports.
There is no difference in performance between these two configurations. It may be easier to cable the configuration shown in Figure 3-6 because the cables from one switch (or switch zone) both go to the ports on the same side of both controllers (for example, port 1 of both controllers).
Figure 3-6: Multiple-Bus NSPOF Configuration Number 1
Figure 3-7: Multiple-Bus NSPOF Configuration Number 2
The configuration shown in Figure 3-8 is not a recommended cluster configuration because it has multiple single-points-of-failure that can cause a loss of access to storage.
Figure 3-8, in a single-system configuration, is not an NSPOF configuration.
Note
Previous documentation erroneously showed Figure 3-8 as an NSPOF configuration.
If you have a configuration like the one shown in Figure 3-8, change the switch to HSG80 cabling to match the configurations shown in Figure 3-6 or Figure 3-7.
The single-system configuration shown in Figure 3-9 is also a configuration that is not recommended. It also has a single point of failure.
Note
If you have a configuration like the one shown in Figure 3-9, convert it to a recommended configuration by adding the following two cables:
From the left-hand switch to controller B Port 2.
From the right-hand switch to controller A Port 2.
Figure 3-8: A Configuration That Is Not Recommended
Figure 3-9: Another Configuration That Is Not Recommended
Figure 3-10
shows the maximum
supported arbitrated loop configuration of a two-node Fibre Channel
cluster with two RA8000 or ESA12000 storage arrays, each with
dual-redundant HSG80 controllers and two DS-SWXHB-07 Fibre Channel hubs.
This provides a NSPOF configuration.
Figure 3-10: Arbitrated Loop Maximum Configuration
QuickLoop supports Fibre Channel arbitrated loop (FC-AL) devices within a fabric. This logical private loop fabric attach (PLFA) consists of multiple private arbitrated loops (looplets) that are interconnected by a fabric. A private loop is formed by logically connecting ports on up to two switches.
Note
QuickLoop is not supported in a Tru64 UNIX Version 5.1 or TruCluster Server Version 5.1 configuration.
This section provides a brief overview of zoning.
A zone is a logical subset of the Fibre Channel devices that are connected to the fabric. Zoning allows partitioning of resources for management and access control. In some configurations, it may provide for more efficient use of hardware resources by allowing one switch to serve multiple clusters or even multiple operating systems. Zoning entails splitting the fabric into zones, where each zone is essentially a virtual fabric.
Zoning may be used:
When you want to set up barriers between systems of different operating environments or uses, for instance to allow two clusters to utilize the same switch.
To create test areas that are separate from the rest of the fabric.
To provide better utilization of a switch by reducing the number of unused ports.
Note
Any initial zoning must be made before connecting the host bus adapters and the storage to the switches, but after zoning is configured, changes can be made dynamically.
3.7.1 Switch Zoning Versus Selective Storage Presentation
Switch zoning and the selective storage presentation (SSP) feature of the HSG80 controllers have similar functions.
Switch zoning controls which servers can communicate with each other and each storage controller host port. SSP controls which servers will have access to each storage unit.
Switch zoning controls access at the storage system level, whereas SSP controls access at the storage unit level.
The following configurations require zoning or selective storage presentation:
When you have a TruCluster Server cluster in a storage array network (SAN) with other stand-alone systems (UNIX or non-UNIX), or other clusters.
Any time you have Windows NT or Windows 2000 in the same SAN with Tru64 UNIX. (Windows NT or Windows 2000 must be in a separate switch zone.)
The SAN configuration has more than 64 connections to an RA8000, ESA12000, MA6000, MA8000, or EMA12000.
The use of selective storage presentation is the preferred way to
control access to storage (so zoning is not required).
3.7.2 Types of Zoning
There are two types of zoning, soft and hard:
Soft zoning is a software implementation based on the Simple Name Server (SNS) enforcing a zone. Zones are defined by either the node or port World Wide Names (WWN), or the domain and port numbers in the form of D,P, where D is the domain and P is the physical port number on the switch.
A host system requests a list of all adapters and storage controllers that are connected to the fabric. The name service provides a list of all ports that are in the same zone or zones as the requesting host bus adapter.
Soft zoning only works if all hosts honor it; it does not work if a host is not programmed to allow for soft zoning. For instance, if a host tries to access a controller that is outside the zone, the switch does not prevent the access.
Tru64 UNIX honors soft zoning and does not attempt to access devices outside the zone.
If you have used the WWN to define the zone and replace a KGPSA host bus adapter, you must modify the zone configuration and SSP because the node World Wide Name has changed.
With hard zoning, zones are enforced at the physical level across all fabric switches by hardware blocking of Fibre Channel frames. Hardware zone definitions are in the form of D,P, where D is the domain and P is the physical port number on the switch. An example might be 1,2 for switch 1, port 2.
If a host attempts to access a port that is outside its zone, the switch hardware blocks the access.
You must modify the zone configuration if you move any cables from one port to another within the zone.
If you want to guarantee that there is no access outside any zone, either use hard zoning, or use operating systems that state that they support soft zoning.
Table 3-2
lists the
types of zoning that are supported on each of the supported Fibre
Channel switches.
Table 3-2: Type of Zoning Supported by Switches
| Switch Type | Type of Zoning Supported | ||
| DS-DSGGA | Soft | ||
| DS-DSGGB | Soft and Hard | ||
| DS-DSGGC | Soft and Hard |
Figure 3-11
provides an example configuration using
zoning.
This configuration consists of two
independent zones with each zone containing an independent cluster.
Figure 3-11: A Simple Zoned Configuration
For information on setting up zoning, see the SAN Switch Zoning
documentation that is provided with the switch.
3.8 Cascaded Switches
Multiple switches may be connected to each other to form a network of switches, or cascaded switches.
A cascaded switch configuration, which allows for network failures
up to and including the switch without losing a data path to a
SAN connected node, is called a mesh or meshed fabric.
Figure 3-12
shows an example meshed fabric with
three cascaded switches.
This is not a
no-single-point-of-failure (NSPOF) configuration.
Figure 3-12: Meshed Fabric with Three Cascaded Switches
Figure 3-13
shows an example meshed
resilient fabric with four cascaded interconnected switches.
This configuration will tolerate multiple data path failures, and
is an NSPOF configuration.
Figure 3-13: Meshed Resilient Fabric with Four Cascaded Switches
Note
If you lose an ISL, the communication can be routed through another switch to the same port on the other controller. This can constitute the maximum allowable two hops.
You can find the following information about storage array networks (SAN) in the Compaq StorageWorks Heterogeneous Open SAN Design Reference Guide located at:
http://www5.compaq.com/products/storageworks/techdoc/san/AA-RMPNA-TE.html
Supported SAN topologies
SAN fabric design rules
SAN platform and operating system restrictions (including the number of switches supported)