newbook.document.writeln(''); newbook.document.close(); newbook.frames[2].location.href = targetFile; } // -->

1 Fibre Channel Overview

Fibre Channel is designed to carry various command sets and their associated architectures over the same physical interface. Fibre Channel is primarily a transport medium; therefore, it is independent of the function for which it is used.

The TruCluster Software Products use the Fibre Channel Protocol (FCP) for SCSI-3 to use Fibre Channel as the physical interface.

Fibre Channel, with its serial transmission method, overcomes the limitations of parallel SCSI by providing:

Data rates of 100 MB/sec, 200 MB/sec, and 400 MB/sec

Support for multiple protocols

Better scalability

Improved reliability, serviceability, and availability

Fibre Channel uses an extremely high transmit clock frequency to achieve the high data rate. Using optical fibre transmission lines allows the high-frequency information to be sent up to 70 km, which is the maximum distance between transmitter and receiver. Copper transmission lines may be used for shorter distances. (Tru64 UNIX does not support copper transmission lines.)

Note

The TruCluster Software Products support 500 meters between the KGPSA and the switch or hub, and 500 meters between the switch or hub and the HSG80 array controller. See Figure 1-4 for an example cluster configuration.

1.1 Basic Fibre Channel Terminology

This section describes the basic Fibre Channel terminology:

AL_PA: The Arbitrated Loop Physical Address (AL_PA) is used to address nodes on the Fibre Channel loop. When a node is ready to transmit data, it transmits Fibre Channel primitive signals that include its own identifying AL_PA.
Frame: All data is transferred in a packet of information called a frame. A frame is limited to 2112 bytes. Information that consists of more than 2112 bytes is divided into multiple frames.
Node: The source and destination of a frame. A node may be a computer system, redundant array of independent disks (RAID) array controller, or a disk device. Each node has a 64-bit unique node name (worldwide name) that is built into the node when it is manufactured.
N_Port: Each node must have at least one Fibre Channel port from which to send or receive data. This node port is called an N_Port. Each port has a 64-bit unique port name (worldwide name) that is built into the node when it is manufactured. An N_Port is connected directly to another N_Port in a point-to-point topology. An N_Port is connected to an F_Port in a fabric topology.
NL_Port: In an arbitrated loop topology, information is routed around a loop. A node port that can operate on the loop is called an NL_Port (node loop port). The information is repeated by each NL_Port until it reaches its destination. Each port has a 64-bit unique port name (worldwide name) that is built into the node when it is manufactured.
Fabric: A switch, or multiple interconnected switches, that route frames between the originator node (transmitter) and destination node (receiver). Fabrics do not originate nor are they the final recipient of frames; they pass frames on to the destination.
F_Port: The ports within the fabric (fabric port). The port is called an F_Port. Each F_Port has a 64-bit unique port name (worldwide name) that is built into the node when it is manufactured.
FL_Port: A fabric port that contains the loop functionality is called an FL_Port.
Link: The physical connection between an N_Port and another N_Port or between an N_Port and an F_Port. A link consists of two fibres - one to transmit information and one to receive information. The transmit fibre on one node is the receive fibre on the node at the other end of the link. A link may be optical fibre, coaxial cable, or shielded twisted pair.
E_Port: An expansion port on a switch that connects two switches in the fabric.

1.2 Fibre Channel Topologies

Fibre Channel supports three different interconnect topologies:

Point-to-point (Section 1.2.1)

Fabric (Section 1.2.2)

Arbitrated loop (Section 1.2.3)

Note

You can interconnect an arbitrated loop with fabric, creating a hybrid configuration. The fabric must have ports with loop functionality (FL_Ports) to attach an arbitrated loop to the fabric. Tru64 UNIX does not support hybrid configurations and they are not discussed in this document.

The following sections discuss these topologies.

1.2.1 Point-to-Point Topology

The point-to-point topology is the simplest of the Fibre Channel topologies. In a point-to-point topology, one N_Port is connected to another N_Port by a single link.

Frames require no routing; all frames transmitted by one N_Port are received by the other N_Port, and in the same order in which they were sent.

Figure 1-1 shows an example of point-to-point topology.

Figure 1-1: Point-to-Point Topology

1.2.2 Fabric Topology

The fabric topology provides more connectivity than point-to-point topology. The fabric topology allows up to 2²⁴ ports to be connected.

The fabric examines the destination address in the frame header and routes the frame to the destination node.

A fabric may consist of a single switch, or several interconnected switches. Each switch contains two or more fabric ports (F_Port) that are interconnected by the fabric switching function, which routes the frame from one F_Port to another F_Port within the fabric. Communication between two switches is routed between two expansion ports (E_Port).

When an N_Port is connected to a fabric F_Port, the fabric is responsible for the assignment of the Fibre Channel address to the N_Port that is attached to the fabric. The fabric is also responsible for selecting the route that the frame will take, within the fabric, to be delivered to the destination.

When the fabric consists of multiple switches, the fabric may determine an alternate route to ensure that a frame gets delivered to its destination.

Figure 1-2 shows an example of fabric topology.

Figure 1-2: Fabric Topology

1.2.3 Arbitrated Loop Topology

In an arbitrated loop topology, frames are routed around a loop that is set up by the links between the nodes. The hub maintains loop continuity by bypassing a node when the node or its cabling fails, when the node is powered down, or when the node is removed for maintenance. The hub is transparent to the protocol. It does not consume any Fibre Channel arbitrated loop addresses so it is not addressable by a Fibre Channel arbitrated loop port.

The nodes arbitrate to gain control (become master) of the loop. After a node becomes master, the nodes select (by way of setting bits in a bitmask) their own Arbitrated Loop Physical Address (AL_PA). The AL_PA is used to address nodes on the loop. The AL_PA is dynamic and can change each time the loop is initialized, a node is added or removed, or at any other time that an event causes the membership of the loop to change. When a node is ready to transmit data, it transmits Fibre Channel primitive signals that include its own identifying AL_PA.

In the arbitrated loop topology, a node port is called an NL_Port (node loop port), and a fabric port is called an FL_Port (fabric loop port).

Figure 1-3 shows an example of an arbitrated loop topology.

Figure 1-3: Arbitrated Loop Topology

1.3 Fibre Channel Topology Comparison

This section compares and contrasts the fabric and arbitrated loop topologies and describes why you might choose to use them.

When compared with the fabric (switched) topology, arbitrated loop is a lower cost, if lower performance, alternative. Arbitrated loop reduces Fibre Channel cost by substituting a lower-cost, often nonintelligent and unmanaged hub, for a more expensive switch. The hub operates by collapsing the physical loop into a logical star. The cables, associated connectors, and allowable cable lengths are similar to those of a fabric. Arbitrated loop supports a theoretical limit of 127 nodes in a loop. Arbitrated loop nodes are self-configuring and do not require Fibre Channel address switches.

Arbitrated loop provides reduced cost at the expense of bandwidth; all nodes in a loop share the bandwidth (100 MB/sec per loop), and bandwidth degrades slightly as nodes and cables are added. Nodes on the loop see all traffic on the loop, including traffic between other nodes. The hub can include port-bypass functions that manage movement of nodes on and off the loop. For example, if the port bypass logic detects a problem, the hub can remove that node from the loop without intervention. Data availability is then preserved by preventing the down time associated with node failures, cable disconnections, and network reconfigurations. Traffic caused by node insertion and removal, errors, and so forth, can cause temporary disruption on the loop.

Although more expensive, the fabric topology provides both increased connectivity and higher performance; switches provide a full-duplex 100 (200) MB/sec point-to-point connection to the fabric. Switches also provide improved performance and scaling because nodes on the fabric see only data destined for themselves, and individual nodes are isolated from reconfiguration and error recovery of other nodes within the fabric. Switches can provide management information about the overall structure of the Fibre Channel fabric, which may not be the case for an arbitrated loop hub.

Table 1-1 shows a comparison between the fabric and arbitrated loop topologies.

Table 1-1: Fibre Channel Fabric and Arbitrated Loop Comparison

When to Use Arbitrated Loop	When to Use Fabric
In clusters of up to two members	In clusters of more than two members
In applications where low total solution cost and simplicity are key requirements	In multinode cluster configurations when possible temporary traffic disruption due to reconfiguration or repair is a concern
In applications where the shared bandwidth of an arbitrated loop configuration is not a limiting factor	In high bandwidth applications where a shared arbitrated loop topology is not adequate
In configurations where expansion and scaling are not anticipated	In cluster configurations where expansion is anticipated and requires performance scaling

1.4 Example Fibre Channel Configurations Supported by the TruCluster Software Products

This section provides diagrams of some of the configurations that Tru64 UNIX and TruCluster Software Products Version 1.6 support.

Figure 1-4 shows a typical Fibre Channel cluster configuration using transparent failover mode.

Figure 1-4: Fibre Channel Single Switch Transparent Failover Configuration

In transparent failover, units D00 through D99 are accessed through port 1 of both controllers. Units D100 through D199 are accessed through port 2 of both HSG80 controllers (with the limit of a total of 128 storage units).

You cannot achieve no-single-point-of-failure (NSPOF) with the configuration shown in Figure 1-4 because the switch represents a single-point-of-failure.

You can, however, add the hardware for a second bus (another KGPSA, switch, and RA8000/ESA12000 with associated cabling) and use the Logical Storage Manager (LSM) to mirror across the buses to achieve a NSPOF cluster.

Figure 1-5 shows a transparent failover configuration set up to use LSM to mirror storage units on one RAID array controller to the other controller.

Figure 1-5: Mirroring Storage Units with LSM in a Cluster

Figure 1-6 shows a cluster with a full complement of eight targets on one Fibre Channel switch. Each member system represents one target, and each dual-redundant HSG80 port pair represents one target.

Figure 1-6: Single Switch Configuration with Full Complement

Figure 1-7 shows a two-node Fibre Channel arbitrated loop cluster with a single RA8000 or ESA12000 storage array with dual-redundant HSG80 controllers and an SWXHB-07 Fibre Channel hub.

Figure 1-7: Arbitrated Loop Configuration with One Storage Array

Figure 1-8 shows the maximum supported arbitrated loop configuration of a two-node Fibre Channel cluster with two RA8000 or ESA12000 storage arrays, each with dual-redundant HSG80 controllers and two SWXHB-07 Fibre Channel hubs.

Figure 1-8: Arbitrated Loop Maximum Configuration

1.5 Zoning and Cascaded Switches

This section provides a brief overview of zoning and cascaded switches.

1.5.1 Zoning

A zone is a logical subset of the Fibre Channel devices that are connected to the fabric. Zoning allows partitioning of resources for management and access control. In some configurations, it may provide for more efficient use of hardware resources by allowing one switch to serve multiple clusters or multiple operating systems.

Figure 1-9 provides an example configuration using zoning. This configuration consists of two independent zones with each zone containing an independent cluster.

Remember, with a configuration such as the one shown in Figure 1-9, the switch is a single point of failure.

Figure 1-9: A Simple Zoned Configuration

Note

Only static zoning is supported; zones can only be changed when all connected systems are shut down.

For information on setting up zoning, see the SAN Switch Zoning documentation that is provided with the switch.

1.5.2 Cascaded Switches

Multiple switches may be connected to each other. When cascading switches, a maximum of three switches is supported, with a maximum of two hops between switches. The maximum hop length is 10 km longwave single-mode or 500 meters shortwave multimode Fibre Channel cable.