X3T9.2/88-160 December 2, 1988 To: X3T9.3 Members Subject: Fiber Channel The Fiber Channel working group has gathered a considerable amount of information on the technology available to implement a Fiber Channel, and has also studied various planned and implemented applications. The following attachment represents a summation of the characteristics which we expect to provide in the standard, and contains a glossary of terms to describe the functions. At this time, no company has stepped forward with a supported proposal suitable for standardization. ICL has provided documentation on its implementation of MacroLAN which will be distributed in the next mailing of X3T9.3. It should be recognized that ICL has not committed to active participation in editing or the technical development of the Fiber Channel. Tony Salthouse provided the technical material at a working group earlier this year and with over 20,000 installed nodes it has been a highly successful product. An appeal has been broadcast for proposals. In the event that nothing suitable is volunteered, the ICL documents plus the SCSI, IPI and HSC documentation will be used to develop the standard. Yours sincerely, I. Dal Allan Fiber Channel Description 1. Scope The objective of the Fiber Channel is to provide a transport vehicle which is capable of replacing the SCSI, IPI and HSC Physical Interfaces with a protocol-efficient alternative that provides performance improvements in distance and/or speed. SCSI and IPI commands may be intermixed, without impact on one another, on the Fiber Channel. Proprietary and other command sets may also use and share the Fiber Channel, but such use is not defined. The application of the Fiber Channel is to replace the first two layers of the following figure, which is a broad generalization of characteristics: +--------------+ +--------------+ +--------------+ |SCSI Commands | |IPI-3 Commands| | VU Commands | +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ | MSG | | IPI-2 | | Packets | |Link Layer Ctl| | Packets | | Commands | +--------------+ +--------------+ +--------------+ +--------------+ +--------------+ | Half Duplex | | Half Duplex | | Half Duplex | +--------------+ | Physical I/F | | Physical I/F | | Physical I/F | | Simplex I/F | +--------------+ +--------------+ +--------------+ +--------------+ 1.1 General Included is an introduction and definition of terms. 1.2 General Description Previous generations of channels have gained performance advantages in the transfer of data from a concept variously described as Data Streaming (IPI) or Synchronous Transfers (SCSI). This advantage has not been provided by protocols, which have relied on Interlocked (IPI) or Asynchronous (SCSI) controls. The Fiber Channel introduces Control Streaming, the ability to stream the protocol to improve performance. In addition to reducing the overhead associated with processing each step in a protocol from opposite ends of the cable, cable transmission delays are reduced to a minimum. Flow control over data transfers such as Throttling (IPI) is available as well as the equivalent to the Prolong Data Streaming (Enhanced IPI) and Ready (HSC) signals. The longer the distance between any two units attached to a Fiber Channel, the more important these features become to maximize effective throughput and utilization of bandwidth. The Fiber Channel is optimized for predictable transfers of large blocks of data such as used for file transfers between processors (super, mainframe, super-mini etc), storage systems (disk- and tape), communications, and to output only devices such as laser printers and raster scan graphics terminals. The physical components (cables, conductors, components) and the control protocol for the transmission of digital data between pieces of equipment are defined by the proposed standard. The Fiber Channel protocol is simple in order to minimize implementation cost and to enhance throughput. The transmission medium is isolated from the control protocol so that implementation of point to point links, multidrop bus, rings, crosspoint switches, or other special requirements may be made in a technology best suited to the environment of use. Although described as the Fiber Channel there may be three different kinds of implementation depending on the technology of implementation. +--------------+--------------+--------------+--------------+------------+ | Serialized | Optical to | Serialized | Serial to | Parallel | | Fiber | Electrical | Copper | Parallel | Copper | | Channel | Conversion | Channel | Conversion | Channel | +--------------+--------------+--------------+--------------+------------+ Serialized does not mean that transmission occurs on only one conductor. Both control and data signals are multiplexed into one or more conductors. Information transfers occur at the high repetition rates associated with communications technology. A parallel copper channel should represent a relatively small step for the existing SCSI and IPI, as the installed base will be able to more easily identify with the new physical interface. The only advantage to a parallel version of IPI would be the Control Streaming service, and this is unlikely to provide enough technical advantage to justify its implementation. There are several features desired in SCSI-3 which will affect the existing SCSI physical interface, and if new silicon is needed then it would be desirable that it be a decomposition of the serialized Fiber Channel. Systems integrators will be able to take advantage of the a common channel architecture over several physical media. Fiber is well-suited for use over long distances and copper is an inexpensive medium over short distances and cabling internal to the cabinet. Given the ability of technology in serialized transfers, it is estimated that the signal count for a parallel implementation will be about 25:1 over fiber i.e. a 2-fiber Fiber Channel would be the equivalent of a 50-pin parallel implementation. 1.3 Performance Multiple physical implementations may be made of the Fiber Channel, and the technology of each will dictate the performance achievable per conductor, and more than one conductor can be ganged together to build a composite transfer rate many times higher than the speed of a minimum attachment. The Fiber Channel uses level definitions similar to those of IPI. - FC-0 defines the cabling medium, connectors and transmission criteria - FC-1 defines the signaling protocol The Fiber Channel signaling protocol is consistent across a range of different FC-0 implementation alternatives. Transmission criteria such as framing and the data reduction scheme are not part of the signaling protocol. There are at least three types of data reduction choices which shall be FC-0 dependent, and not affect the FC-1 protocol. - encoding - scrambling - naked transmission with in-line error detection and correction If a Fiber Channel was implemented with an FDDI-compatible FC-0 then the nominal repetition rate of the transmitters/receivers would be 125 MHz. FDDI uses a 4B/5B code which reduces this to a data transfer rate of 100 Mbs. Assuming all-FDDI transmission characteristics, the transfer rate of a Fiber Channel with three additional Data Bus Groups would be 50 MBs over a distance of up to 2 Km. The performance goal of the Fiber Channel is to provide at least equivalent, and preferably superior, transfer rates to the existing Physical Interface definitions of SCSI, IPI and HSC (which covers a range from 1.5 MBs to 100 MBs with one cable). A minimum Fiber Channel Physical Interface consists of control signaling and one Data Bus Group. Additional Data Bus Groups may be added to increase the transfer rate. If the additional Data Bus Groups are physically contained in separate cables, they may have associated minimum signaling to control transfer timing. The media choices for typical Fiber Channel configurations are: - a copper plant for distances up to 300M. - a multimode fiber plant for distances up to 2 KM. - a single mode fiber plant for distances up to 50 KM. A single installation may contain more than one cabling plant but the various transmission media cannot be mixed and matched without conversion through an adapter. The signaling and control sequences are low in overhead. On large multiplexed file transfers the effective transfer rate shall approach the nominal maximum. A unit which initiates an action on the Fiber Channel is solely responsible to terminate the action by removing the control signaling associated with it i.e. there is closure of the control signaling between source and destination. A single cable may be used for point-point and ring configurations. This is more susceptible to a single point of failure than having two primary buses. The latter lends itself to star configurations as well as rings. A Fiber Channel could be cables using a single Primary Bus e.g. +------+ +------+ +------+ +=> | | (m)================> | | (n)================> | |==+ +===|======|======================|======|======================|======|==+ +------+ m>0 +------+ n>0 +------+ Figure 1-1 Single Primary Bus Configuration A Fiber Channel can be cabled with two Primary Buses between each unit e.g. +------+ +------+ +------+ | | (m)================> | | (n)================> | | | | <================(n) | | <================(m) | | +------+ m>0 n>0 +------+ m>0 n>0 +------+ Figure 1-2 Dual Single Primary Bus Configuration A Fiber Channel can have additional Data Bus Groups, which may be asymmetric e.g. if there is a higher transfer rate required for input to an attachment, then there may be more Data Bus Groups on Input than on Output. By adding seven Data Bus Groups to Input, an attachment could have a bandwidth asymmetry of 8:1. This ability could prove useful to output-only devices such as laser printers or high resolution monitors. Reconfiguration of this capability on a dynamic basis is outside the scope of this proposal. 2. Definition of Terms Burst A burst is the lowest indivisible element of an Information Transfer which may be sent between a source to a destination. Signal conditions established during transmission of a burst are used to identify whether or not an Information Transfer is to be terminated at the end of the burst or followed without interruption by another burst. The length of a burst is defined by the Burst Prefix. Burst Prefix The control information preceding an Information Transfer burst. +-----+-------------------------+ +-----+--------------+ | 0 | Bit 7 - IPI-3 | | 1-3 | Burst Size | | | Bit 6 - IPI-2 | +-----+--------------+ | | Bit 5 - SCSI | | 4 | Source | | | Bit 4 - Vendor Unique | | 5 | Destination | | | Bit 3-1 - Reserved | +-----+--------------+ | | Bit 0 - Link Member | | 6-7 | Burst Number | +-----+-------------------------+ +-----+--------------+ Burst Suffix A single word of information sent from a source to a destination at the end of each burst. If the contents of the word are all zero there is no error checking on the integrity of the transfer. If the contents are nonzero, then the word contains the integrity check data for a polynomial previously agreed upon by the source and destination. NOTE: Neither the Burst Transfer Prefix nor the Burst Transfer Suffix are included in the packet length. Control Streaming Actions initiated by a source are concatenated together on the assumption that the destination will accept the entire sequence e.g. a SCSI or IPI sequence to write data requires an acknowledgment from the destination between selection, command transfer and data transfer. With Control Streaming the source concatenates them so that there is no cable delay between each. If the source accepts the sequence it responds with concatenated sequences that the source uses to confirm that the actions were accepted. Conductor A single entity capable of carrying control and/or data signals. A conductor may consist of more than one element e.g. if a Fiber Channel conductor was implemented using differential logic then there would be two physical pieces of twisted pair wire used to carry the signals. Data Bus Group A set of conductors which carry data. The number of conductors in a group is a function of the technology used. More than one Data Bus Group may be added to the Primary Bus to increase the bandwidth of the Fiber Channel. Destination This term describes the equipment to which information flows. A destination may be a master or slave (IPI), initiator or target (SCSI). NOTE: It is possible for information to be transferred from one Source to more than one Destination by using the Link Member in the Burst Prefix. Duplex A configuration consisting of two Primary Buses, each transferring in a different direction. Typically, but not necessarily, the number of Data Bus Groups in each direction is the same. Information Transfer Information transferred which may be data, commands or responses. Connection A sequence of signals between a source and destination prior to performing an Information Transfer. Disconnection A sequence of signals between a destination and source following an Information Transfer. A logical connection is required to perform further Information Transfers. Multiplex The ability to intermix data bursts for each packet on the interface with control sequences between each burst. Data bursts for different packets may be intermixed and are identified as such by the source in the Burst Prefix. Packet A transfer sent during one logical connection which may be composed of one or more bursts. No maximum size is specified by the channel but a maximum may be required by a higher level protocol. Primary Bus The minimum number of conductors necessary to carry the signaling protocol and one Data Bus Group. In a serialized implementation, this could be a single conductor. A Fiber Channel requires two Primary Buses, one for Input and the other for Output. Source This term describes the equipment from which information flows. A source may be a master or slave (IPI), initiator or target (SCSI). Unit A unit defines any equipment which is attached to the Fiber Channel and is capable of executing the signaling sequences of the Fiber Channel. 4. Physical Specifications 4.1 Electrical Parallel to serial conversion can be done with chips available on the market, such as the AMD TAXI chip. There are other competitive components. Input to the TAXI chip is byte wide and the output is an encoded bit serial data stream suitable for export over coaxial or fiber optic cables. Each bit serial stream is rated at 12.5 MBs. Suitable transmitters/receivers are needed to support the physical medium chosen. 4.2 Cable and Connector Specifications The cables and connectors are TBD. It is desirable that a compact form factor be used to suit the smaller cabinetry of today's computers. Recommendations on these is solicited from connector and cable manufacturers. Parallel fiber is available in a cable size that is significantly smaller than coaxial but the connectors are quite large. 5. Functional Criteria The following is a list of characteristics and criteria which represent the domain of operation of the Fiber Channel. A rough but not completely inaccurate grouping was made into major categories. This list was developed at a working group focused on Functional Requirments. 5.1 Application - Resources dedicated to a single operating system - Closed system i.e. defined set of resources - No provision for inter-networking - Known topology - Path implicit from address - Master-Slave relationship may be dynamic - Designed for the control of peripherals - IPI and SCSI devices can coexist on the same channel - Multiple processors can coexist on the same channel - Coordinated access to shared peripherals - Compatible with existing SCSI and IPI-3 command sets 5.2 Cost/Distance - Able to use multiple technology choices - Achieve approximate cost parity with a copper connection in an equivalent environment - Medium designed to accommodate future product generations - High performance and long distance possible at a cost premium that scales with the parameters - Cost effective at 50 meters, and designed to promote operation over longer distances 5.3 Environment - Office compatible in terms of "weather" and shock and vibration - User education requirements equivalent to FDDI, and greater than for existing copper interfaces 5.4 Fault Tolerance - Feasible to create configurations with no single point of failure - Support multiple pathing - Continue operations with attachments powered off - Support on-line servicing - Permit operation-critical active elements if have acceptable MTBF - Support system detection and isolation of all failures (including tolerated ones) 5.5 Speed - Repetition rate dependent on technology 5.6 Performance - Performance expressed as burst data rate in real bytes - Granularity of 5, 10, 50 and 200 Megabytes/second - 80% protocol efficiency @ 4K burst and maximum number of units attached - Hardware error detection on tbd boundaries - Flow control supported 5.7 Distance - Distance can vary with different physical medium: Optical - 5M to 2KM with a median of 500M Copper - 0 to 10M 5.8 Error Rates - Random bit error rates on medium are a product of the technology - Error rate of catastrophic events e.g. control protocol failure are 10E12 on optical medium - Aim for copper to achieve same rates (have EMI concerns) 5.9 Protocol etc - Low latency control protocol - Minimize station delays - Provide for power sequencing - Limited number of connections to 32 - Channel length (of all connections) measured at 20 usec (4KM) round trip for performance judgments - A connection is the sum of all components from PCB trace to PCB trace - Cable costs are additional to the estimated cost of a connection - A connection will offer at least the same MTBF as SCSI or IPI today - A limit will be set for the number of passive connector pairs between active units - Connector footprint <= equivalent SCSI or IPI today - Fiber Channel can achieve higher transfer rates using parallel conductors - Fit into confined spaces with tight cable bend (not worse than Fast SCSI, Enhanced IPI, or HSC cable)