March 10, 1989 TO: X3T9.3 Fiber Optic Study Group Members FROM: Roger Cummings SUBJECT: FIBER OPTIC CHANNEL WORKING GROUP MINUTES Please find attached a draft of the minutes of the ANSI X3T9.3 Fiber Optic Channel Working Group that was hosted by Dal Allan of ENDL Consulting at the Sunnyvale Hilton, Sunnyvale CA on January 30 and 31, 1989. Note that there are also eight Attachments to the minutes that relate to presentations at the meeting. Please find attached a schedule for the Fiber Channel and HSC Working Meeting for 1989. Apologies for the delay in producing these minutes. My intention is not to mail them at this late date but merely to have copies available at the March working meeting at Pixar in San Rafael, CA. Note that a meeting notice has been mailed separately by Wayne Sanderson of Control Data in Minneapolis. In fact Wayne has organized the mailing of the last several minutes, and therefore deserves the thanks of the Working Group. If there are any corrections required to, or omissions noted from, the minutes I can be reached as follows: Phone: Business (416) 826-8640 x3332 Home (416) 625-4074 (ans machine) Telex/MCI Mail: 650-289-5060 (USA) Fax: (416) 821-6363 Regards ______________________ Roger Cummings Principal Engineer, I/O and Peripherals Systems and Strategies Group Control Data Canada Ltd. 1855 Minnesota Court Mississauga, Ontario L5N 1K7 Canada #ya/rc MINUTES OF THE FIFTH FIBER OPTIC WORKING GROUP MEETING The Fifth meeting of the ANSI X3T9.3 Fiber Optic Working Group was hosted by Dal Allan of ENDL Consulting at the Sunnyvale Hilton, Sunnyvale CA on January 30 and 31, 1989. A total of 38 people attended, as follows: AMD Wayne Wong AMDAHL Masanori Motegi AMP Charles Brill ANCOR COMMUNICATION Terry Anderson CANSTAR Stewart Lee Kumar Malavalli Stan Smyth Warren Taylor CDC Wayne Sanderson CDC CANADA Roger Cummings CIPRICO Bill Winterstein Consultant Stewart Sando CRAY RESEARCH INC. Wayne Roiger DATA GENERAL John Allen DONAGAN INTERNATIONAL Don Pederson ENDL I Dal Allan FUJITSU AMERICA Bob Driscal GAZELLE MICROCIRCUITS David Macmillan Chris Popat GRUMMAN Tony Iacona GRUMMAN Napoleon Avaneas HONEYWELL SSPL Jerry Quam IBM Henry Brandt John Grzinch Ken Meifert Ron Soderstrom Horst L Truestedt INTEGRATED PHOTONICS Wally St. John LAWRENCE LIVERMORE NATIONAL LABS. Paul Rupert LOS ALAMOS NATIONAL LAB Don Tolmie NETWORK SYSTEMS Ken Drewlo SUPERCOMPUTER SYSTEMS INC. Leonard Veil TANDEM COMPUTERS Duc Pham Phil Sinykin Jim Smith TEKTRONIX Steve Blazo UNISYS Bob Bergey Peter Dougherty January 30&31 Fiber Optic Working Group Minutes Page 2 The meeting agenda had been set by Dal Allan in his position as the Chairman of the Working Group. A copy of Dal's agenda is Attachment 1. After the introductions, the first presentation was made by Canstar. Warren Taylor began by giving a short corporate profile of Canstar, and he then handed over to Stewart Lee of the University of Toronto to describe the Hubnet fiber optic network system. A copy of the Canstar slides is Attachment 2. Stewart noted that the Canstar product name is Supernetwork, and described the scheme of expandable hubs and the protocol used to access them. He then moved on to consider a complex topology that consisted of a central hub and a number of interconnected sub-hubs. Horst Truestedt of IBM asked how a host connected to a sub-hub with a long retry time ever gained access to the network, and was told that simulations had demonstrated that times with in an order of magnitude had little effect of the success rate. Don Tolmie of Los Alamos National Labs asked if the selection of the successful accessor was truly random and Stewart replied that it was, and that if a race condition occurs one participant is picked. Stewart went on to define the hub in greater detail as being composed of a selection module and a broadcast module connected by a "golden nexus". He stated that the implementation of the hub is simple and requires only 44 TTL chips. Dal Allan asked how the retry time was defined for each hub port, and was told that it was done in hardware by switches or other means. Bob Bergey of Unisys asked if there was a limit on the number of access ports in a system, and Stewart replied that there was no limit for all practical purposes. It was also noted that the network uses a 4B/5B coding scheme. Stewart then moved on to describe an experiment in which the performance of Hubnet was evaluated. It consisted of measuring both the time and number of retries taken to route a packet through a simple network in the presence of various types of random loadings. For the configuration shown the path time was approximately 56 microseconds, and each test consisted of sending 2K packets and using the last 1K as the sample. The predicted and actual results were then compared in detail. A scheme for avoiding long delays in Hubnet by designating alternate packets "red" and "blue" was also described. The principle is that the central hub will only accept packets of a specified colour during a contention period. The colour of the contention period is established by the colour of the first packet accepted, and this is done by monitoring the broadcast side of the node. The scheme gives a deterministic upper bound to the delay time encountered in transmitting a packet. January 30&31 Fiber Optic Working Group Minutes Page 3 The most difficult part of the above scheme was said to be the detection of the end of the contention period. Ron Soderstrom of IBM Rochester asked if the length of the contention period was dependent on knowledge about the number of ports of a specified colour. Stewart replied that the detection was handled by the state machines in the selection module alone. The expiry of a period without receipt of a packet causes the port to lose its colour, and if all nodes enter this state the node becomes "gray" such that the first packet subsequently received will define the colour of the new period. Error sources in Hubnet were then briefly considered with particular reference to a Canstar 50 Megabit network that has been operational for 5 or 6 years. Stewart described the 5 or 6 faults that had occurred in that period as being related to blown power supplies and difficulties with programmable logic. Stewart then moved on to the consideration of different sorts of redundant configurations. Wayne Sanderson of CDC asked if multiple "golden links" were possible, and Stewart replied that work was going on in this area, and that it would require a smarter hub and handling of the contention at the receiving port. In response Wayne commented on the tradeoff between a simple hub versus queuing packets in the host etc. Stewart agreed that the simplicity of Hubnet was a major advantage, and described a campus site where Hubnet acted as the major backbone of the network and yet was almost unknown to users because of its high reliability. Bill Winterstein of Ciprico asked about the efficiency or throughput of Hubnet, and was told that it was close to optimal for a single server philosophy and that the inter-packet gap is the only waste. Napoleon Avaneas of Grumman asked how clocks were synchronized in Hubnet, and Stewart replied that clock was carried on all the broadcast links. Ron Soderstrom asked how much of the hub was fiber-optic, and was told that the hub is mainly electrical with fiber-optic transceivers at each port. Warren Taylor then concluded the Canstar presentation by detailing the current Super 100 product which uses FDDI compatible components and supports the Ethernet Device Interface as one option. Roger Cummings asked if the same protocol is used as on Ethernet, and Warren replied that the use of Super 100 was completely transparent except that no collisions ever ocurred. Warren also briefly reviewed Canstar's future product plans including trusted networks. Dal Allan asked if there were any issues with regards to patents, and Warren replied that some of the protocols used, and the red/blue scheme, were covered by patents but that no decision had been reached with regards to their disposition. January 30&31 Fiber Optic Working Group Minutes Page 4 The next presentation was made by Napoleon Avaneas of Grumman Corporation on the subject of the Grumman High Speed LAN (G-LAN 800). A copy of Napoleon's slides is Attachment 3. He described G-LAN as using a hybrid topology and noted that in the slide showing both main and secondary subrings the signal splitters used were 50%-50% types. The active delay was stated to be adjusted so that the secondary path lagged the main path by eight bit times and later a topology that incorporated private connections was also described. Napoleon then described the data clock synchronization scheme and then moved on to consider the protocol implications and to detail the message types and formats. The voice/video user message type identified in the message header was stated to be ISDN traffic. The characteristics of the User Agent (the upper three OSI model layers) were then described, and the presentation concluded with a description of the G-LAN Node physical implementation and a development schedule for G-LAN which included a 1.6 GBps LAN due to become operational in late 1991. Napoleon was asked how the 1.6 GBps would be achieved, and he replied that he believed that wavelength division multiplexing would be used. He was also asked which interfaces were available for G-LAN today and replied that a DR-11 type is available with an Ethernet Device Interface in progress and a very programmable dma controller under development to act as the core of future interfaces. Napoleon had made a point throughout the presentation of emphasizing the amount of "protocol" decoding that was done on the fly in hardware. Therefore he was asked which tasks were performed by the 68000 in the node, and replied that it generated software messages with the next layer being implemented by programming a fast transfer memory. Don Tolmie asked if the G-LAN project was being sponsored by a committee, and Napoleon replied that it had its roots in an avionic distributed processing system and was being created independently. Don further enquired about issues related to patents, and was told that the clock synchronization scheme, and the CRC generator that operates at up to 5 Gb/s, were covered by patents. Napoleon stated however that Grumman were considering the patent position, and wished to be part of the standards activity. He noted that in FDDI the retransmission delay is 2.5 microseconds while in G-LAN it is 52 nanoseconds largely due to the clock scheme. In response to a question he also noted that there is no requirement in G-LAN for interoperability at the TCP/IP level. January 30&31 Fiber Optic Working Group Minutes Page 5 The third presentation was made by Ken Meifert of IBM Poughkeepsie after an introduction by Henry Brandt of IBM Kingston. A copy of the IBM presentation is Attachment 4. Henry stated that the IBM requirements were for a high speed, long distance interface and that the presentation would concentrate on the protocol requirements rather than the components. Ken then began with a set of objectives and physical characteristics, and noted in response to a question that the proposed connector was an FDDI type with tighter tolerance. The proposed code is an 8B/10B type, and it was noted that this code is patented. Copies of the relevant patents were provided, and are included in Attachment 4. John Grzinch of IBM also noted that cross-licensing could be a problem. The running disparity check built in to the code was described as catching single bit errors within a byte with the CRC providing a second level of detection for error bursts of up to 31 bits. However Napoleon Avaneas cautioned that performing the CRC check in hardware at the speeds of the Fiber Channel may be a problem. Control and Data frame structures were then described in detail, with the note that all fields were defined on four byte boundaries. Ken stated that the number of Idles between packets needed definition but that one would be the minimum. He was asked where the CRC coverage began, and replied that it began immediately after the start of frame field. He thus agreed that the entire packet would have to be buffered before it could be known to be correct. Ken then went on to describe the Transmit and Receive Data Flow, and provided examples of how the HSC Signal Sequences could be mapped into a Fiber Channel using the packet structures defined. Ken was asked why an 8B/10B code had been chosen in preference to a 4B/5B scheme similar to that used in FDDI. He replied that 8B/10B provided a much larger symbol space, and that 4B/4B codes created problems with the heavy usage of the J/K symbols. January 30&31 Fiber Optic Working Group Minutes Page 6 The next presentation was made by John Allen of Data General on experience gained with their Fiber Inter-Chassis Bus (ICB). A copy of John's slides is Attachment 5. He described the ICB as a proprietary interconnect for DG systems which used FDDI class technology and a CRC-16 implemented in gate arrays. John also passed around a card containing an implementation of one end of the ICB. He noted that the clock required is at the controller rate of 46-100 nanoseconds. The message format was also described, and the start and end fields were stated to be identical to FDDI. The Rcv Seq field identifies the last message received to implement the sliding window protocol, and the cmd field is dedicated to handling Acks. The maximum link length is presently defined as 2 KM, but the protocol could support a 20 KM link. John noted that a copper interconnect could be used for shorter distances, and estimated that the maximum length for copper would be in the order of 100 feet at 200 MBaud. A particularly useful part of John's presentation was a detailed costing of the Fiber ICB components. He stated that the gate array is an AMD ECL type, and that the static rams are also ECL devices. He noted that other implementations with a different split of functions between CMOS and ECL are possible. Roger Cummings asked where the Fiber ICB was used in the DG systems, and John replied that it connects a new i/o subsystem (MRC) to the MV series computers. It was noted that a mimimum length of fiber may have to be defined for this type of interface to avoid problems due to reflection, modal scattering etc. January 30&31 Fiber Optic Working Group Minutes Page 7 Tuesday morning began with a presentation by Terry Anderson of Ancor Communications. A copy of Terry's slides and an Ancor brochure is Attachment 6. He began by tracing the evolution of the communications requirements from the tape turnkey systems of ten years ago to the present requirements for networks with thousands of nodes. Terry sees the cost of connection as a very significant issue given the wide disparity of types of equipment that have be interconnected. He then described Ancor's approach to a universal interface, which uses a dual buffering scheme and can be configured into a redundant ring or a circuit switch architecture. The interface is based on a GaAs LSI gate array named VHSCC, and Terry displayed architectures of a VME and an HSC interface based upon the VHSCC. The array itself is designed with a control interface suitable for connection to a number of RISC processors and incorporates a digital phase lock loop that operates at 1 Gigabit and has a lock time of sixteen bit transitions. It has differential interfaces to the fiber-optic transmitters and receivers and is contained in a 176 pin grid array. A 40 bit wide word is used and Terry noted that the need for good dc balance in the encoding scheme is not confirmed. A burst transmission format similar to the parallel format of HSC is used. The VME interface is being designed as part of a military contract, and has 2x2 Megabyte receive buffers and 2x4 Kilobyte transmit buffers. The HSC interface Terry estimated will occupy a card 4 inches x 14 inches and cost $1K. Terry then moved on to describe the CXT circuit switch system. He noted that a circuit switch allows parallel communications paths while a LAN is serial. The CXT module contains a space switch built of identical pcbs and can handle 4096 non-blocking channels for 8.2 Terabits of system bandwidth. Kumar Malavalli of Canstar asked when the VHSCC array will be commercially available, and Terry replied that he expects a full speed part to be available this year and that the target cost is $253. Wayne Sansderson asked what the $1K for the HSC interface included, and was told that it included the full card including the 1 Gigabit/s optics (Terry expects a cost of 20c/Mbs at year end). Steve Blazo of Tektronix asked if the switch in the CXT module was electrical - which was confirmed. Terry noted that Ancor had found fiber to be extremely rugged in the field providing that it is installed correctly - he had even known it to survive a very serious fire. He also noted that in his experience the alignment problems on short links were not as serious as many feared. He stated that Ancor had never used any single mode technology. January 30&31 Fiber Optic Working Group Minutes Page 8 Wayne Sanderson asked if mixed topologies were possible using the CXT, and was told that they were. Terry also confirmed that there is no buffering in the CXT and stated that in one application FM radio had been routed through the switch. Don Tolmie asked for a cost comparison of the circuit switch approach versus a LAN. Terry replied that in the CXT the cost of a full duplex channel is less than 1K and that a 1 Gb channel is expected to be in the range of $2K-3K. Dal Allan noted that the circuit switch is a way to get multiplexing for "free". Terry concluded with the remark that he saw the trend towards visualization i.e. animation rather than static graphics, as becoming the driving factor towards high performance networks and increasing memory and disc bandwidths. The final presentation was made by Chis Popat of Gazelle Microcircuits on their Hot Rod high-speed serial link that is implemented in GaAs. A data sheet of the Hot Rod is Attachment 7. Chris noted that samples are due to be available in 3Q89 and are projected to cost $300 at introduction and $75 eighteen months later. Roger Cummings queried the ECL 100K compatibility, and Chris replied that it was compatible when the ECL was operated between +5 volts and ground. Dal Allan enquired about future plans and was told that the next generation will double the data rate and include buffer management. Dal Allan then concluded the meeting by contrasting the features of the various systems that had been presented. Dal also solicited the following presentations on the subjects as they related to the Fiber Channel: 1) Paul Rupert of Lawrence Livermore Labs. on the subject of security. 2) Terry Anderson of Ancor on the subject of circuit switching. 3) Kumar Malavalli of Canstar on the subject of active stars.