Minutes from 9/94 FC-AL Ad Hoc Meeting

Kurt Chan kc at core.rose.hp.com
Thu Sep 29 12:31:23 PDT 1994

From:  Kurt Chan, HP (kc at core.rose.hp.com)
To:    FC, SCSI Reflectors
Subj:  Minutes of 9/7/94 FC-AL Ad Hoc Meeting
Date:  9/29/94

           FC-AL Direct Attach Disk Ad Hoc Meeting Minutes
                      Red Lion Stapleton
                         Denver, CO


  Dal Allan, ENDL                  250-1752 at mcimail.com
  Scott Bruns, NCR                 scott.bruns at ftcollinsco.ncr.com
  Kurt Chan, HP                    kc at core.rose.hp.com
  Jim Coomes, Seagate              jim_coomes at notes.seagate.com
  Steve Dean, HP                   dean at core.rose.hp.com
  Brian Doore, Seagate             brian_doore at notes.seagate.com
  David Ford, Cambex               dford at cambex.com
  Giles Frazier, IBM               gfrazier at ausvm6.vnet.ibm.com
  Norm Harris, Adaptec             nharris at eng.adaptec.com
  Tim Johnson, Cray Research       tjohnson at cray.com
  Jean Kodama, QLogic              j_kodama at qlc.com
  Glen Koziuk, NCR                 glen.koziuk at ftcollinsco.ncr.com
  Scott Mindemann, Interphase      scottm at iphase.com
  Yogi Schaffner, Silicon Systems  yogi at tus.ssi1.com
  Greg Scherer, Emulex             g_scherer at emulex.com
  Ken Smith, Conner                ken.smith at conner.com
  Jeff Stai, Western Digital       stai at dt.wdc.com
  Horst Truestedt, IBM             truested at vnet.ibm.com
  Fanny Wong, IBM                  fwong at vnet.ibm.com

- Private loop devices must observe normal FC-AL rules that require a
  device which has a "hard" address to defer to a Fabric assigned

- Private loop addresses must have zeroes in the upper 16 bits.

- Public loop devices may not have all zeroes in the upper 16 bits.

- FL_Ports are prohibited from forwarding frames into a loop if the
  frames contain all zeroes in the upper 16 bits.  This is considered
  frame misrouting.  Neither the fabric nor any devices on the other
  side of the fabric should be attempting to communicate with Private
  Loop devices.

The standard contains the following Login rules: 

   - Private loop devices are only allowed to log in with other
     private loop devices, and may not attempt login with public loop
     devices or with FL_Port.

   - Public loop devices may log in with all three - public private,
     and FL ports and must attempt FL_Port login as part of their
     initialization procedure (per FC-PH).

Profiles may further restrict these login rules.  For example, the
following table defines the Logins allowed by the Private Loop Disk
Profile when Private, Public, and FL ports coexist on the same loop.

                                   LOGIN RECIPIENT
                    * ---------------------------------------------
                    |  Private   Public  Private   Public    
  LOGIN INITIATOR   | Initiator  Target  Target   Initiator  Fabric
------------------- | ---------  ------  -------  ---------  ------
Private SCSI Init   |    Yes       No      Yes*      No        No 
Public SCSI Target  |    Yes       Yes     No**      No**      Yes*
Private SCSI Target |    No        No      No        No        No 
Public SCSI Init    |    Yes       Yes**   Yes       Yes       Yes*

  No    = prohibited by standard and/or profile
  Yes   = allowed by standard, but not yet required or prohibited
          by any loop profile
  Yes*  = required, either by a standard or by Private Loop profile
  No**  = anticipated to be prohibited by Public Loop Profile  
  Yes** = anticipated to be required by Public Loop Profile  

If a Public Initiator only supports class 2, it cannot successfully
log into a Private Target, which only supports class 3. 

For this profile, PLOGI shall only be attempted by SCSI Initiators to
devices with D_ID = 0000+AL_PA hex in the upper 16 bits.  (Public loop
devices performing discovery functions may need to attempt two PLOGIs to
each AL_PA, one to 0000+AL_PA and one to xxxx+AL_PA, where xxxx is the
upper 16 bits of the attached FL_Port address).

Login reception was discussed when login resource limits are reached.
Rather than reject new PLOGIs, it was recommended that all PLOGIs be
accepted, and the "oldest" PLOGI be implicitly logged out when resource
limits are reached.  [Ed note: discuss at next meeting whether this
behavior needs to be specified for interoperability, and if so whether
explicit LOGO to "oldest PLOGI" be attempted before ACC to newest PLOGI is
transmitted.  Also discuss requirement to transmit LOGO in response to
frames received from a device which is not logged in.]

1) For the upper 16 bits of an NL_Port native address identifier, FC-AL 
   now requires that:
    -  all private loop devices or public loop devices which do not
       share the loop with an FL_Port use 0000h.
    -  all public loop devices which share the loop with an FL_Port
       use the (nonzero) value assigned to them by the FL_Port.

2) For devices which are configured with a hard address, if the device
   does not acquire it's hard address, it shall accept a soft address
   and accept PRLI with the response code '0101' in word 1, bits
   [11:8] ("a predefined condition precludes establishing this image
   pair - PRLI shall not be retried") and implicitly log out all

3) For devices which are NOT configured with a hard address, if the
   device does not receive it's previously acquired address, it shall
   accept a new soft address and implicitly log out all devices.

4) For ALL devices, if neither a hard nor soft address is acquired,
   the device shall go to non-participating mode and implicitly log
   out all devices.

In most current implementations, an NL_port will not respond per FC-AL when
it receives a NOS, OLS, LR, or LRR while in the OPENED state.  This is
because these Primitive Sequences were intended to initialize point-point
N_Port-F_Port links, not link segments on a shared medium where the
output and input of any given port do not go to the same N_Port. Requiring
NOS/OLS to work across a Loop Circuit would be analagous to requiring
class 1 connections to stay up while performing this protocol.

Since LIPs are used as link-level resets in this profile, private NL_Ports
are prohibited from issuing NOS, OLS, LR, or LRR unless in OLD-PORT state.
What NL_Ports do if they receive any of these Primitive Sequences while in
the OPENED state is vendor-specific.  If the port is in MONITORING state,
it should propagate the Primitive Sequence.

There are three basic types of Loop Initialization Primitives.  Here's my
understanding of how these LIPs are now being used, and their effect on
private loop NL_Ports (some of this is borrowed from Version 0.40).  We
will review these on Oct 4.

a) L_Bit

   Private loop devices are prohibited from setting the L_bit.  It is
   expected that the L_bit will be used only by FL_Ports.  [ed note - do we
   anticipate the use of NFL_Ports in the private loop profile?  If so,
   should NFL_Ports be allowed to set the L_bit?]

b) Power-on reset

   LIP(AL_PD,AL_PS) is a request from the transmitting NL_Port (at AL_PA =
   AL_PS) to the receiving NL_port (at AL_PA = AL_PD) to perform a power-on
   reset.  There is no broadcast LIP which causes a power-on reset.  Only
   SCSI Initiators may transmit this LIP. See the table on page 22 of the
   0.50 profile for what objects are reset by this function.

c) Loop initialization

   Both SCSI Initiators and SCSI Targets may transmit this LIP type.

   LIP(F7,F7) indicates the transmitting NL_Port is attempting to acquire
   an AL_PA due to either new insertion or power-on initialization, or has
   been unable to win arbitration within E_D_TOV.  Receiving devices shall
   go immediately to OPEN-INIT state, regardless of their current state.  On
   private loops, only the Sequence in progress may be disrupted, resulting
   in standard error recovery (ULP timeout) if any frames in transit are

   LIP(F7,AL_PS) indicates the transmitting NL_Port (with AL_PA = AL_PS) is
   attempting to reinitialize the Loop.  Receiving devices shall go
   immediately to the OPEN-INIT state.

d) Loop Failure

   Both SCSI Initiators and SCSI Targets may transmit this LIP type.
   LIP(F8,F7) indicates the transmitting NL_Port has detected a Loop
   failure at its receiver before it has acquired a valid AL_PA.  Receiving
   devices shall go immediately to OPEN-INIT state.

   LIP(F8,AL_PS) indicates the transmitting NL_Port (with AL_PA = AL_PS)
   has detected a Loop failure at its receiver.  Receiving devices shall go
   immediately to OPEN-INIT state.

The new Primitive Sequence name is Loop Port Enable (LPE) and Loop Port
Bypass (LPB).  My understanding of the desired behavior is that a bypassed 

a) upon receipt of LPB, shall go to MONITORING state and shall
   not change state until LPE is received.

b) loses it's AL_PA (goes to non-participating mode) if a LIP is received,
   but does not go to OPEN-INIT state and does not clear the P_BYPASS state
   variable [ed note: is this also true for LIP(AL_PD,AL_PS)?]

c) does not propagate OPNyy or OPNyx addressed to it (does propagate
   OPNfr and OPNyr, or OPNs to other ports).

d) does not attempt arbitration 

e) sets the P_BYPASS state variable which will activate any Loop Bypass
   Circuit, if present.

Furthermore, any port which does not pass self-test shall bypass
itself until the it is functional.  LPE or REQ(init) to a bypassed
port will reset the P_BYPASS variable.

For private loop devices, OLD-PORT state is optional (since FL_Port
communication is prohibited anyhow).  However, silicon designers
looking toward the future may want to design this function in for both
point-point fabric connect or FL_Port compatibility.

OPNyr (selective replicate) will be documented in FC-AL 4.34, but
generation of OPNyr will not be mandatory in the private loop profile.
HP will not support generation on first implementation since OPN is
sent as part of automated Sequence transmission and HP silicon lacks
the "phase control" needed to transmit consecutive OPNs.  [ed note:
do the OPNr rules also apply to OPNyr, or was selective replicate
intended to be a reliable service?]  

Note that OPNyr FC-AL broadcast is even LESS reliable from a link
layer perspective than class 3 since R_RDYs are not used and class 3
frame reception is subject to local buffer availability in the
receiving NL_Ports. Flow control is completely ULP-managed.

The following descriptions were transcribed from Jean Kodama notes due to
mental siezures on the part of the editor. The discussion was more related
to FC-AL than this profile.

             +--> B --> C ---+
             |               |           
           Port_A          Port_D
             |               |           
             +--- F <-- E <--+

There are three problems uncovered that ultimately manifest themselves
as some form of unfairness on the loop.  It was anticipated that

a) fairness is optional in FC-AL,
b) FL_Ports and SCSI Initiators will be allowed to be unfair in some profiles,
c) there are no FC-AL limits on how long a port can hold a circuit, which
   is another form of unfairness

that the following cornercases would not impact Loop functionality

Problem 1: bottom half of loop could starve

- Port_A receives ARB(F0) and begins transmitting IDLs
- Port_B sees IDL, transmits one IDL, and begins to arbitrate
- Port_D receives one IDL then a stream of ARBx's while transmitting a frame
- ARBx is the fill word when frame transmission is complete
- IDL is lost and Port_E and Port_F do not get a chance to arbitrate

This used to work because ARB(F0) changed to IDL only when it was guaranteed
that no frames were in flight. Possible fix is to go back to original scheme 
(don't change ARB(F0) to IDL until a CLS is received).

Problem 2: can lose fairness token (IDL) due to smoothing

- Port_A sends OEF and 2 ARB(F0)s 
- Port_A receives an ARB(F0) and begins sending IDLs
- Port_B passes through the EOF and 2 ARB(F0)s
- Port_B sees the IDLs, transmits one of them, and begins arbitration
- Port_C needs to perform a fill word deletion because of clock skew,
  but has to wait for EOF and two fill words to pass
- Port_C receives the EOF and 2 ARB(F0)s, then deletes the next IDL
- IDL is lost, and nobody else gets to arbitrate

Problem 3: opened node could starve

- Port_D is arbitrating and is the last device in the current arbitration window.
- Port_D is opened.  When it receives ARB(F0) it replaces its CFW (own ARBx)
  with ARB(F0) per FC-AL rules
- Port_A receives ARB(F0) and resets the access window

A possible fix is, if attempting to arbitrate, to test the priority of
an incoming ARBx before replacing the CFW in OPENED, TRANSMITTED CLS,
or RECEIVED CLS states. This may require a history variable. Also,
a port controller should keep REQ(arb) asserted until ARBITRATION WON
state is achieved to prevent "rogue" ARBx's.

10. Error Recovery
There were a couple of problems uncovered by a subsequent FCSI meeting
with respect to Target-Initiated ABTS that will be discussed at the
October meeting.  It is possible that ACA and UA definitions in SAM
are inadequate for networked disks.  The suggested wording changes I
posted to fix FCP rev 9 were not incorporated since there are other
problems related to Task Management which making Target-Initiated ABTS
optional does not solve.  Oct 4 we will discuss whether or not
creating a new IU (FCP_RSP to Task Mgmt functions) is a necessary but
insufficient solution to the problem.

The table on page 22 of the profile is split here into two tables, one
for FC-1/FC-2 actions, and the other for FCP actions. We will review
this in Oct. Note numbers are in (parentheses).

                                               FC-1/2 ACTION
                                LIP(y,x)   LR   LOGO,   FLOGI   ABTS   PRLI,
                                  (7)     (12)  PLOGI    (4)    (8)    PRLO
----- FC2 Objects -----------  --------- ------ ----- -------- ------ ------
PLOGI parmameters                 Y(8)     N      Y     N(2)     N      N
Open Seq's, per Initiator          Y      N(9)    Y    N(1,2)    Y      Y
Open Seq's, all Initiators         Y      N(9)    N    N(1,2)    N      N
EE_Credit_CNT                      Y      Y(10)   Y      N       N      N
BB_Credit_CNT                      Y       Y      Y      Y       N      N
AL_PA                             N(1)     na     N      Y       N      N
FLOGI parms                       na       N      N      Y       N      N
---- FC4, ULP Objects -------
PRLI parms                        Y(7)     N     Y(3)    N       N      Y
Single open Task, per Init         Y      N(13)  Y(3)    N       Y      Y
All open Tasks, per Init           Y       N     Y(3)    N       N      Y
All open Tasks, all Inits         Y(5)     N      N      N       N      N
Mode page parameters              Y(5)     N     Y(3)    N       N     Y(3)

                                       FCP ACTION
                                          Clear   Abort
                                 Target   Task    Task
                                 Reset     Set    Set
----- FC2 Objects -----------  --------- ------- ------- 
PLOGI parmameters                  N       N      N  
Open Seq's, per Initiator        N(14)    N(14)  N(14)
Open Seq's, all Initiators       N(14)    N(14)  N(14)
EE_Credit_CNT                      N       N      N    
BB_Credit_CNT                      N       N      N   
AL_PA                              N       N      N
FLOGI parms                        N       N      N 
---- FC4, ULP Objects -------
PRLI parms                         N       N      N    
Single open Task, per Init         Y       Y      Y    
All open Tasks, per Init           Y       Y      Y   
All open Tasks, all Inits         Y(5)    Y(5)    N   
Mode page parameters              Y(5)     N      N  


1.  Unless AL_PA's have changed (i.e., preferred address has already
    been taken by another port)

2.  Unless Worldwide Port Name of FL_Port has changed, or unless
    Worldwide Port Name of N_Port has changed in a point-point

3.  For Process or N_Port Login Initiator only, not for all

4.  Public loop device must discard all frames until FLOGI to AL_PA=00
    has been completed.

5.  Parameters cleared for all Initiators.  Unit Attention conditions
    shall be created per SAM requirements.

6.  Unless this object is retained across power-cycles.

7.  Also known as LIP(AL_PS,AL_PD) or FC-AL Selective Reset.  The
    selected Loop device performs power-on reset upon receipt of this
    primitive.  FL_Ports and Public Loop Devices are prohibited from
    issuing Selective Reset.

8.  ABTS with L_S=1 in the ACC is how the SAM Abort Task function is
    performed in FCP.

9.  In Class 1 all active/open Sequences are terminated.  In class 2,
    open Sequences may eventually be aborted when E_D_TOV timeouts
    occur as a result of missing frames/ACKs.  If credit happens to be
    balanced when the LR/LRR protocol is invoked, open Sequences may
    be unaffected.  See FC-PH 4.3,

10.  EE_Credit_CNT only affected for class 1.  Class 2 EE_Credit_CNT
     is unaffected.

11.  Disregard this row for Private Loop devices (which have no Fabric
     login parameters)

12.  This profile prohibits NOS, LR, ARB+OPN+NOS, and ARB+OPN+LR on a
     Loop.  NOS and LR only affect Loop devices when they are in OLD
     PORT state.  A port which has received NOS or OLS cannot become
     active without participating in the LR/LRR protocol (see FC-PH
     4.3, Annex Q).  Therefore, only the effects of LR are listed

13.  Open Tasks may be affected depending upon class of service and
     BB_Credit_CNT when LR is received - see note 10.

14.  Tasks are cleared internally within the Target, but open FC-PH
     Sequences must be individually aborted by the SCSI Initiator via

The next two ad hoc meetings will be 8:30am at the Stapleton Red Lion:

Tues Oct 4
Tues Nov 1

I'd also like to discuss at the next meeting whether or not we should
continue to meet in Denver after November to develop the Open Loop
Profile versus attempting to meet during ANSI week, and your views on
which environment would result in the greatest productivity.

More information about the T10 mailing list