FCP Out of Order Data Delivery

Bob Snively Bob.Snively at Eng.Sun.COM
Tue Dec 20 23:15:54 PST 1994

One of the reasons that we (Sun) are not personally worried about this function
is that our I/O Bus (IEEE 1496, SBus) uses virtual addresses in
its DMA function, making scatter-gather mostly unnecessary or automatic. 
For our Fibre Channel array subsystem, out of order delivery is routine, but
typically on large blocks and on disk block boundaries.

The other reason I am not worried about this function is that
the EMDP bit could be used unchanged to manage this special case.  Phrased
another way, it is not an FCP problem, but rather a choice to
implement the EMDP bit.

I believe that the equivalence between Modify Data Pointers and
the data descriptor is already well documented in section 7.2.1 on page 28
of FCP Revision 10.

Note that none of the Profiles address this question and would probably
resist the requirement of implementing this additional function.


----- Begin Included Message -----

>From jjw1 at PO9.RV.unisys.com Wed Dec 21 08:16 PST 1994
To: "Bob.Snively" <Bob.Snively at Eng>,
        fibre-channel-ext <fibre-channel-ext at Think.COM>,
        scsi-request <scsi-request at wichitaks.ncr.com>
Cc: kge3 <kge3 at PO6.RV.unisys.com>, let2 <let2 at PO9.RV.unisys.com>,
        PDougherty <PDougherty at PO2.MV.unisys.com>,
        rhs1 <rhs1 at PO9.RV.unisys.com>
Subject: Re: FCP Out of Order Data Delivery
Date: Wed, 21 Dec 94 10:15:00 CST
Encoding: 116 TEXT
X-Mailer: Microsoft Mail V3.0
Content-Type: text
Content-Length: 5371
X-Lines: 116

Thank you for your response.  I would still like to go forward
with my request for the reasons noted below:
>From: Bob.Snively
>To: fibre-channel-ext; scsi-request; jjw1
>Cc: PDougherty; kge3; let2; rhs1
>Subject: Re: FCP Out of Order Data Delivery
>Date: Monday, December 19, 1994 1:43PM
>I believe that Fibre Channel already does everything you want,
>without any modifications in FCP.  At present, all point-to-point
>transfers are performed in order if continuously increasing
>RO is required by the PLOGIN function.  If a fabric is present,
>the fabric can be requested to provide in-order delivery for
>Class 2 and Class 3 behavior, where order might otherwise be
>confused somewhat.
>In the absence of these two control functions being properly
>established, no bit that FCP has control over will guarantee
>in order delivery.  In the presence of these two control functions,
>in order delivery will usually be guaranteed regardless of the FCP.

I agree with this. Sufficient controls exist within a Sequence.

>Among separate sequences of data, a well-behaved RAID box
>performing striping will not operate efficiently if it is required
>to buffer up all data until all bursts can be provided in order.
>Of course, within a sequence (for the transfer of a subblock)
>everything will be nicely in order if the two control functions
>are properly established, requiring you to look up the starting
>point only once at the beginning of each sequence.

I assume you are referring o a RAID 4 or 5 implementation.  This
implementation only makes sense if the typical IO request is small
and doesn't span disk striping boundaries very often.  Otherwise
a RAID 3 implementation would be more prudent because of the
greater bandpass. If the RAID subsystem is attachable to SCSI
Parallel, it has to be able to handle this case anyhow because of
the EMDP bit support requirement.  Since Modify Data Pointers
is optional in SCSI 2, I suspect that current RAID subsystems
handle also this case.  In any event, if there is a performance
consequence, it should be the Host system's choice rather than
the subsystem absolutely requiring support of out of Order
Data.  What constitutes efficient operation is application dependent.

>For very extreme cases, like those you describe, I would expect that
>buffering of the entire data block would take place, followed by
>a subsequent distribution of the data under host adapter or
>software control.
This solution is not workable in a Class 2 environment because we
would have to prefetch the entire scatter gather list at IO
Initiation Time and supply enough buffering for all Outstanding
IOs.  Currently we only prefetch the first 4 to 8 Scatter/Gather
control words.  The Initiator has a much greater problem that the
Target because it has to provide sufficient buffering for all
Targets in the Fabric that it communicates with, while the Target
only has to provide buffering for the attached Logical Units.

>The FCSI Profiles do not require subblocks of data to be transferred
>in order, although they do require that the transfers be
>continuously increasing within a subblock.
>The same EMDP bit could be used to require targets to
>send data without using the Modify Data Pointers function,
>although as mentioned, the FCSI Profiles explicitly allow that
>function and would prohibit the EMDP bit.  Using the bit might limit the
>maximum length of a transfer to be less than the maximum burst
>size specified by the Disconnect Reconnect Page of the MODE SENSE/SELECT

>I hope this clarification helps you understand why I could not
>recommend the change you suggest.
I recognize where you are coming from and I hope you are gaining a
sense for the problems we have to deal with.  I don't believe I am
asking for any functionality that was not already present in SCSI 2
and is present in SCSI 3 Parallel.  My proposal would not restrict
an application from using Out of Order data delivery if it makes
sense.  I am merely asking for  a methodology to specify in order
delivery for our applications.  An individual subsystem supplier could
choose to reject this  request if it did not want to support it, but
if it supports both In Order and Out of Order Data delivery, we need
a way to specify In Order only.

I am surprised that I haven't heard a response from any other system
houses,  Are we the only one that supports arbitrary scatter/gather
and plan to implement FCP?  If not, how do you solve the Out of Order


>Robert Snively
>Sun Microsystems Computer Company
>Mail Stop UMPK 12-204
>2550 Garcia Avenue
<Mountain View
<California 94043-1100

Jerry Witalka  M.S. 4873                 EMail:jjw1 at po9.rv.unisys.com
Unisys Corporation                          Voice:  612-635-7958
P.O. Box 64942
 St. Paul, MN 55164-094

----- End Included Message -----

More information about the T10 mailing list