FCP-2 error recovery - an alternative

[Zeitler, Carl]

* From the T10 Reflector (t10 at t10.org), posted by:
* "Zeitler, Carl" <Carl.Zeitler at compaq.com>
*
Matt's idea is kind of intriguing.  Another variation would be to shift the
old Exchange context  to the Exchange carrying SRR.  A new bit in the SRR
Payload would indicate the shift.  The old Exchange is restarted using the
new Exchange ID.  Then add a bit to RRQ to kill the original Exchange after
R_A_TOV.  

Regards, Carl

Carl Zeitler
Compaq Computer Corporation
MS 150801, 20555 SH249, Houston, TX 77070
Phone:281-518-5258 Fax: 281-514-5270
E-Mail: Carl.Zeitler at compaq.com


-----Original Message-----
From: Matt Wakeley [mailto:matt_wakeley at agilent.com]
Sent: Tuesday, July 11, 2000 3:25 AM
To: t10 at t10.org; fc at network.com
Subject: FCP-2 error recovery - an alternative


*
* From the fc reflector, posted by:
* "Matt Wakeley" <matt_wakeley at agilent.com>
*
Summary of the proposal:  remove out all the "sequence error recovery"
stuff.
Keep the error *detection* stuff (issuing REC periodically).  When an error
is
detected, abort the exchange with an indication that the command is not
being
aborted, only moved to a new exchange. Reissue the command on a new exchange
with an indication that the command is not a new command, but a retry of the
erred command on the new exchange.

Discussion: (this might even be as good as Dal's ENDL writing! :)

FC-PH defined "tools" for "recovery" of errors on fibre channel sequences.
When there is an error in a fibre channel sequence (think of a sequence as a
message), FC-PH defines a "recovery qualifier" to reject all *frames* on the
exchange based on the "sequence count range" of the frames within the erred
sequence (message) (to reject these frames of this erred message if they
should happen to pop out of an out-of-order fabric after the sequence error
was detected/retried) - even if the frames received have a different
sequence_id (stupid).

This is causing all kinds of headaches for FCP-2 out of order sequence level
error recovery. All kinds of hoops have to be thought out and implemented to
perform recovery of the sequence (and continuing the exchange) without
reusing
sequence count numbers that are "set aside" for R_A_TOV (this really
presents
a problem when continuously increasing sequence count is used - once the
exchange wraps around and hits the range of sequence counts that are "set
aside", the whole exchange must stop until the R_A_TOV has elapsed).

Note that all previous implementations of protocols on FC have not even
attempted this - they just give up and abort the whole exchange (both PLDA
and
FC-VI do this).

What FC-PH *should* have done is create a "recovery qualifier" based on the
*sequence_id* that had the error.  That way, when the "sequence recovery" is
performed, there would be no worry about reusing sequence count numbers that
are "set aside".  Instead, recovery would be performed using a different
sequence_id and the exchange can continue using all values of sequence count
and all values of sequence_id except for the one with the error.  While the
R_A_TOV time-out is running, whenever the sequence recipient receives a
frame
whose sequence_id matches the one with the error, the frame is discarded.

"Oh my god, this is a huge change to FC-PH/FC-FS and existing
implementations!"  Yeah, so?  It's broke, it needs to be fixed.

It could be fixed in FC-FS by adding a bit to the ABTS field that says "use
the new smart recovery qualifier based on sequence_id".  But that may not
help
old implementations and those that are reluctant to change might put up
resistance...

So, getting back to what is done today, FC disc solutions implementing PLDA
simply abort the exchange whenever an error is detected.  The disc command
is
then re-issued on a new exchange and any frames received for the old
exchange
are discarded.

The FC-Tape/FCP-2 projects where started because "FC tapes could not abort
an
exchange (command) and then re-perform it".  I'm not sure I buy that
argument
anymore.  A tape vendor in the July 10 meeting said "We lie about status.
We
always immediately send the response before the data is committed to tape.
We
buffer 1000's of commands..."  Ok, if they are buffered, why can't they be
aborted and re-issued?  Just replace the erred task in your "buffer" with
the
re-issued one...

But anyway, so this error recovery scheme was invented.  It can be summed up
as, a command is issued to the tape device.  The initiator polls the tape
device to determine forward progress and error detection.  In order to
facilitate error recovery, the tape device must buffer and maintain the
complete I/O until it can be unambiguously determined that the I/O completed
successfully.  To perform error recovery, the initiator must determine what
state the I/O is in and attempt to (re)transfer with the target the erred
data
and/or response, attempting to do so without reusing the "set aside"
sequence
counts, and taking into account out of order frames popping out of the
fabric.

(if you think that that "out of order" case does not need to be considered,
think about the flurry of "solutions" that are connecting "FC SANs" together
by tunneling over the "internet".  There certainly is no ordering guaranteed
across the "internet".

So, let's take a look at the solution that we know works today with discs...
When an error occurs, abort the exchange.  For tapes, we can add a simple
extension to this idea (to preserve the desire to *not* re-issue the
command).  Instead of "aborting" the exchange, we define a mechanism that
says
"abandon this exchange for this command - this command will be recovered on
a
new exchange".  This could be a new bit in the ABTS, or a new BLS.

Existing hardware implementations already are able to "abort" an exchange
and
try it on an new exchange (disc error recovery today). To address existing
hardware implementations that may not be able to "move" the I/O to a new
exchange, I propose that there be a bit added to the FCP_CMND that indicates
that this command is being "retried".  That way, as far as the initiator is
concerned, the erred I/O was aborted (but the target is notified that the
command is not being aborted, just moved to a new exchange) and retried on a
new exchange.  When the target receives the re-issued command with the retry
bit set, it knows this is not a new command.  Instead, it performs all the
actions as if it was a new command (since all the information is already
buffered up anyway).

[this has the added nicety that the initiator, for both tapes and discs, can
always issue the new "abort" and always re-issue the command with this new
retry bit set.  Tapes that know about these new features will perform the
correct behavior.  Discs that don't know about these new behaviors will
ignore
the new "flags" and perform the correct behavior, and it will all seem the
same to the initiator.]

Summary of the proposal:  remove out all the "sequence error recovery"
stuff.
Keep the error *detection* stuff (issuing REC periodically).  When an error
is
detected, abort the exchange with an indication that the command is not
being
aborted, only moved to a new exchange. Reissue the command on a new exchange
with an indication that the command is not a new command, but a retry of the
erred command on the new exchange.

Comments? (shields are engaged)

Matt Wakeley
Agilent Technologies


*
* For T10 Reflector information, send a message with
* 'info t10' (no quotes) in the message body to majordomo at t10.org