Subject: [T11.3] Re: FCP-4: Items for discussion Date: Mon, 1 May 2006 16:37:31 -0700 From: "Robert Snively" <rsnively@Brocade.COM> To: <Gary.Franco@Emulex.Com>, <David.Peterson@mcdata.com>, <t10@t10.org>, <t11_3@t11.org> Cc: <Bob.Nixon@Emulex.Com> Formatted message: HTML-formatted message Please let me add some explanations to this to reflect some of the early design intentions. My comments in red. I please ask you to discuss in the FCP-4 WG the possibility to mandate continuous increasing SEQ_CNT in FCP-4. Relying on it greatly simplifies the detection of a missing Sequence and consequently simplifies error recovery, but today is optional in FCP-x (while is mandated by IP over FC and is going to be mandated by FC-SATA). While handy enough, continuously increasing sequence count is not a requirement except for streamed sequences. See FC-PLDA for the original view of that. Note that some implementations choose to zero the sequence count in each sequence because so many exchanges may be in process at the same time. With the offset available in each frame, the sequence count continuously increasing across sequences is not necessary to recover all critical information. Additional items for FCP-4 discussion follows (all due to doubts submitted to me...): - Bidirectional Commands: I think we need to count bytes for both data-in and data-out. Which of these two counters should be put in the FC4VALUE field of the REC ELS? Wouldn't that be dependant on the type of command? FCP read or write types. The device knows the current command type. See FC-LS, clause 4.2.55.4. The E_STAT value indicates which end holds initiative and whether the exchange is complete. On bidirectional commands, I would expect the FC4VALUE to reflect the phase that is active when the REC arrives. Note that REC is supplemented by lots of other information in determining this value, so it is the responder who must make up its mind and make sure that information is consistent. - Data Overlay: the FCP-3 definition of data overlay says "see SAM-3", but SAM-3 says nothing on data overlay. I would say that EMDP should be disabled cause with the end to end data protection I do not think you can even enable this option because the DIFs would get corrupted would they not? Realistically EMDP=1 behavior is rarely implemented. However, when it is implemented, there is an understanding that the target and initiator know how to handle it. DIFs would only be corrupted if the target or initiator did NOT know how to handle EMDP. If DIFs were used with EMDP, I would expect that full buffer accumulation and post-processing would be required to accumulate the proper check values, making it even less likely that EMDP would be supported, but not creating grounds to forbid it from the architecture. - Data Overlay: In which way could it be possible detecting a missing Sequence when data overlay is used and continuously increasing SEQ_CNT is not used? (it seems to us that there is no way, but others may have a different opinion...). I agree, I think there is no possible way, at least real-time as the data is being streamed in. At FCP response time the overrun/underrun and residual length would cause a re-execution of the command. Continuously increasing sequence count does not help recover from failures in proper transmission during data overlay transfers. If you are using data overlay, the receiving device must keep a mask indicating which data has been transferred and which has not in order to maintain the data counts correctly. Note that data overlay may even transfer the same data twice, but it must be counted only once. A missing block in the middle is treated as defined in FCP-3 clause 9.4.1, below: " If error conditions occur that prevent the transfer of data in the middle of a data transfer, the FCP_SNS_INFO shall indicate that only data from the offset of zero up to the first byte of missing data has been transferred " - Data Overlay: how can the FC4VALUE counters can be accurate when data overlap (i.e., how to avoid to count twice the overlapping data)? I would think that the ability to resend data blocks already sent was discussed in the specification as a bad thing. If not then I believe the only way to catch a misbehaving target would be to track the data blocks already received and keep track of the holes left behind. The device would also have to track the resent data and not include the resent data as data received to satisfy a read command. The resending of data blocks may be a good thing if the first copy was sent with incorrect data. However, you are completely correct when you say that the initiator must track the data blocks received, counting each one only once even if it was received multiple times. Note that there is no requirement for the initiator to guarantee data integrity with such tools, but it would be foolish to allow EMDP=1 behavior without such checks. The members of the working group came to no real concensus/resolution per the questions and would like to open up the discussion to the T10/T11 commitees, specifically making Continuously Increasing SEQ_CNT a requirement for FCP-4. Making this a requirement would be a much need improvement, and one that has been needed for a while. I disagree. See above and below. Regarding Continuously Increasing SEQ_CNT: We have discussed requiring Continuously Increasing SEQ_CNT during each previous FCP-x standard development efforts and folk opted to not specify it as a requirement since some vendors did not yet fully support it. We may have now moved past that issue. Sorry, while that may be true of individual drives, it would be a problem for devices with large numbers of exchanges outstanding, each requiring context information for transfers in both directions. There is no particular improvement in checking because of this, since data offsets are carried in every header. Regarding Bidirectional Commands: FCP-3 states: "Sequence level error recovery as described in 12.4 shall not be used for bidirectional SCSI commands." So the question regarding the FC4VALUE field is moot until if/when we want to support FCP-x error detection and recovery for bidirectional commands. I believe we can allow recovery for bidirectional SCSI commands using the additional information in the E_STAT field in the REC and the information in the SRR. It would take a bit of work, but it should not be too demanding an exercise since bidirectional commands are required to complete one transfer before starting the other, simplifying the requirements significantly. SRR may need another bit or two added. The only reason we didn't do it in the earlier revisions was that bidirectional SCSI commands were added late in one of the FCP-x documents and nobody has bothered to revisit the retry processes with such commands in mind. Regarding Data Overlay: The reference to SAM-3 is not intended to refer the reader to SAM-3 for data overlay, but since it is outside the sentence this is what it means. The reference will be removed since it provides value in this context (i.e., the intent was to refer the reader to SAM-3 for application client buffer offset but that is already covered). The other two questions are vendor implementation specifc in my mind, others may share if they wish... Agreed. See above.