From: "Elliott, Robert (Server Storage)" <Elliott@hp.com> To: "t10@t10.org" <t10@t10.org> Date: Fri, 3 Oct 2008 18:58:32 +0000 Subject: RE: Potentially wrong assumption of SATA spec in SAS standard X-Message-Number: 9172 Formatted message: HTML-formatted message As originally pointed out, the SATA rule just covers "dwords", and does not exclude any particular dwords. A SATA frame transmitter may insert SATA_HOLD at any time as transmit flow control, and there is no wait for SATA_HOLDA required to resume transmitting data dwords. So, those need to be included in the count. In the SATA link layer state machine, the LT6:L_SendHold state does honor an incoming SATA_HOLD and transition to LT5:L_RcvrHold to respond with SATA_HOLDA; it doesn't get hung sending SATA_HOLDs. For STP, an expander might need infinite buffer storage if it tried to buffer every primitive. "Dword" even includes invalid dwords. As Kishore mentions, deletable primitves are likely to exist but are not necessary to directly forward (and would cause overflows if forwarding to a slower logical link). Since the expander must participate in flow control on each side, SATA_HOLD and SATA_HOLDA are not necessary to directly forward either. The only dwords that matter are SATA_SOF, data dwords in the frame, and SATA_EOF; everything else just supports transmission of those. --- Rob Elliott, HP Industry Standard Server Storage ________________________________ From: Karthikeyan, Kishore K [mailto:kishore.k.karthikeyan@intel.com] Sent: Friday, October 03, 2008 1:15 PM To: Day, Brian; Elliott, Robert (Server Storage); t10@t10.org Subject: RE: Potentially wrong assumption of SATA spec in SAS standard Actually, now that I look at it again I missed one issue in statement #1. See highlighted text. "When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords (e.g., including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving SATA_HOLD before responding with SATA_HOLDA." We cannot include SATA_HOLDs in the 19 dwords. It is possible that while transmitting SATA frame to a SATA physical link and after HOLD is transmitted by SATA device, the stp/sata bridge also decides to transmit HOLDs as its transmit buffer went empty. So there is a scenario where HOLDs are transmitted by both host and sata device on the wire. In such a scenario, the host will not respond to HOLD from the device with HOLDA, till it is ready to transmit again (its transmit buffer is not empty), at which point if the sata device is still transmitting HOLDs, it will transition to transmitting HOLDA. So #1 should read as When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords ( including data dwords, deletable primitives and SATA_EOF) after receiving SATA_HOLD and before responding with SATA_HOLDA." As for #2, I thought #2 was intented for the STP link and not the sata physical link. And on STP link, we cannot say dwords because dwords include rate matching aligns and clk_skew aligns too. Then in a 6G link rate matched to 1.5G connection rate (3 rate matching ALIGNs for every other dword), we can only transmit 6 data dwords (36 dwords/4 = 9 - 2 (clkskew aligns) - 1 SATA_EOF = 6 data dwords) after receiving HOLD. So I believe for STP links, we need to be careful when we say dwords and not specify what kind of dwords. Kishore ________________________________ From: Day, Brian [mailto:Brian.Day@lsi.com] Sent: Friday, October 03, 2008 9:09 AM To: Karthikeyan, Kishore K; Elliott, Robert (Server Storage); t10@t10.org Subject: RE: Potentially wrong assumption of SATA spec in SAS standard Re #2... why do we need to treat SATA_EOF and data dwords as special? The transmit requirement in #1 isn't specific on what type of dword. Seems to me the real intent was generic "dwords", regardless of what they are. Why not: a) 24 dwords at the 1.5 Gbps connection rate; b) 28 dwords at the 3 Gbps connection rate; or c) 36 dwords at the 6 Gbps connection rate, and shall expect to receive SATA_HOLDA within that number of dwords." (Plus changing "data dwords" to just "dwords" for the remainder of this section relative to any flow control) Brian Day LSI Corp. ________________________________ From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Karthikeyan, Kishore K Sent: Thursday, October 02, 2008 6:58 PM To: Elliott, Robert (Server Storage); t10@t10.org Subject: RE: Potentially wrong assumption of SATA spec in SAS standard Hi Rob The changes that you intend to make in the text look good Thanks Kishore ________________________________ From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Elliott, Robert (Server Storage) Sent: Thursday, September 25, 2008 1:03 PM To: t10@t10.org Subject: RE: Potentially wrong assumption of SATA spec in SAS standard I don't think any expanders try to push the limits here, but agree the discrepency should be corrected. 1. I will change SAS-2 rules about transmission on the SATA physical link to: "When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords (e.g., including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving SATA_HOLD before responding with SATA_HOLDA." 2. Although the budget for STP should not include deletable primitives or transmit-direction SATA_HOLDs (there's no reason to put those in the buffer), it should include both data dwords and SATA_EOF. I'll add "or SATA_EOFs" in several places, like this: "After transmitting SATA_HOLD, it shall accept at least the following number of data dwords or SATA_EOFs for the SATA frame into its STP flow control buffer: a) 24 data dwords or SATA_EOFs at the 1.5 Gbps connection rate; b) 28 data dwords or SATA_EOFs at the 3 Gbps connection rate; or c) 36 data dwords or SATA_EOFs at the 6 Gbps connection rate, and shall expect to receive SATA_HOLDA within that number of data dwords or SATA_EOFs." ________________________________ From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Karthikeyan, Kishore K Sent: Monday, September 22, 2008 9:19 AM To: t10@t10.org Subject: Potentially wrong assumption of SATA spec in SAS standard All In Section 7.18.2 STP Flow Control, it talks about 20 DATA dwords when referring to frame transmission and reception on the SATA physical link of the STP/SATA bridge. But SATA spec does not say "data dwords" but simply "dwords" which means it includes primitives too (for eg ALIGNs and EOF) essentially reducing the number of data dwords that can be transmitted after reception of HOLD and before transmission of HOLDA and some SATA disk drive vendors have interpreted the SATA spec as such. Hence these drives cannot accept more than 20 dwords (including primitives like ALIGNs and EOF) after it has transmitted HOLD. Such SATA drives will have incompatibility with expanders whose STP/SATA bride is designed according to SAS standard (which says 20 DATA dwords instead of just 20 dwords). For eg. if the expander is forwarding a frame to the SATA drive and it receives a HOLD from this drive when only 18 dwords were remaining to be transmitted, it can potentially just transmit all the remaining 18 dwords + 2 CLKSKEW ALIGNs + CRC + EOF = 22 dwords and not transmit HOLDA. SAS standard can assume data dwords on STP phys because it is talking to a SAS device but when referring to traffic on SATA physical links; it has to match whatever is mentioned in the SATA specs to be compatible with existing SATA drives in the market. I would think that this needs to be fixed to prevent incompatiblity between STP/SATA bridge designs in expanders and SATA drives in market. Given below is the text from 7.18.2 in SAS Standard rev sas2r14d When a SATA host phy in an STP/SATA bridge is receiving a SATA frame from a SATA physical link, it shall transmit a SATA_HOLD when it is only capable of receiving 21 more data dwords. It shall stop transmitting SATA_HOLD (e.g., return to transmitting SATA_R_IP) when it is capable of receiving at least 21 more data dwords. NOTE 83 - SATA requires that frame transmission cease and SATA_HOLDA be transmitted within 20 data dwords of receiving SATA_HOLD. Since the SATA physical link has non-zero propagation time, one dword of margin is included. When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 data dwords after receiving SATA_HOLD. NOTE 84 - SATA assumes that once a SATA_HOLD is transmitted, frame transmission ceases and SATA_HOLDA arrives within 20 dwords. Since the SATA physical link has non-zero propagation time, one dword of margin is included. Given below is the text from the SATA 2.6 spec 9.4.7 Flow Control Signaling Latency In the case where the receiver wants to flow control the incoming data, it transmits HOLDP characters on the back channel. Some number of received Dwords later, valid data ceases, and HOLDAP characters are received. The larger the latency between transmitting HOLDP until receiving HOLDAP, the larger the receive FIFO needs to be. Within a single HOLDP/ HOLDAP sequence, the maximum allowed latency from the time the MSB of the initial HOLDP is on the wire, until the MSB of the initial HOLDAP is on the wire shall be no more than 20 Dword times. The LSB is transmitted first. A receiver shall be able to accommodate reception of 20 Dwords of additional data after the time it transmits the HOLDP flow control character to the transmitter, and the transmitter shall respond with a HOLDAP in response to receiving a HOLDP within 20 Dword times. The 20 Dword latency specification is not applicable to any subsequent transmissions of the HOLDP flow control character within the same sequence. Upon each new instantiation of a HOLDP/ HOLDAP sequence, the receiver and transmitter shall meet the 20 Dword latency specification