Apparently SATA has two rules that cannot simultaneously be met.  The 20 DWord rule lists no exceptions.
 
In 2004 during ATA/ATAPI-7 v3 letter ballot comment resolution, I submitted two letter ballot comments on r4a in that area (see T13/e04124r0; also posted on the T13 reflector on 9 Feb 2004):
Page: 146
Author: relliott
Date: 4/7/2004 1:15:24 PM
ALIGNs sent by the frame transmitter are not supposed to count against this budget, although they do consume a "DWORD time."
Sending 10 data dwords, 2 ALIGNs, and 10 data dwords would technically violate the above statement but not really be an error.
Plenary Comments:
Making the proposed change would make the requirement less restrictive. The current requirement is functional and more constraining, but not a functional issue. Recommend review in ATA-8 for improved requirements.
Plenary Disposition: Rejected 4/20/2004 9:25:53 AM
Author: relliott
Date: 4/7/2004 1:15:20 PM
HOLDs sent by the frame transmitter cannot always count against this budget. Assume the frame transmitter is in LT4:L_SendData and it finds it has no more data to send, so it moves to LT6:L_SendHOLD and starts transmitting HOLDs. It remains in that state as long as it has no more data to send. If a HOLD shows up, it does not leave the state and switch to transmitting HOLDAs; it continues sending HOLDs. Only when it decides it has no more data to send does it switch states; it goes to LT5:L_RcvrHold until the incoming HOLDs go away.
Unlike ALIGNs, the HOLDs cannot just be excluded from the budget. The frame transmitter won't ever leave the HOLD state to send a few more data dwords before sending the HOLDA in reply to a HOLD it received while it was transmitting HOLD itself; it always goes directly to sending HOLDA.
 
Plenary Resolution:
Add note for exception case where HOLD/HOLD condition overrides 20 DWORD times requirement and HOLDA transition:
(See figures (link xmit & Link Receive) for HOLD/HOLD transition cases that do respond with HOLDA within 20 DWORD
latency times)
Status:
Plenary Disposition: Accepted 4/20/2004 9:40:20 AM
 
The minutes (e04125r0) say:

...The section on DWORD times with regards to ALIGN behavior would make the requirement, as currently stated, less restrictive and should be deferred to a future standards project (potentially ATA-8). In the case of the HOLD/HOLD case, text will be added to the section (15.4.8.1) to cross-reference to the appropriate sections on HOLD behavior in relation to HOLDA reception within 20 DWORD latency times. In addition the 20 DWORD latency requirements were left unchanged for this standard as the requirement currently exceeds designs and has sufficient margin. However, this issue should be studied further for the next ATA project.

This sentence was added in r4b to address the HOLD/HOLD issue:

See Figure 59 and Figure 60 for HOLD/HOLD transition cases that do respond with HOLDA within 20 DWORD latency times.

I don't see how that was supposed to resolve the question, though. The parenthetical wording might have been intended as a reminder to the editor about which state machines to review to construct a proper resolution, but accidentally incorporated as the resolution).  SATA-IO did not adopt any such sentence in subsequent releases.

It would be strange to exclude HOLD (which must be fed into state machines by the receiver) but include ALIGN (which is expected to be deleted very early in the receiver).
 
The phrase "20 Dwords of additional data" is also present, using terminology that is not well defined.  "Data dwords" might have been the intent.
 
Someone involved in SATA-IO may wish to ask this question in that forum.
 

From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Karthikeyan, Kishore K
Sent: Friday, October 03, 2008 3:24 PM
To: t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

See embedded comments below

 

From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Elliott, Robert (Server Storage)
Sent: Friday, October 03, 2008 11:59 AM
To: t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

 

As originally pointed out, the SATA rule just covers "dwords", and does not exclude any particular dwords.  A SATA frame transmitter may insert SATA_HOLD at any time as transmit flow control, and there is no wait for SATA_HOLDA required to resume transmitting data dwords.  So, those need to be included in the count.  In the SATA link layer state machine, the LT6:L_SendHold state does honor an incoming SATA_HOLD and transition to LT5:L_RcvrHold to respond with SATA_HOLDA; it doesn't get hung sending SATA_HOLDs.

[Kishore K] LT6:L_SendHold state will transition to LT5:L_RcvrHold state only if two conditions are satisfied as specified in point 3 of that state in the SATA spec which says, “More data ready to transmit and HOLDP received from Phy”.

Just because it receives HOLD from device, it cannot transition to LT5 state from LT6 state. It should 1)have data ready to transmit and also 2) HOLD is received will cause it to transition to LT5 state. So what I am trying to say is we cannot include SATA_HOLD in the count.

 

For STP, an expander might need infinite buffer storage if it tried to buffer every primitive.  "Dword" even includes invalid dwords.  As Kishore mentions, deletable primitves are likely to exist but are not necessary to directly forward (and would cause overflows if forwarding to a slower logical link).  Since the expander must participate in flow control on each side, SATA_HOLD and SATA_HOLDA are not necessary to directly forward either.  The only dwords that matter are SATA_SOF, data dwords in the frame, and SATA_EOF; everything else just supports transmission of those.

 

---
Rob Elliott, HP Industry Standard Server Storage

 

From: Karthikeyan, Kishore K [mailto:kishore.k.karthikeyan@intel.com]
Sent: Friday, October 03, 2008 1:15 PM
To: Day, Brian; Elliott, Robert (Server Storage); t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

Actually, now that I look at it again I missed one issue in statement #1. See highlighted text.

 

"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords (e.g., including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving SATA_HOLD before responding with SATA_HOLDA."

 

We cannot include SATA_HOLDs in the 19 dwords. It is possible that while transmitting SATA frame to a SATA physical link and after HOLD is transmitted by SATA device, the stp/sata bridge also decides to transmit HOLDs as its transmit buffer went empty. So there is a scenario where HOLDs are transmitted by both host and sata device on the wire. In such a scenario, the host will not respond to HOLD from the device with HOLDA, till it is ready to transmit again (its transmit buffer is not empty), at which point if the sata device is still transmitting HOLDs, it will transition to transmitting HOLDA.

 

So #1 should read as

When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords ( including data dwords, deletable primitives and SATA_EOF) after receiving SATA_HOLD and before responding with SATA_HOLDA."

 

As for #2, I thought #2 was intented for the STP link and not the sata physical link. And on STP link, we cannot say dwords because dwords include rate matching aligns and clk_skew aligns too. Then in a 6G link rate matched to 1.5G connection rate (3 rate matching ALIGNs for every other dword), we can only transmit 6 data dwords (36 dwords/4 = 9 - 2 (clkskew aligns) – 1 SATA_EOF = 6 data dwords) after receiving HOLD.

 

So I believe for STP links, we need to be careful when we say dwords and not specify what kind of dwords.

 

 

Kishore

 

 

 

 

From: Day, Brian [mailto:Brian.Day@lsi.com]
Sent: Friday, October 03, 2008 9:09 AM
To: Karthikeyan, Kishore K; Elliott, Robert (Server Storage); t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

 

Re #2... why do we need to treat SATA_EOF and data dwords as special?  The transmit requirement in #1 isn't specific on what type of dword.

Seems to me the real intent was generic "dwords", regardless of what they are.

 

Why not:

a) 24 dwords at the 1.5 Gbps connection rate;

b) 28 dwords at the 3 Gbps connection rate; or

c) 36 dwords at the 6 Gbps connection rate,

and shall expect to receive SATA_HOLDA within that number of dwords."  (Plus changing "data dwords" to just "dwords" for the remainder of this section relative to any flow control)

 

Brian Day

LSI Corp.

 

From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Karthikeyan, Kishore K
Sent: Thursday, October 02, 2008 6:58 PM
To: Elliott, Robert (Server Storage); t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

Hi Rob

The changes that you intend to make in the text look good

 

Thanks

Kishore

 

 

From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Elliott, Robert (Server Storage)
Sent: Thursday, September 25, 2008 1:03 PM
To: t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard

 

I don't think any expanders try to push the limits here, but agree the discrepency should be corrected.

 

1. I will change SAS-2 rules about transmission on the SATA physical link to:

 

"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall transmit no more than 19 dwords (e.g., including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving SATA_HOLD before responding with SATA_HOLDA."

 

2. Although the budget for STP should not include deletable primitives or transmit-direction SATA_HOLDs (there's no reason to put those in the buffer), it should include both data dwords and SATA_EOF.  I'll add "or SATA_EOFs" in several places, like this:

 

"After transmitting SATA_HOLD, it shall accept at least the following number of data dwords or SATA_EOFs for the SATA frame into its STP flow control buffer:

a) 24 data dwords or SATA_EOFs at the 1.5 Gbps connection rate;

b) 28 data dwords or SATA_EOFs at the 3 Gbps connection rate; or

c) 36 data dwords or SATA_EOFs at the 6 Gbps connection rate,

and shall expect to receive SATA_HOLDA within that number of data dwords or SATA_EOFs."


 

From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Karthikeyan, Kishore K
Sent: Monday, September 22, 2008 9:19 AM
To: t10@t10.org
Subject: Potentially wrong assumption of SATA spec in SAS standard

All

 

In Section 7.18.2 STP Flow Control, it talks about 20 DATA dwords when referring to frame transmission and reception on the SATA physical link of the STP/SATA bridge.

But SATA spec does not say “data dwords” but simply “dwords” which means it includes primitives too (for eg ALIGNs and EOF) essentially reducing the number of data dwords that can be transmitted after reception of HOLD and before transmission of HOLDA and some SATA disk drive vendors have interpreted the SATA spec as such. Hence these drives cannot accept more than 20 dwords (including primitives like ALIGNs and EOF) after it has transmitted HOLD. Such SATA drives will have incompatibility with expanders whose STP/SATA bride is designed according to SAS standard (which says 20 DATA dwords instead of just 20 dwords). For eg. if the expander is forwarding a frame to the SATA drive and it receives a HOLD from this drive when only 18 dwords were remaining to be transmitted, it can potentially just transmit all the remaining 18 dwords + 2 CLKSKEW ALIGNs + CRC + EOF = 22 dwords and not transmit HOLDA.

 

SAS standard can assume data dwords on STP phys because it is talking to a SAS device but when referring to traffic on SATA physical links; it has to match whatever is mentioned in the SATA specs to be compatible with existing SATA drives in the market.

I would think that this needs to be fixed to prevent incompatiblity between STP/SATA bridge designs in expanders and SATA drives in market.

 

 

Given below is the text from 7.18.2 in SAS Standard rev sas2r14d

 

When a SATA host phy in an STP/SATA bridge is receiving a SATA frame from a SATA physical link, it shall

transmit a SATA_HOLD when it is only capable of receiving 21 more data dwords. It shall stop transmitting

SATA_HOLD (e.g., return to transmitting SATA_R_IP) when it is capable of receiving at least 21 more data

dwords.

 

NOTE 83 - SATA requires that frame transmission cease and SATA_HOLDA be transmitted within 20 data

dwords of receiving SATA_HOLD. Since the SATA physical link has non-zero propagation time, one dword of

margin is included.

 

When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a SATA physical link, it shall

transmit no more than 19 data dwords after receiving SATA_HOLD.

 

NOTE 84 - SATA assumes that once a SATA_HOLD is transmitted, frame transmission ceases and

SATA_HOLDA arrives within 20 dwords. Since the SATA physical link has non-zero propagation time, one

dword of margin is included.

 

 

 

Given below is the text from the SATA 2.6 spec

 

9.4.7 Flow Control Signaling Latency

In the case where the receiver wants to flow control the incoming data, it transmits HOLDP

characters on the back channel. Some number of received Dwords later, valid data ceases, and

HOLDAP characters are received. The larger the latency between transmitting HOLDP until

receiving HOLDAP, the larger the receive FIFO needs to be. Within a single HOLDP/ HOLDAP

sequence, the maximum allowed latency from the time the MSB of the initial HOLDP is on the

wire, until the MSB of the initial HOLDAP is on the wire shall be no more than 20 Dword times.

The LSB is transmitted first. A receiver shall be able to accommodate reception of 20 Dwords of

additional data after the time it transmits the HOLDP flow control character to the transmitter, and

the transmitter shall respond with a HOLDAP in response to receiving a HOLDP within 20 Dword

times. The 20 Dword latency specification is not applicable to any subsequent transmissions of

the HOLDP flow control character within the same sequence. Upon each new instantiation of a

HOLDP/ HOLDAP sequence, the receiver and transmitter shall meet the 20 Dword latency

specification