Potentially wrong assumption of SATA spec in SAS standard

Elliott, Robert (Server Storage) Elliott at hp.com
Fri Oct 3 17:30:13 PDT 2008


Formatted message: <A HREF="r081003a_f.htm">HTML-formatted message</A>

Apparently SATA has two rules that cannot simultaneously be met.  The 20
DWord rule lists no exceptions.
In 2004 during ATA/ATAPI-7 v3 letter ballot comment resolution, I submitted
two letter ballot comments on r4a in that area (see T13/e04124r0; also posted
on the T13 reflector on 9 Feb 2004):
Page: 146
Author: relliott
Date: 4/7/2004 1:15:24 PM
ALIGNs sent by the frame transmitter are not supposed to count against this
budget, although they do consume a "DWORD time."
Sending 10 data dwords, 2 ALIGNs, and 10 data dwords would technically
violate the above statement but not really be an error.
Plenary Comments:
Making the proposed change would make the requirement less restrictive. The
current requirement is functional and more constraining, but not a functional
issue. Recommend review in ATA-8 for improved requirements.
Plenary Disposition: Rejected 4/20/2004 9:25:53 AM
Author: relliott
Date: 4/7/2004 1:15:20 PM
HOLDs sent by the frame transmitter cannot always count against this budget.
Assume the frame transmitter is in LT4:L_SendData and it finds it has no more
data to send, so it moves to LT6:L_SendHOLD and starts transmitting HOLDs. It
remains in that state as long as it has no more data to send. If a HOLD shows
up, it does not leave the state and switch to transmitting HOLDAs; it
continues sending HOLDs. Only when it decides it has no more data to send
does it switch states; it goes to LT5:L_RcvrHold until the incoming HOLDs go
away.
Unlike ALIGNs, the HOLDs cannot just be excluded from the budget. The frame
transmitter won't ever leave the HOLD state to send a few more data dwords
before sending the HOLDA in reply to a HOLD it received while it was
transmitting HOLD itself; it always goes directly to sending HOLDA.
Plenary Resolution:
Add note for exception case where HOLD/HOLD condition overrides 20 DWORD
times requirement and HOLDA transition:
(See figures (link xmit & Link Receive) for HOLD/HOLD transition cases that
do respond with HOLDA within 20 DWORD
latency times)
Status:
Plenary Disposition: Accepted 4/20/2004 9:40:20 AM
The minutes (e04125r0) say:
...The section on DWORD times with regards to ALIGN behavior would make the
requirement, as currently stated, less restrictive and should be deferred to
a future standards project (potentially ATA-8). In the case of the HOLD/HOLD
case, text will be added to the section (15.4.8.1) to cross-reference to the
appropriate sections on HOLD behavior in relation to HOLDA reception within
20 DWORD latency times. In addition the 20 DWORD latency requirements were
left unchanged for this standard as the requirement currently exceeds designs
and has sufficient margin. However, this issue should be studied further for
the next ATA project.
This sentence was added in r4b to address the HOLD/HOLD issue:
See Figure 59 and Figure 60 for HOLD/HOLD transition cases that do respond
with HOLDA within 20 DWORD latency times.
I don't see how that was supposed to resolve the question, though. The
parenthetical wording might have been intended as a reminder to the editor
about which state machines to review to construct a proper resolution, but
accidentally incorporated as the resolution).  SATA-IO did not adopt any such
sentence in subsequent releases.
It would be strange to exclude HOLD (which must be fed into state machines by
the receiver) but include ALIGN (which is expected to be deleted very early
in the receiver).
The phrase "20 Dwords of additional data" is also present, using terminology
that is not well defined.  "Data dwords" might have been the intent.
Someone involved in SATA-IO may wish to ask this question in that forum.
________________________________
From: owner-t10 at t10.org [mailto:owner-t10 at t10.org] On Behalf Of Karthikeyan,
Kishore K
Sent: Friday, October 03, 2008 3:24 PM
To: t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
See embedded comments below
________________________________
From: owner-t10 at t10.org [mailto:owner-t10 at t10.org] On Behalf Of Elliott,
Robert (Server Storage)
Sent: Friday, October 03, 2008 11:59 AM
To: t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
As originally pointed out, the SATA rule just covers "dwords", and does not
exclude any particular dwords.	A SATA frame transmitter may insert SATA_HOLD
at any time as transmit flow control, and there is no wait for SATA_HOLDA
required to resume transmitting data dwords.  So, those need to be included
in the count.  In the SATA link layer state machine, the LT6:L_SendHold state
does honor an incoming SATA_HOLD and transition to LT5:L_RcvrHold to respond
with SATA_HOLDA; it doesn't get hung sending SATA_HOLDs.
[Kishore K] LT6:L_SendHold state will transition to LT5:L_RcvrHold state only
if two conditions are satisfied as specified in point 3 of that state in the
SATA spec which says, "More data ready to transmit and HOLDP received from
Phy".
Just because it receives HOLD from device, it cannot transition to LT5 state
|from LT6 state. It should 1)have data ready to transmit and also 2) HOLD is
received will cause it to transition to LT5 state. So what I am trying to say
is we cannot include SATA_HOLD in the count.
For STP, an expander might need infinite buffer storage if it tried to buffer
every primitive.  "Dword" even includes invalid dwords.  As Kishore mentions,
deletable primitves are likely to exist but are not necessary to directly
forward (and would cause overflows if forwarding to a slower logical link). 
Since the expander must participate in flow control on each side, SATA_HOLD
and SATA_HOLDA are not necessary to directly forward either.  The only dwords
that matter are SATA_SOF, data dwords in the frame, and SATA_EOF; everything
else just supports transmission of those.
---
Rob Elliott, HP Industry Standard Server Storage
________________________________
From: Karthikeyan, Kishore K [mailto:kishore.k.karthikeyan at intel.com]
Sent: Friday, October 03, 2008 1:15 PM
To: Day, Brian; Elliott, Robert (Server Storage); t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Actually, now that I look at it again I missed one issue in statement #1. See
highlighted text.
"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a
SATA physical link, it shall transmit no more than 19 dwords (e.g., including
data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving
SATA_HOLD before responding with SATA_HOLDA."
We cannot include SATA_HOLDs in the 19 dwords. It is possible that while
transmitting SATA frame to a SATA physical link and after HOLD is transmitted
by SATA device, the stp/sata bridge also decides to transmit HOLDs as its
transmit buffer went empty. So there is a scenario where HOLDs are
transmitted by both host and sata device on the wire. In such a scenario, the
host will not respond to HOLD from the device with HOLDA, till it is ready to
transmit again (its transmit buffer is not empty), at which point if the sata
device is still transmitting HOLDs, it will transition to transmitting HOLDA.
So #1 should read as
When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a
SATA physical link, it shall transmit no more than 19 dwords ( including data
dwords, deletable primitives and SATA_EOF) after receiving SATA_HOLD and
before responding with SATA_HOLDA."
As for #2, I thought #2 was intented for the STP link and not the sata
physical link. And on STP link, we cannot say dwords because dwords include
rate matching aligns and clk_skew aligns too. Then in a 6G link rate matched
to 1.5G connection rate (3 rate matching ALIGNs for every other dword), we
can only transmit 6 data dwords (36 dwords/4 = 9 - 2 (clkskew aligns) - 1
SATA_EOF = 6 data dwords) after receiving HOLD.
So I believe for STP links, we need to be careful when we say dwords and not
specify what kind of dwords.
Kishore
________________________________
From: Day, Brian [mailto:Brian.Day at lsi.com]
Sent: Friday, October 03, 2008 9:09 AM
To: Karthikeyan, Kishore K; Elliott, Robert (Server Storage); t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Re #2... why do we need to treat SATA_EOF and data dwords as special?  The
transmit requirement in #1 isn't specific on what type of dword.
Seems to me the real intent was generic "dwords", regardless of what they
are.
Why not:
a) 24 dwords at the 1.5 Gbps connection rate;
b) 28 dwords at the 3 Gbps connection rate; or
c) 36 dwords at the 6 Gbps connection rate,
and shall expect to receive SATA_HOLDA within that number of dwords."  (Plus
changing "data dwords" to just "dwords" for the remainder of this section
relative to any flow control)
Brian Day
LSI Corp.
________________________________
From: owner-t10 at t10.org [mailto:owner-t10 at t10.org] On Behalf Of Karthikeyan,
Kishore K
Sent: Thursday, October 02, 2008 6:58 PM
To: Elliott, Robert (Server Storage); t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Hi Rob
The changes that you intend to make in the text look good
Thanks
Kishore
________________________________
From: owner-t10 at t10.org [mailto:owner-t10 at t10.org] On Behalf Of Elliott,
Robert (Server Storage)
Sent: Thursday, September 25, 2008 1:03 PM
To: t10 at t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
I don't think any expanders try to push the limits here, but agree the
discrepency should be corrected.
1. I will change SAS-2 rules about transmission on the SATA physical link to:
"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a
SATA physical link, it shall transmit no more than 19 dwords (e.g., including
data dwords, deletable primitives, SATA_HOLD, and SATA_EOF) after receiving
SATA_HOLD before responding with SATA_HOLDA."
2. Although the budget for STP should not include deletable primitives or
transmit-direction SATA_HOLDs (there's no reason to put those in the buffer),
it should include both data dwords and SATA_EOF.  I'll add "or SATA_EOFs" in
several places, like this:
"After transmitting SATA_HOLD, it shall accept at least the following number
of data dwords or SATA_EOFs for the SATA frame into its STP flow control
buffer:
a) 24 data dwords or SATA_EOFs at the 1.5 Gbps connection rate;
b) 28 data dwords or SATA_EOFs at the 3 Gbps connection rate; or
c) 36 data dwords or SATA_EOFs at the 6 Gbps connection rate,
and shall expect to receive SATA_HOLDA within that number of data dwords or
SATA_EOFs."
________________________________
From: owner-t10 at t10.org [mailto:owner-t10 at t10.org] On Behalf Of Karthikeyan,
Kishore K
Sent: Monday, September 22, 2008 9:19 AM
To: t10 at t10.org
Subject: Potentially wrong assumption of SATA spec in SAS standard
All
In Section 7.18.2 STP Flow Control, it talks about 20 DATA dwords when
referring to frame transmission and reception on the SATA physical link of
the STP/SATA bridge.
But SATA spec does not say "data dwords" but simply "dwords" which means it
includes primitives too (for eg ALIGNs and EOF) essentially reducing the
number of data dwords that can be transmitted after reception of HOLD and
before transmission of HOLDA and some SATA disk drive vendors have
interpreted the SATA spec as such. Hence these drives cannot accept more than
20 dwords (including primitives like ALIGNs and EOF) after it has transmitted
HOLD. Such SATA drives will have incompatibility with expanders whose
STP/SATA bride is designed according to SAS standard (which says 20 DATA
dwords instead of just 20 dwords). For eg. if the expander is forwarding a
frame to the SATA drive and it receives a HOLD from this drive when only 18
dwords were remaining to be transmitted, it can potentially just transmit all
the remaining 18 dwords + 2 CLKSKEW ALIGNs + CRC + EOF = 22 dwords and not
transmit HOLDA.
SAS standard can assume data dwords on STP phys because it is talking to a
SAS device but when referring to traffic on SATA physical links; it has to
match whatever is mentioned in the SATA specs to be compatible with existing
SATA drives in the market.
I would think that this needs to be fixed to prevent incompatiblity between
STP/SATA bridge designs in expanders and SATA drives in market.
Given below is the text from 7.18.2 in SAS Standard rev sas2r14d
When a SATA host phy in an STP/SATA bridge is receiving a SATA frame from a
SATA physical link, it shall
transmit a SATA_HOLD when it is only capable of receiving 21 more data
dwords. It shall stop transmitting
SATA_HOLD (e.g., return to transmitting SATA_R_IP) when it is capable of
receiving at least 21 more data
dwords.
NOTE 83 - SATA requires that frame transmission cease and SATA_HOLDA be
transmitted within 20 data
dwords of receiving SATA_HOLD. Since the SATA physical link has non-zero
propagation time, one dword of
margin is included.
When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame to a
SATA physical link, it shall
transmit no more than 19 data dwords after receiving SATA_HOLD.
NOTE 84 - SATA assumes that once a SATA_HOLD is transmitted, frame
transmission ceases and
SATA_HOLDA arrives within 20 dwords. Since the SATA physical link has
non-zero propagation time, one
dword of margin is included.
Given below is the text from the SATA 2.6 spec
9.4.7 Flow Control Signaling Latency
In the case where the receiver wants to flow control the incoming data, it
transmits HOLDP
characters on the back channel. Some number of received Dwords later, valid
data ceases, and
HOLDAP characters are received. The larger the latency between transmitting
HOLDP until
receiving HOLDAP, the larger the receive FIFO needs to be. Within a single
HOLDP/ HOLDAP
sequence, the maximum allowed latency from the time the MSB of the initial
HOLDP is on the
wire, until the MSB of the initial HOLDAP is on the wire shall be no more
than 20 Dword times.
The LSB is transmitted first. A receiver shall be able to accommodate
reception of 20 Dwords of
additional data after the time it transmits the HOLDP flow control character
to the transmitter, and
the transmitter shall respond with a HOLDAP in response to receiving a HOLDP
within 20 Dword
times. The 20 Dword latency specification is not applicable to any subsequent
transmissions of
the HOLDP flow control character within the same sequence. Upon each new
instantiation of a
HOLDP/ HOLDAP sequence, the receiver and transmitter shall meet the 20 Dword
latency
specification



More information about the T10 mailing list