Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Date: Fri, 3 Oct 2008 13:23:39 -0700
From: "Karthikeyan, Kishore K" <kishore.k.karthikeyan@intel.com>
To: <t10@t10.org>
X-Message-Number: 9174
Formatted message: HTML-formatted message

See embedded comments below
________________________________
From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Elliott,
Robert (Server Storage)
Sent: Friday, October 03, 2008 11:59 AM
To: t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
As originally pointed out, the SATA rule just covers "dwords", and does
not exclude any particular dwords.  A SATA frame transmitter may insert
SATA_HOLD at any time as transmit flow control, and there is no wait for
SATA_HOLDA required to resume transmitting data dwords.  So, those need
to be included in the count.  In the SATA link layer state machine, the
LT6:L_SendHold state does honor an incoming SATA_HOLD and transition to
LT5:L_RcvrHold to respond with SATA_HOLDA; it doesn't get hung sending
SATA_HOLDs.
[Kishore K] LT6:L_SendHold state will transition to LT5:L_RcvrHold state
only if two conditions are satisfied as specified in point 3 of that
state in the SATA spec which says, "More data ready to transmit and
HOLDP received from Phy".
Just because it receives HOLD from device, it cannot transition to LT5
state from LT6 state. It should 1)have data ready to transmit and also
2) HOLD is received will cause it to transition to LT5 state. So what I
am trying to say is we cannot include SATA_HOLD in the count.
For STP, an expander might need infinite buffer storage if it tried to
buffer every primitive.  "Dword" even includes invalid dwords.	As
Kishore mentions, deletable primitves are likely to exist but are not
necessary to directly forward (and would cause overflows if forwarding
to a slower logical link).  Since the expander must participate in flow
control on each side, SATA_HOLD and SATA_HOLDA are not necessary to
directly forward either.  The only dwords that matter are SATA_SOF, data
dwords in the frame, and SATA_EOF; everything else just supports
transmission of those.
--- 
Rob Elliott, HP Industry Standard Server Storage 
________________________________
From: Karthikeyan, Kishore K [mailto:kishore.k.karthikeyan@intel.com] 
Sent: Friday, October 03, 2008 1:15 PM
To: Day, Brian; Elliott, Robert (Server Storage); t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Actually, now that I look at it again I missed one issue in statement
#1. See highlighted text.
"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame
to a SATA physical link, it shall transmit no more than 19 dwords (e.g.,
including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF)
after receiving SATA_HOLD before responding with SATA_HOLDA."
We cannot include SATA_HOLDs in the 19 dwords. It is possible that while
transmitting SATA frame to a SATA physical link and after HOLD is
transmitted by SATA device, the stp/sata bridge also decides to transmit
HOLDs as its transmit buffer went empty. So there is a scenario where
HOLDs are transmitted by both host and sata device on the wire. In such
a scenario, the host will not respond to HOLD from the device with
HOLDA, till it is ready to transmit again (its transmit buffer is not
empty), at which point if the sata device is still transmitting HOLDs,
it will transition to transmitting HOLDA.
So #1 should read as
When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame
to a SATA physical link, it shall transmit no more than 19 dwords (
including data dwords, deletable primitives and SATA_EOF) after
receiving SATA_HOLD and before responding with SATA_HOLDA."
As for #2, I thought #2 was intented for the STP link and not the sata
physical link. And on STP link, we cannot say dwords because dwords
include rate matching aligns and clk_skew aligns too. Then in a 6G link
rate matched to 1.5G connection rate (3 rate matching ALIGNs for every
other dword), we can only transmit 6 data dwords (36 dwords/4 = 9 - 2
(clkskew aligns) - 1 SATA_EOF = 6 data dwords) after receiving HOLD.
So I believe for STP links, we need to be careful when we say dwords and
not specify what kind of dwords.
Kishore
________________________________
From: Day, Brian [mailto:Brian.Day@lsi.com] 
Sent: Friday, October 03, 2008 9:09 AM
To: Karthikeyan, Kishore K; Elliott, Robert (Server Storage);
t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Re #2... why do we need to treat SATA_EOF and data dwords as special?
The transmit requirement in #1 isn't specific on what type of dword.
Seems to me the real intent was generic "dwords", regardless of what
they are.
Why not:
a) 24 dwords at the 1.5 Gbps connection rate;
b) 28 dwords at the 3 Gbps connection rate; or
c) 36 dwords at the 6 Gbps connection rate,
and shall expect to receive SATA_HOLDA within that number of dwords."
(Plus changing "data dwords" to just "dwords" for the remainder of this
section relative to any flow control)
Brian Day
LSI Corp.
________________________________
From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of
Karthikeyan, Kishore K
Sent: Thursday, October 02, 2008 6:58 PM
To: Elliott, Robert (Server Storage); t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
Hi Rob
The changes that you intend to make in the text look good
Thanks
Kishore
________________________________
From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of Elliott,
Robert (Server Storage)
Sent: Thursday, September 25, 2008 1:03 PM
To: t10@t10.org
Subject: RE: Potentially wrong assumption of SATA spec in SAS standard
I don't think any expanders try to push the limits here, but agree the
discrepency should be corrected.
1. I will change SAS-2 rules about transmission on the SATA physical
link to:
"When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame
to a SATA physical link, it shall transmit no more than 19 dwords (e.g.,
including data dwords, deletable primitives, SATA_HOLD, and SATA_EOF)
after receiving SATA_HOLD before responding with SATA_HOLDA."
2. Although the budget for STP should not include deletable primitives
or transmit-direction SATA_HOLDs (there's no reason to put those in the
buffer), it should include both data dwords and SATA_EOF.  I'll add "or
SATA_EOFs" in several places, like this:
"After transmitting SATA_HOLD, it shall accept at least the following
number of data dwords or SATA_EOFs for the SATA frame into its STP flow
control buffer:
a) 24 data dwords or SATA_EOFs at the 1.5 Gbps connection rate;
b) 28 data dwords or SATA_EOFs at the 3 Gbps connection rate; or
c) 36 data dwords or SATA_EOFs at the 6 Gbps connection rate,
and shall expect to receive SATA_HOLDA within that number of data dwords
or SATA_EOFs."
________________________________
From: owner-t10@t10.org [mailto:owner-t10@t10.org] On Behalf Of
Karthikeyan, Kishore K
Sent: Monday, September 22, 2008 9:19 AM
To: t10@t10.org
Subject: Potentially wrong assumption of SATA spec in SAS standard
All
In Section 7.18.2 STP Flow Control, it talks about 20 DATA dwords when
referring to frame transmission and reception on the SATA physical link
of the STP/SATA bridge.
But SATA spec does not say "data dwords" but simply "dwords" which means
it includes primitives too (for eg ALIGNs and EOF) essentially reducing
the number of data dwords that can be transmitted after reception of
HOLD and before transmission of HOLDA and some SATA disk drive vendors
have interpreted the SATA spec as such. Hence these drives cannot accept
more than 20 dwords (including primitives like ALIGNs and EOF) after it
has transmitted HOLD. Such SATA drives will have incompatibility with
expanders whose STP/SATA bride is designed according to SAS standard
(which says 20 DATA dwords instead of just 20 dwords). For eg. if the
expander is forwarding a frame to the SATA drive and it receives a HOLD
from this drive when only 18 dwords were remaining to be transmitted, it
can potentially just transmit all the remaining 18 dwords + 2 CLKSKEW
ALIGNs + CRC + EOF = 22 dwords and not transmit HOLDA.
SAS standard can assume data dwords on STP phys because it is talking to
a SAS device but when referring to traffic on SATA physical links; it
has to match whatever is mentioned in the SATA specs to be compatible
with existing SATA drives in the market.
I would think that this needs to be fixed to prevent incompatiblity
between STP/SATA bridge designs in expanders and SATA drives in market.
Given below is the text from 7.18.2 in SAS Standard rev sas2r14d
When a SATA host phy in an STP/SATA bridge is receiving a SATA frame
from a SATA physical link, it shall
transmit a SATA_HOLD when it is only capable of receiving 21 more data
dwords. It shall stop transmitting
SATA_HOLD (e.g., return to transmitting SATA_R_IP) when it is capable of
receiving at least 21 more data
dwords.
NOTE 83 - SATA requires that frame transmission cease and SATA_HOLDA be
transmitted within 20 data
dwords of receiving SATA_HOLD. Since the SATA physical link has non-zero
propagation time, one dword of
margin is included.
When a SATA host phy in an STP/SATA bridge is transmitting a SATA frame
to a SATA physical link, it shall
transmit no more than 19 data dwords after receiving SATA_HOLD.
NOTE 84 - SATA assumes that once a SATA_HOLD is transmitted, frame
transmission ceases and
SATA_HOLDA arrives within 20 dwords. Since the SATA physical link has
non-zero propagation time, one
dword of margin is included.
Given below is the text from the SATA 2.6 spec
9.4.7 Flow Control Signaling Latency
In the case where the receiver wants to flow control the incoming data,
it transmits HOLDP
characters on the back channel. Some number of received Dwords later,
valid data ceases, and
HOLDAP characters are received. The larger the latency between
transmitting HOLDP until
receiving HOLDAP, the larger the receive FIFO needs to be. Within a
single HOLDP/ HOLDAP
sequence, the maximum allowed latency from the time the MSB of the
initial HOLDP is on the
wire, until the MSB of the initial HOLDAP is on the wire shall be no
more than 20 Dword times.
The LSB is transmitted first. A receiver shall be able to accommodate
reception of 20 Dwords of
additional data after the time it transmits the HOLDP flow control
character to the transmitter, and
the transmitter shall respond with a HOLDAP in response to receiving a
HOLDP within 20 Dword
times. The 20 Dword latency specification is not applicable to any
subsequent transmissions of
the HOLDP flow control character within the same sequence. Upon each new
instantiation of a
HOLDP/ HOLDAP sequence, the receiver and transmitter shall meet the 20
Dword latency
specification