Clarification of SAS 1.1 Sections 7.12.3 and 126.96.36.199
Jarrod A. Webb
jwebb at dalsemi.com
Fri Sep 8 11:16:55 PDT 2006
* From the T10 Reflector (t10 at t10.org), posted by:
* "Jarrod A. Webb" <jwebb at dalsemi.com>
I could use some clarification for portions of section 7.12.3 and
section 188.8.131.52 in the SAS-1.1 Rev
A portion of section 7.12.3 reads as follows:
"Each SAS port and expander port shall include an Arbitration Wait Time
timer which counts the time from the moment when the port makes a
connection request until the request is accepted or rejected. The
Arbitration Wait Timer is in the port layer state machine (see8.2.2)."
Further on it then states:
"A SAS port should set the Arbitration Wait Time timer to zero when it
transmits the first OPEN address frame for the connection request ...
The expander port that receives an OPEN address frame shall set the
Arbitration Wait Time timer to the value of the incoming ARBITRATION
WAIT TIME field and start the Arbitration Wait Time timer as it
arbitrates for internal access to the outgoing expander port."
In addition, Section 184.108.40.206 has a sentence that reads as follows:
"Several of the Request Path arguments are used for arbitration. The
Arbitration Wait Time, Source SAS Address, and Connection Rate arguments
are filled in from the received OPEN address frame and are used to by
the ECM to compare Request Path requests."
So, here's my question regarding the above portions of the spec:
(1) When the Request Path arguments are being "filled in from the
received OPEN address frame", is the Arbitration Wait Time field filled
in with a static value (i.e. the AWT value at the time the Request Path
arguments are being filled in) or is the Arbitration Wait Time field a
dynamically updated field that is constantly fed to the ECM?
It seems to me that the spec is indicating static values for the AWT
field of the Request Path that only will be updated when a frame is
being forwarded on to the next port downstream. Please clarify what the
intent of the spec is in this section of the SAS standard.
My other questions are related to Arbitration Fairness, particularly
starvation issues with SATA devices connected to SAS expanders.
If a SATA device is connected to an expander that contains several other
expanders and SAS devices, it appears to the SATA device as though he's
directly talking the the host. And, from the host's perspective, he
thinks he's talking to a SAS device because of the fake SAS address that
the expander provides for the SATA port.
Let's say that it just so happens that the SAS address assigned to the
port containing the SATA device is the lowest SAS address out of all of
the other SAS devices in the system. Now, if there is a bottleneck such
that the SATA device is constantly having to compete with most of the
other SAS devices for the same resouce (i.e. phy) then, with a
sufficient amount of traffic, any requests from the SATA device will
most likely be starved due to the following reasons:
(a) lowest SAS address
(b) AWT field is zero (SATA devices don't understand what AWT is)
(c) retry priority status bit set to zero by default (SATA devices don't
understand about this field)
(d) SATA device doesn't understand about sending a BREAK and then
retrying with a higher AWT value
With this starvation scenario, an ARB (Normal) would be sent to the Link
layer from the ECM followed, most likely, by either a ARB (Waiting on
Partial), ARB (Blocked on Partial), or ARB (Waiting on Connection).
But, it's only until the traffic from ther other inherently higher
priority SAS devices (with the higher SAS addresses and AWT fields) dies
down a bit that there would even be a window where the SATA request
could get through. In a busy system this could be quite a long time for
the SATA device to wait.
I think at this point, you can see why I wanted to clarify my
understanding of the spec in my first question.
So, finally, here's my last three questions:
(2) What is the appropriate handling of SATA devices (with low SAS
addresses) in systems that contain other SAS expanders and devices with
regard to preventing starvation of the low priority SATA device that
doesn't understand about AWT and Retry Priority Status, etc?
(3) As a follow up to the last question, is there a particular layer
that is responsible for preventing this particular starvation issue
(i.e. ECM, Link, Port, etc)?
(4) Is there any other useful information/tips/recommendations that you
may have that you could provide that you feel would aid me in
understanding how some of these system level issues are supposed to
I look forward to hearing what you have to say regarding these four
jwebb at dalsemi.com
* For T10 Reflector information, send a message with
* 'info t10' (no quotes) in the message body to majordomo at t10.org
More information about the T10