SAS Re: Routing Table Entries

Dennis Moore dmoore at ix.netcom.com
Thu Feb 27 10:41:10 PST 2003


* From the T10 Reflector (t10 at t10.org), posted by:
* "Dennis Moore" <dmoore at ix.netcom.com>
*
This is a multi-part message in MIME format.

------=_NextPart_000_0052_01C2DE4C.BDDBBAD0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Stephen,
I agree with your notion of what the route table is there for and your
contention that there only needs to be one entry per phy. I disagree
with reaching that conclusion based on the current text. The table I
included (with 31 entries) in my last email is the one most commonly
generated by the students in the SAS seminar after they read through
clause 4.6.11.3 through 4.6.11.5.=20
=20
I will work on some wording and send to Rob as to what I think needs to
be there. But I guess I would like to get agreement that there should =
in
fact only be one entry per phy, or was the intent of the original text
to generate the table with 31 enties and not 21. Does someone think 31
is the correct for some reason I don't know about? Any input would be
welcome.
dmoore

----- Original Message -----=20
From: Johnson, Stephen   B.=20
To: Elliott, Robert (Server Storage)   ;
t10 at t10.org <mailto:t10 at t10.org> =20
Sent: Thursday, February 27, 2003 8:32 AM
Subject: RE: Routing Table Entries

Denis,
=20
Your latest table is incorrect.
=20
Here are some conceptual thoughts about table routing, some of this is
obvious so bear with me.
=20
The purpose of routing tables is if an expander receives a SAS address
and it is
not a directly attached address it will know how to route it.
=20
The route table for a given phy only needs ONE entry for each phy =
behind
it.

=20
The table is nothing more than a list of ALL the PHY's in order behind
the phy in question.
=20
Count all the current phys behind it and there is your check that the
number of entries in the table is correct.
=20

In your example Initiator C345 cannot send an SMP request to Expander
4200 or 4300 until 1000's tables are programmed
with those addresses.
=20
In your example the route table for 1000:0 (PHY 0 Expander 1000) only
needs to have a total of (12+5+5) phy entries.
(one for each phy in each expander behind the phy) So, one entry for =
all
level 1 through level n PHY's.=20
=20
For expander 2100 (level 1) the first expander behind 1000:0 (level 0)
you only need 12 entries in order from 0 - B in the table
4200 the first expander behind 2100 (level 2) only needs 5 entries in
the table.
4300 the second expander behind 2100 (level 2) only needs 5 entries in
the table.
=20
Same applies for the 2100 routes tables. It only needs a total of 5
entries for each route table phy.
=20
Steve
=20
=20
-----Original Message-----
From: Elliott, Robert (Server Storage) [mailto:Elliott at hp.com]
Sent: Thursday, February 27, 2003 8:42 AM
To: t10 at t10.org
Subject: RE: Routing Table Entries


We obviously need to work on these rules and I welcome wording
suggestions.  I'd also be willing to put your example in the standard =
if
you find it more helpful than the ones already there.
=20
=20
1. The SAS address of the device directly attached to the phy (the 2100
in your example) in question is NOT put in the routing table.  At one
time we discussed having each phy own a routing table with at least one
entry, where the first entry would contain that directly-attached SAS
address.  However, we chose to say that the directly-attached SAS
addresses are outside of the routing tables visible via SMP and are =
just
automatically known to the expander.
=20
In this wording:
"For purposes of configuring the expander route table, the edge =
expander
devices attached to the expander
phy are assigned levels:
1) the attached edge expander device is considered level 1;
2) devices attached to it are considered level 2;
3) devices attached to level 2 edge expander devices are considered
level 3; and
4) etc.
=20
The expander route table shall be configured starting from expander
route index 0 by level (e.g., all level 1
entries first, then all level 2 entries, then all level 3 entries, =
etc.)
up to the value of the EXPANDER ROUTE
INDEXES field reported by the SMP REPORT GENERAL function (see
10.3.1.2).
=20
Assuming the attached edge expander devices each have N phys, the first
N entries shall be the SAS
addresses of the devices attached to the attached edge expander device,
ordered from expander phy 0
through expander phy N."
=20
The last paragraph doesn't say anything about putting the attached edge
expander device's address in the table; it jumps right to the addresses
of the devices attached to the attached edge expander device.
=20
=20
2. You are correct that address 1000 entries are disabled.  If that
address reaches the fanout expander, it shouldn't match in the phy 0
routing table and also match its own address.
=20
=20
3. The discover process does not probe the end devices; only expanders.
So, the last entry is #21 (probing the last phy in the expander with
address 4300).
=20
This wording intends to communicate that:
"For each of the level 2 devices that:
a) is an edge expander device with M expander phys; and
b) is attached to an expander phy in the level 1 edge expander device
with the table routing attribute,
the next M entries shall be the SAS addresses of the devices (level 3)
attached to that level 2 edge expander
device.
=20

This process shall repeat for all levels of edge expander devices in =
the
edge expander device set.
"

Case a) fails when probing after #21.

4. We don't seem to have a rule anywhere that states when no address is
attached to one of the remote phys, that entry in the local table is
still there but marked disabled (entries 7, 8, 9 in your example).
Changing:


Assuming the attached edge expander devices each have N phys, the first
N entries shall be the SAS
addresses of the devices attached to the attached edge expander device,
ordered from expander phy 0
through expander phy N.

to something like:

 Assuming the attached edge expander devices each have N phys, the =
first
N entries shall be used for expander phy 0 through expander phy N. If
the corresponding expander phy is attached to a device, the expander
route entry shall contain the SAS address of the device and be enabled.
If the corresponding expander phy is not attached, the expander route
entry shall be disabled.

(all the uses of "device" are suspect here and other cleanups are
pending)

--=20
Rob Elliott, elliott at hp.com=20
Hewlett-Packard Industry Standard Server Storage Advanced Technology=20
https://ecardfile.com/id/RobElliott
<https://ecardfile.com/id/RobElliott> =20


-----Original Message-----
From: Dennis Moore [mailto:dmoore at ix.netcom.com]=20
Sent: Wednesday, February 26, 2003 6:56 PM
To: Johnson, Stephen B.; t10 at t10.org
Subject: Re: Routing Table Entries


Stephen,
There are a few more differences between your table and my
interpretation. I will omit the 2100 entry for the moment and address
that later. I believe you left out several entries in your table. I
think the table should be as follows where E=3Denabled and =
D=3Ddisabled:
=20

0	 1000	 D=09
1	 1000	 D=09
2	 1000	 D=09
3	 1000	 D=09
4	 A123	 E=09
5	 4200	 E=09
6	 4200	 E=09
7	 xxxx	 D=09
8	 xxxx	 D=09
9	 xxxx	 D=09
10	 4300	 E=09
11	 4300	 E=09
12	 2100	 D=09
13	 2100	 D=09
14	 6001	 E=09
15	 6002	 E=09
16	 6003	 E=09
17	 2100	 D=09
18	 2100	 D=09
19	 6001	 E=09
20	 6002	 E=09
21	 6003	 E=09
22	 2100	 D=09
23	 2100	 D=09
24	 7001	 E=09
25	 7002	 E=09
26	 7003	 E=09
27	 2100	 D=09
28	 2100	 D=09
29	 7001	 E=09
30	 7002	 E=09
31	 7003	 E=09
=20
Let me know what you think,
dmoore
=20

----- Original Message -----=20
From: Johnson,   Stephen B.=20
To: Dennis Moore   ; t10 at t10.org
<mailto:t10 at t10.org> =20
Sent: Tuesday, February 25, 2003 5:38 PM
Subject: RE: Routing Table Entries


Denis,
=20
I'll take a stab at this for you.
=20
Attached (xls spread sheet) are my results for discovery and for each =
of
the expander route tables starting from Initiator C345.
=20
I like to build up an ordered table All the PHY's in the topology (see
attachment).=20
The table is built in the order of discovery where everything is =
scanned
by PHY identifier number from 0-n.
Once you have the table it is easy to determine each of the Route Table
entries by looking for
each of the PHY's that have the attribute Table Routing and making each
PHY behind the connected PHY an entry in the table.
=20
The actual algorithm you use for determining the tables is up you. I
have verified the recursive one in the back
of the spec and have also created a few others.
=20
Since you didn't say I assumed some PHY's are subtractive. The =
initiator
needs to
get this info about each PHY using the SMP Discover response.
=20
I haven't really thought too much about if the PHY's are enabled or not
but,
So someone please correct me if I'm wrong here.
=20
I consider all PHY's that are attached to something to have one of the
"enabled" values (0,2,3,8, or 9) in table 154 of the SAS-r03d spec.
=20
So, for PHY 0 of the Fanout I get the following Route Table: ( note: =
all
phy's are enabled but 7,8,9 )
=20


0
1000

1
1000

2
1000

3
1000

4
A123

5
4200

6
4200

7
...

8
...

9
...

10
4300

11
4300

12
2100

13
2100

14
6001

15
6002

16
6003

17
2100

18
2100

19
7001

20
7002

21
7003

=20
=20
If you start from another initiator the order of my "All PHY's Table" =
is
different but the Route Tables are exactly the same.
=20
Hope this helps.



Steve Johnson=20
LSI Logic=20
719 533 7511=20
sjohnson at lsil.com=20

-----Original Message-----
From: Dennis Moore [mailto:dmoore at ix.netcom.com]
Sent: Tuesday, February 25, 2003 9:31 AM
To: t10 at t10.org
Subject: SAS: Routing Table Entries


Rob,
I think I reject your reject of my comments on the expander routing
table stuff. I have conducted four SAS Seminars at four different
companies and have yet to get agreement among even one of groups on =
what
the contents of the routing table should be. I do an exercise during
class where I ask the students to construct the routing table entries
for Phy 0 of the fanout expander in the domain shown in the attached
file. Even after an hour of discussion I have not been able to get a
consensus on what it should look like. So, can you and anyone else that
thinks they know please send me what you think should be the routing
table entries for the attached domain drawing fanout expander Phy 0. I
ask the students for a table that gives the index, the SAS address, and
the Enable/Disable bit, and it looks something like this:
=20
Index    Phy 0     Enable/Disable
   0       2100           D
   1       1000           D
   2       1000           D
   3       1000           D
   4       1000           D
   5       A123          E
   6       4200           E
etc.
=20
I would very much appreciate anyone's input on what all the entries for
Phy 0 on the fanout expander should be.
Thanks,
dmoore
  =20
=20


------=_NextPart_000_0052_01C2DE4C.BDDBBAD0
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">

Message Stephen,
 I agree with your notion of what the = route table is=20 there for and your contention that there only needs to be one entry per = phy. I=20 disagree with reaching that conclusion based on the current text. The = table I=20 included (with 31 entries) in my last email is the one most commonly = generated=20 by the students in the SAS seminar after they read through clause = 4.6.11.3=20 through 4.6.11.5. 
 
 I will work on some wording and send = to Rob as to=20 what I think needs to be there. But I guess I would like to get = agreement that=20 there should in fact only be one entry per phy, or was the intent of = the=20 original text to generate the table with 31 enties and not 21. Does = someone=20 think 31 is the correct for some reason I don't know about? Any input = would be=20 welcome.
 dmoore
 ----- Original Message ----- 
black">From:=20 Johnson, Stephen=20 B. 
To: Elliott, Robert (Server Storage) ; = t10 at t10.org = 
Sent: Thursday, February 27, = 2003 8:32=20 AM
 Subject: RE: Routing Table = Entries
 

Denis,
  
 Your = latest table is=20 incorrect.
   Here are = some=20 conceptual thoughts about table routing, some of this is obvious so bear = with=20 me.
  
 The purpose of routing tables is if an = expander=20 receives a SAS address and it is
 not a directly attached address it will know = how to route=20 it.
  
 The route table for a given phy only=20 needs ONE entry for each phy behind=20 it.
  
 The = table is=20 nothing more than a list of ALL the PHY's in order behind the = phy in=20 question.
  
 Count all the current phys behind it and = there is your=20 check that the number of entries in the table is = correct.
  
 In your example = Initiator C345=20 cannot send an SMP request to Expander 4200 or 4300 until 1000's = tables are=20 programmed

 with those addresses.
  
 In your example the route table for 1000:0 = (PHY=20 0 Expander 1000) only needs to have a total of (12+5+5) phy=20 entries.
 (one for each phy in each expander = behind the=20 phy) So, one entry for all level = 1 through=20 level n PHY's.=20  
 For expander 2100 (level 1) the first = expander behind=20 1000:0 (level 0) you only need = 12 entries in=20 order from 0 - B in the table
 4200 the first expander behind 2100 (level = 2) only needs=20 5 entries in the table.
 4300 the second expander behind 2100 (level = 2) only needs=20 5 entries in the table.
  
 Same applies for the 2100 routes tables. It = only needs a=20 total of 5 entries for each route table phy.
 Unicode"=20 color=3D#000080 size=3D2> 
 Unicode"=20 color=3D#000080 size=3D2>Steve
 Unicode"=20 color=3D#000080 size=3D2> 
  

 -----Original Message-----
From: Elliott, Robert = (Server=20 Storage) [mailto:Elliott at hp.com]
Sent: Thursday, February = 27, 2003=20 8:42 AM
To: t10 at t10.org
Subject: RE: Routing = Table=20 Entries


 We=20 obviously need to work on these rules and I welcome wording = suggestions. =20 I'd also be willing to put your example in the standard if you find = it more=20 helpful than the ones already there.
  
  
 1.=20 The SAS address of the device directly attached to the phy (the 2100 = in your=20 example) in question is NOT put in the routing table.  At one = time we=20 discussed having each phy own a routing table with at least one = entry, where=20 the first entry would contain that directly-attached SAS = address. =20 However, we chose to say that the directly-attached SAS addresses are = outside=20 of the routing tables visible via SMP and are just automatically = known to the=20 expander.
  
 In=20 this wording:
 "For purposes = of=20 configuring the expander route table, the edge expander devices = attached to=20 the expander
 phy are assigned = levels:
 1) the attached = edge expander=20 device is considered level 1;
 2) devices = attached to it are=20 considered level 2;
 3) devices = attached to level 2=20 edge expander devices are considered level 3; and
 4) = etc.
  
 The expander = route table shall=20 be configured starting from expander route index 0 by level (e.g., = all level=20 1
 entries first, = then all level 2=20 entries, then all level 3 entries, etc.) up to the value of the = EXPANDER = ROUTE
 INDEXES = field reported by = the SMP REPORT=20 GENERAL function (see 10.3.1.2).
  
 Assuming the = attached edge=20 expander devices each have N phys, the first N entries shall be the=20 SAS
 addresses of the = devices=20 attached to the attached edge expander device, ordered from expander = phy=20 0
 through = expander phy=20 N."
   The=20 last paragraph doesn't say anything about putting the attached edge = expander=20 device's address in the table; it jumps right to the addresses of the = devices=20 attached to the attached edge expander device.
  
  
 2.=20 You are correct that address 1000 entries are disabled.  If that = address=20 reaches the fanout expander, it shouldn't match in the phy 0 routing = table and=20 also match its own address.
  
  
 3. The discover process does not = probe the end=20 devices; only expanders.  So, the last entry is #21 (probing the = last phy=20 in the expander with address 4300).
  
 This=20 wording intends to communicate that:
 "For each of the level 2 = devices=20 that:
 a) is an edge expander = device with M=20 expander phys; and
 b) is attached to an = expander phy in=20 the level 1 edge expander device with the table routing=20 attribute,
 the next M entries = shall be the SAS=20 addresses of the devices (level 3) attached to that level 2 edge=20 expander
 device.
  
 This process shall repeat for all levels of edge = expander=20 devices in the edge expander device set.
 "
 Case = a) fails when=20 probing after #21. 4. We don't seem to have a rule = anywhere that=20 states when no address is attached to one of the remote phys, that = entry in=20 the local table is still there but marked disabled (entries 7, 8, 9 = in your=20 example).  Changing: Assuming the attached=20 edge expander devices each have N phys, the first N entries shall be = the=20 SAS
 addresses of the devices=20 attached to the attached edge expander device, ordered from expander = phy=20 0
 through = expander phy=20 N.
 to=20 something like:  Assuming the attached edge expander = devices each=20 have N phys, the first N entries shall be used for = expander phy 0 through expander phy N. If the corresponding = expander phy=20 is attached to a device, the expander route entry shall contain the = SAS=20 address of the device and be enabled. If the corresponding expander = phy is not=20 attached, the expander route entry shall be=20 = disabled.
 (all the uses of "device" are suspect here = and other=20 cleanups are pending) --=20 
Rob Elliott,=20 elliott at hp.com 
Hewlett-Packard Industry Standard Server Storage Advanced=20 Technology 
https://ecardfile.com/id/Ro= bElliott=20 

-----Original Message-----
From: = Dennis Moore=20 [mailto:dmoore at ix.netcom.com] 
Sent: Wednesday, February = 26, 2003=20 6:56 PM
To: Johnson, Stephen B.; = t10 at t10.org
Subject:=20 Re: Routing Table Entries


 Stephen,
 There are a few more differences = between your=20 table and my interpretation. I will omit the 2100 entry for the = moment and=20 address that later. I believe you left out several entries in your = table. I=20 think the table should be as follows where E=3Denabled and=20 D=3Ddisabled:
  
 0 1000 D 1 1000 D 2 1000 D 3 1000 D 4 A123 E 5 4200 E 6 4200 E 7 xxxx D 8 xxxx D 9 xxxx D 10 4300 E 11 4300 E 12 2100 D 13 2100 D 14 6001 E 15 6002 E 16 6003 E 17 2100 D 18 2100 D 19 6001 E 20 6002 E 21 6003 E 22 2100 D 23 2100 D 24 7001 E 25 7002 E 26 7003 E 27 2100 D 28 2100 D 29 7001 E 30 7002 E 31 7003 E
  
 Let me know what you think,
 dmoore
  
 ----- Original Message ----- = 
Johnson,=20 Stephen B. 
To: Dennis Moore ; t10 at t10.org 
Sent: Tuesday, February = 25, 2003 5:38=20 PM
 Subject: RE: Routing Table = Entries
 

Denis,
  
 I'll take a stab at this for = you.
  
 Attached (xls spread sheet) are my = results for=20 discovery and for each of the expander route tables starting from = Initiator C345.
  
 I like to build up an ordered = table All the=20 PHY's in the topology (see attachment). 
The table is built in the order of = discovery where=20 everything is scanned by PHY identifier number from=20 0-n.
 Once you have the table it is easy = to determine=20 each of the Route Table entries by looking = for
 each of the PHY's that have the=20 attribute Table Routing and making each PHY behind the = connected PHY=20 an entry in the table.
  
 The actual algorithm you use for = determining the=20 tables is up you. I have verified the recursive one in the=20 back
 of the spec and have also created a=20 few others.
  
 Since you didn't say I assumed some = PHY's are=20 subtractive. The initiator needs to
 get this info about each PHY using the = SMP Discover=20 response.
  
 I haven't really thought too much about = if the PHY's=20 are enabled or not but,
 So someone please correct me if I'm = wrong=20 here.
  
 I consider all PHY's that are attached = to something=20 to have one of the
 "enabled" values (0,2,3,8, or 9) in = table 154 of the=20 SAS-r03d spec.
  
 So, for PHY 0 of the Fanout I get the = following Route=20 Table: ( note: all phy's are enabled but 7,8,9 = )
  
 0
 1000
 1
 1000
 2
= 1000
 3
 1000
 4
 A123
 5
 4200
 6
 4200
 7
 ...
 8
 ...
 9
 ...
 10
 4300
 11
 4300
 12
 2100
 13
 2100
 14
 6001
 15
 6002
 16
 6003
 17
 2100
 18
 2100
 19
 7001
 20
 7002
 21
 7003

=  
  
 If you start from another initiator the = order of my=20 "All PHY's Table" is different but the Route Tables = are exactly the=20 same.
  
 Hope this helps.
 
Steve Johnson 
LSI Logic 
719 533 = 7511=20 
sjohnson at lsil.com=20 
-----Original Message-----
From: Dennis Moore=20 [mailto:dmoore at ix.netcom.com]
Sent: Tuesday, February = 25, 2003=20 9:31 AM
To: t10 at t10.org
Subject: SAS: Routing = Table=20 Entries


 Rob,
 I think I reject your reject of = my comments=20 on the expander routing table stuff. I have conducted four SAS = Seminars at=20 four different companies and have yet to get agreement among even = one of groups on what the contents of the routing table = should be. I=20 do an exercise during class where I ask the students to construct = the=20 routing table entries for Phy 0 of the fanout expander in the = domain shown=20 in the attached file. Even after an hour of discussion I have not = been=20 able to get a consensus on what it should look like. So, can you = and=20 anyone else that thinks they know please send me what you think = should be=20 the routing table entries for the attached domain drawing fanout = expander=20 Phy 0. I ask the students for a table that gives the index, the = SAS=20 address, and the Enable/Disable bit, and it looks = something like=20 this:
  
 Index    Phy=20 0     Enable/Disable
   =20 0      =20 2100          =20 D
   =20 1      =20 1000          =20 D
   =20 2      =20 1000          =20 D
   =20 3      =20 1000          =20 D
   =20 4      =20 1000          =20 D
   =20 5      =20 A123          = E
   =20 6      =20 4200          =20 E
 etc.
  
 I would very much appreciate = anyone's input=20 on what all the entries for Phy 0 on the fanout expander should=20 be.
 Thanks,
 dmoore
  
  


------=_NextPart_000_0052_01C2DE4C.BDDBBAD0--




More information about the T10 mailing list