sorry for the long post ... here is the continuation

conclusion :
- the one-subnet setup doesn't work at all ... if the switches are not connected
- ... and only works 'sometimes' with a crossover link between the two switches

2) Two separate subnets
Although discouraged by every support person I spoke with, I tried this setup :
- in VM : two separate virtual switches (each with its own service console and vmkernel) dedicated for iscsi. One physical nic attached to each virtual switch
With this setup ... :
- I was able to use 2 different data-paths over both of my physical NICs (creating a kind of manual load balancing)
- I could not make this to fail, as long as one of the 4 iscsi-paths was available. Failover sometimes took 'a while', but at least I never lost my data connection.

So "what's the question than" .... you would probably ask :-)
- First of all : I wanted to share this info with others, since not much info about this seems to be available and almost all people I spoke with (as well Dell as VMWare support) were unable to support me with this setup.
- Next, although the problem seems to be solved, I got a couple of concerns about it :
1) 'in theory' the "one-subnet" seems to provide more redundancy (think at nic-teaming, ...), but 'in practice' however ...
2) definitely, the one-subnet setup is easier to setup and maintain
3) VMWare support explicitly told me that the two-subnet setup is officialy not supported and only provided as a 'best effort' solution.

So for that reason, I'm still looking for a way to get this 'one-subnet' setup to work in my configuration ? Or (if applicable) an official recommandation that the 'two-subnet' setup is definitely the best solution.

Thanks in advance !

Koen.

Koen,

How do you have your virtual switches setup for iSCSI in the one subnet solution? I assume that you have a single virtual switch with both physical NICs. Also, how did you test the failures? Did you simply remove network cables? I want to test this same scenario in my lab so I just wanted a few more details of what you did.

All of the best practices that I have seen about this type of setup have recommended a single subnet, with two switches and an innerswitch link. I believe that it is possible to have multiple ISLs for redundancy and increased bandwidth between the switches.

Thanks,
Todd

(sorry, posted to soon)

... and yes : you 're trough.
Everyone advices to use one subnet with ESX
I myself see some extra advantages in it :
- setup is easier
- less resources used for additional vswitch, vmkernel, service console, ...

so if my setup could be fixed to run with one subnet ... that would be great !

thanks,
Koen

Hi Todd,
thanks for "picking this up"
To answer your questions :
1) indeed: one vswitch, two nics, nic teaming with default settings
2) testing, by disconnecting utp cables, or by powering down switches
If you want : I've got a couple of nice drawings (pdf) describing my setup and the test scenario's I went trough. So I would be happy to share them with others ... problem is, that I don't know how to upload them to this site :-(
Maybe someone can advice me on this

thanks in advance for the assistance,
Koen.

I tested the MD3000i with dual controllers, attached to two Power Connect switches with ESX host attached with two physical NICs. Each of the MD3000i controllers has two ethernet ports, so each was controller was attached to both switches. There was an interconnect ISL between the two Power Connect 5448s. The ESX host was attached to both powerconnect switches - One NIC to each. These two NICs were then connected to the same virtual switch.

I tested failures by removing cables for either NIC on the server, powering off one switch at a time, and removing both cables from a storage controller on the MD3000i.

I did not get a failure at any time. I also did not have to manually rescan. So send me your PDF and I will take a look at them. (Check the message that I sent you)

Thanks,
Todd

Koen,

Thanks for sending over the detailed pictures with the specifc failure scenarios you are testing. There are two issues that I found:

1. To test more than one failed component at a time is not really fair. The probability of having multiple failed components at the same time is less than having an entire site failure. Usually testing is done to ensure no single point of failure.

2. The switches that you are using are entry level non-managed switches - the PowerConnet 2716. I would recommend that you use a higher class managed ethernet switch. (Of course it does appear from your testing that the switches do work in normal use).

Thanks,
Todd

Dell's configuration guide http://www.dell.com/downloads/global/solutions/esx_storage_deployment_guide_v1.pdf (Great Document, by the way !) gives 2 nice drawings on how to do the cabling for both possible setups. But refers to vmware's http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_iscsi_san_cfg.pdf for software configuration of it. I reread the chapter about 'setting up software iscsi initiator' (p. 50-55) in this document a hundred times, but could not find a description on how to add a second nic and/or redundant data paths
- should this second nic be added to the same vswitch or should I create an additional vswitch ?
- If using nic teaming, what parameters should be set ?
- or if I add a second vswitch : should I give it an extra vmkernel ? and an extra service console ?
- ...

Thats why I posted my questions in this group, hoping to find some extra advice. (which I 'm happy to find here at this moment :-)

2. Regarding the gigabit switch : I asked my account manager on before what kind of benefits a PC5424 switch would have, but he couldn't find any (for our particular setup). That, on top of the fact that the PC2716 is officialy supported with the MD3000i (http://support.dell.com/support/edocs/systems/md3000i/en/SUPPORTMATRIX/3000isupportmatrix.pdf), made us decide to buy two PC2716 switches.
But knowing what troubles we have with the setup now, I 'm thinking : Would my problems be solved by just changing the switches to PC5424 models ?

Once again : thanks a lot for helping me out with this.

Thanks,
Koen.

Hi Todd,

1. Sorry that I created an un-fair test-scenario. I was not aware of the fact that there were specific rules about this. I was only thinking on 'what can go wrong'. And I agree : the failover scenario I tested is not likely to happen every day but came in my mind at the moment I found out that 'that simple connection' between both switches was responsible for keeping alive 50% of my iscsi-paths (ie: 2 out of 4).
As I described it in my original post :
<
in case of a failure of the interswitch-link (which can't be monitored) a subsequent failure of nic1 or switch1 would cause a failure of the data-connection (and a manual rescan to fail-over to nic2)
>
Especialy the fact that a failure of the inter-switch-link can't be automaticly noticed, makes this (in my opinion) a risky situation.
That, on top of the two other issues I mentioned :
<
- all connections were initiated by the first nic (no load balancing over the 2 nics)
- more often than not, the most NON-optimal 'data paths' were used. I mean : [nic1-switch1-switch2-san] while it could have been [nic1-switch1-san]
>
... gave me (and still gives me) an un-comfortable feeling with this setup.

That's why I came to the 2-subnet-setup (a bit inspired by the FC-setup I 'm used to work with).
At least this setup seems to survive my worst-case-scenario's :-)
And I seem to be able to guide the network traffic over the most optimal data paths, and even trough both nics at the same time (ie: on a per-lun-basis)

Unfortunatly I couldn't find any documentation/whitepaper/recommandation/... that confirms (or objects) and arguments my experiences.
In fact : I couldn't find any document that described how to do the network-part of the iscsi-setup.

sorry (my post is to long ... again) ...

Koen,

I believe that the way to confiure the iSCSI network on the VMware side for this scenario is to create a single virtual switch and attach both physical NICs.

The PC2716 will work with the MD3000i, as you have seen in your testing. However the features that you have in terms of managment and configuration are less than what you would get with a 5424. This additional management and configuaration would most likely help you with the ISL configuration and monitoring. Trying to get into my own 5448 to give some more details about what can be done.

Todd

My understanding is that the PC2716 does not support Multi/Rapid Spanning Tree, Port fast, this would explain why you have to rescan after a link failure. Where the PC5424 has much better Spanning Tree features so can much better detect link changes.
The other advantage of a PC5424 is that you could monitor the trunk between the two switch via snmp, syslog, telnet.