FlowControl - Must-have or Nice-to-have

You can find multiple posts on the internet regarding HP P4000 (formerly Lefthand) and flow control. 1/3 says flow control is a must-have, 1/3 says it's a nice-to-have and 1/3 tells you not to use it at all.

So what option do you choose?

I can't give you a simple answer but I can write down some facts I collected during the last few weeks and also some of my own experience.

First of all HP itself recommends turning on flow control on all switches you use in combination with P4000 systems. You can find this information in their best practice guide. Nevertheless, some of the users talk about performance degradion with flow control enabled and therefore disabled it completely.

You have to see flow-control always in combination with another feature, modern switches offer to you: jumbo frames. Jumbo frames are used to maximize the throughput of the switches by building larger frames with more data inside thus reducing the managament data overhead.

Modern switches offer you both techniques but keep in mind, using both features the same time is not supported or recommended on many switch models. Why? Well, both flow-control and jumbo frames put high load on the switch hardware. The switch has to manage both features and sometimes, the swicht is not fast enough to handle. This can cause packet drops and can slow down the iSCSI traffic.

To avoid packet drops your switch needs cache. To be more exactly, it need buffer cache per port where packets can be stored until the switch can forward them to their destination. Many switches out there which are used for building an iSCSI SAN are not designed to do that kind of job. They suffer the buffer cache. They offer jumbo frames and flow-control but either they forbid to enable both features simultaneously (which would be the safe way) or they allow you to activate both features but won't be able to handle the traffic thus dropping packets all the time.

So, if you think about enabling flow-control first ask yourself "will I also enable jumbo frames?". If yes, check your switch hardware if it provides buffer cache. The amount of buffer cache is also important. It is recommended to have at least 0,5MB/port. That is, if you have a 24port iSCSI switch, make sure you have 12MB buffer cache or reduce the amount of used ports.

If you will only use flow control, the buffer cache isn't that important. Nevertheless, your switch should have at least 2-6MB of buffer cache. This is only provided by midrange or enterprise class switches but do you really want to build your central storage SAN on SOHO-class switches?

To sepak from my own experience, when using P4000 systems with HP ProCurve switches, you should ALWAYS enable flow-control. On the one hand, you have to enable it if you want to use MPIO for the P4000, on the other hand you will get massive packet drops if you leave it disabled. We have a customer where we buit up a P4000 iSCSI SAN. The SAN consists of 2x P4300, 2x P4500 and two ProCurve 2910al switches. The two switches were trunked (Procurve trunk, not LACP) with two connections. First we decided to disable flow-control on all ports, all P4000 nics and on the application servers. After a short time we had several hundreds of packet drops on all used ports. Referred to HP, a drop rate from 1 out of 5000 packets is okay but you should never accept a setup where you get 10.000 drops per day.

So we switched to flow-control enabled on all ports and nics - no more drops. We can not say that the storage performs better now the drops are gone but it gives you a much better feeling...

By the way, the instructions on how to enable flow-control on HP ProCurve switches (especially on the 2910al) that are mentioned in the above mentioned best practice guide from HP is a bit ambiguous.

Here is the right syntax for enabling flow-control:


set interface x-x flow-control

Make sure, all your ports are on auto-sense and also the attached nics. Flow control has to be enabled on ALL components, that is the switch itself and all attached NICs. You can check if flow-control is active with the command:

show interface brief

on the switch or with the corresponding management application on your host/storage. 

Add comment

Security code