without knowing the details, it's likely the FAST cache was dumped to disk (cleared) during the upgrade and it will take time to promote data back into the FAST cache - could take hours or longer - that's the warm up period.

glewn

Thanks for the reply. Yes I'm aware of the "warm up" period. The data I took was about a week and half post upgrade so that should be plenty of time. I've been in contact with another customer with a much larger VNX implementation who has experienced the same problem on a larger scale. I'm hoping that EMC will acknowledge the change and at least explain what was behind making it.

Howdy,

I started a thread on the linkedin forums a while back regarding Flare 32 stability and us upgrading to that version. I had heard from EMC contacts that we should wait for several months due to some unknown patches coming out. We were anxious get to the mixed RAID group pools since 32 had been in the wild for 6 months so what was the problem I was thinking.

That thread generated quite a bit of discussion.. one key point was how a customer had seen a large drop off in Fast Cache hits especially for Writes. I then took some data prior to the upgrade and post.  I graphed the total read/write I/O vs cache hits for both datasets and the drop in hits is so obvious it hits you smack in the forehead.  That said our overall latency figures changed little of any. That may be because we're not pushing the array too hard just yet. I'm not a storage expert but logic to me says that if you're fast cache hits drop off this dramatically, more I/O is hitting spinning disk..

Thus begs the question why did EMC make this change and how should it affect me?

Hi Ron, we're about to shift our ESX data stores over from an older CX4 to a VNX5500 so our workloads are similar; just all kinds of random vmware guests doing different things.  I'm curious what kind of hit ratios you saw before and after?  More importantly, did you notice issues on the ESX/guest side (either in the data or noticed at the guest level) after the update that caused you to start looking at the statistics?

We're going down in spindle count slightly from our CX4 and were hoping between fast cache and the faster bus speeds, newer drives, etc., that we would not see a decrease in performance so your issue has peaked my interest; we're running the same flare that you updated to.

In R32 they changed the way the cache works for certain types of IO. This change is a good thing and should improve FAST cache performance by reducing the sequential IO from being promoted to cache and then never used.

In 31 and prior sequential small blocks were ending up in FAST Cache and were a huge part of the higher demand on it. We don't want to go to FAST Cache for such IO as SP Cache can deal with it much better and destage to disks much more efficiently

This information is found in the Release notes for FAST Cache for Version 05.32.000.5.006 – this is the ONLY release notes for FAST Cache for R32.

https://support.emc.com/docu42103_FAST_Cache_for_VNX_OE_for_Block_Release_Notes_Version_05.32.000.5.006.pdf?language=en_US

On page 3 you can find this change from R31:

Avoidance of sequential small block I/O

This capability reduces performance degradation caused by small block sequential I/O.

In the latest version of the FAST cache White Paper - White Paper EMC FAST Cache — A Detailed Review.pdf (Aug 2012) on page 11

https://support.emc.com/docu32136_White_Paper:_VNX_FAST_Cache_-_A_Detailed_Review.pdf?language=en_US

This is also detailed in Support Solution emc251589 and emc292696 (master list for FAST cache solutions).

glen

Unfortunately I started out with R32 so I don't have a basis of comparison but I can at least confirm that large sequential writes don't appear to be affected by having Fast Cache on or off, and I'm getting a zero cache utilization when doing them with benchmarking tools and nearly the same write speed either way; so this change does at least appear to be not promoting that kind of data into the cache, as it should.  What I am seeing though is an unexplained 40% drop off in read performance when cache is enabled for long sequential reads:

Seeing unusual benchmark results w/Fast Cache and Fast VP

You might check out this thread Is Flare 32 stable? | LinkedIn

One of the people who chimed in has experienced quite a bit of pain.

Thanks for the detailed reply glen. We must have a good % of sequential I/O in our profile that is now bypassing FC which as you say should be a good thing. It was startling to see the drop in hits thus begged the question that FC wasn't working as well as in 31. I'll try to see if we have higher SP Cache utilization in 32.

Rainer,

What kinds of metrics are telling you that it works better? I'm still learning how all this magic works and would appreciate any advice on how to better use things like Analyzer to assess things.

Ron

Hi,

there were changes in FAST VP as well and I wouldn’t rule out that maybe the way the hit rate is being calculated.

All of our internal testing shows that R32 FAST Cache works better than R31.

Rainer