UAT is different to PROD – Who cares?

I’ve lost track over the years of how many times I’ve said to a client – ‘if you don’t have you UAT infrastructure the same as your Production infrastructure, you are essentially integration testing in Production!’.

Yet, businesses large and small continue to plough all of their budget and resources into the production environment and the UAT, system testing and development environments are viewed as poor second cousins. The ‘pre-prod’ environments are cobbled together with bits of string and sticky-back-plastic – not to mention the plastic egg cartons!

There is a very good reason for ensuring that your environments all have the same hardware, sofware, build, o/s versions, database versions, etc… because if they don’t, then you have done ZERO infrastructure testing before you put your new application into production.

I saw a good example of where this can go horribly wrong about 10 years ago. In an effort to improve system performance on a trading platform, I was involved in a performance tuning exercise based around a Sybase database and we were looking at all the whizzy features to improve performance that everyone normally leaves well alone. We identified a few features, tested them out in development, system testing and eventually promoted them to UAT. All was going swimmingly and performance had improved by around 15%-20%.

When applied our changes to the production environment, it all went south. Performance on the trading database was around 400% worse! We had to back out our changes during the online day – carnage and red faces everywhere…

It turned out, that the UAT and PROD environments were identical – almost. We had changed the way the database did it’s scanning and read-ahead on the disk and the PROD environment was using a different RAID array from a different manufacturer than all the other environments. The RAID setup was otherwise the same, but the underlying hardware was different and reacted badly to our changes.

If you don’t have the luxury of a seperate integration environment, and your UAT infrastructure is different to your Production infrastructure, then you should care – very much…..