19th August, 2016
When considering installing a Patch, Hotfix or Security update, its always the best plan to install the fix into a UAT or DEV environment first. Sadly, its often the case that you or your clients do not have UAT and DEV environments available and so the patch can be overlooked due to the scare factor of installing directly on to a production environment. So, what can be done to alleviate those concerns?
In an ideal world, a test bed can be provided. Having a development installation in house is usually preferable, but not always cost effective so asking the question to your support provider is a good starting point. If your support provider has the capacity to provide a temporary solution, then its always safer to implement patches and perform testing in a development environment, rather than a production one.
If a UAT/DEV environment is still not available, then there are several things that can be done to ensure that patching has a minimal impact on production environments. The first of these is a solid Change Control Plan. Change control can ensure that any change has an owner, an audit trail, testing procedures, and a well-understood roll-back plan. These are essential to ensure that your systems can stay up and running. Equally important is ensuring that all the documentation is read and understood. It is not enough to have a single person read the documentation as this will create a single point of failure so a colleague should always review and understand what is being undertaken before any plan is implemented.
It is also important to understand if the hotfix or patch will even resolve the problem and wont cause new issues. Applying patches just because one has been released is not necessarily a good reason to do it. There is an argument that you should only apply a patch when it will actually fix an issue. However, the bug or issue that the security patch/hotfix is designed for may be extremely rare and may never crop up. But, it is certainly worth doing it if the risk of that bug/issue is your system going down, even if this causes some small operational issues going forward.
Before even considering a patch or hotfix, BACK UP YOUR SYSTEM. This should be a mandatory for any business, irrespective of patching. Regular backups that are tested for consistency and recoverability are essential. If it takes up a day, add it to your plan. The consequences of not having one available are unthinkable.
So where ever possible, be it on your own systems or your support providers – Test, test and test again. If you have done all of the above well, you should hit the ground running during your testing phase without issues. However, as with any system be it the latest database or a 15 year old system thats just plodding along, you cant plan for everything. Your testing needs to be stringent and thorough. Test it once, then test it again. Then try a different approach, and then try it again. Get different people involved so all bases are covered.
Once your testing is complete and you are happy there has been no impact, be a good colleague and let your service desk and users know. It doesnt matter how good you are at Systems/Databases/Networks/etc. the chances are someone will find something that didnt come up in testing. Hopefully they wont find anything, but who wants to take the backlash from them finding an issue when you didnt let them know?
As a final thought, a good rule of thumb is to remember this quote from Microsoft: The risk of implementing the service pack, hotfix and security patch should ALWAYS be LESS than the risk of not implementing it.
RDB Concepts can provide a temporary test bed for testing patches, hotfixes and security updates on. Whilst this will not be your hardware and your exact environment, this can help eliminate many of the hurdles that you will face whilst upgrading your systems. For more information on this, please contact us.