So I rolled CU3 into our labs in the past week. Doing this on virtual machines and terminal servers with very low free space and a great distance between themselves on the network was not that fun. Regardless, I was able to patch my SCOM infrastructure. Our labs are a shared environment so I moved on to agent updates and ran into a few problems. Two agents, that I have noticed so far, could not update. The agent logs refered to the CU1 update bits and how the agent was unable to locate them. I thought that was odd. So I had to jump on the box and see what was going on. I tried to do a remote uninstall, but that failed. I tried to do a uninstall of the agent from the actual server itself, but that failed as well with some pop up box asking for the location of the momagent.msi file. I suspect something got corrupt in the registry and will now have to follow Jonathan’s blog post on how to brute force uninstall the agent from the few servers that are behaving like this.
Then, a few days after patching several agents, I checked the patch list and saw the following:
Before deploying CU3 I read Kevin’s post for guidance. I noticed he had made a comment about the patch list may not appear correctly on some patched agents:
“Note: experienced 100% success rate on the agent updates…. however, some of my agents are still reporting both the CU2 and CU3 in patchlist. I am investigating this as it should not be reporting this way.”
At the bottom of his post he did address this:
“4. Agentpatchlist information incomplete. The agent Patchlist is showing CU3, but also CU2 or CU1. The localization ENU update is not showing in patchlist. This appears to be related to the agents needing a reboot. Once they are rebooted, and a repair initiated, the patchlist column looks correct.”
I have quite a few like this, and didn’t want to have to do all of this in order to get this fixed. I verified that the .dll’s on the agent were updated and then I looked for this value that the discovery is pulling from the registy of the agent that displayed a mixed up Patch List:
The key where this information is stored: HKLM\Software\Microsoft\CurrentVersion\Installer\UserData\S-1-5-18\Products7779052F1B26F94B\Patches
The reg keys represent the different patches applied and dictate the order they appear in the patch list. If we look at the values in the key we will notice something different between those that list the correct CU3 patch and those that list the CU3 and older patches:
If the State value is 1, then this patch display name will be listed in your patch list. If the State value is 2, then it will not be listed in the patch list view.
When I followed Kevin’s advice, it did resolve the issue, but that meant that I had several servers that I would have to first reboot, then repair (basically reinstallation of the agent). In the lab that might be ok, but production may pose a bigger issue, especially if my lab patching is any indication of the percentage I will see in production. Furthermore, if the .dll files are updated on the agent, then I would rather just use PSEXEC to batch a reg change on the STATE value and then bounce the health service on that agent. This would save a lot of time for me, and a lot of outages for our mission critical applications. In the screen shot I say a repair is not necessary. This is not the “official” word from MSFT, but just my observations from my lab. I will fix the remainder of my agents modifying the registry and bouncing the health service, then let it cook for a while before I decide to use this method should this problem appear in production when we patch. I recommend you test this in your own environment before coming to any conclusions as to if this is a viable work around. If you feel comfortable with this solution, and have ensured all your workflows and monitoring are working as expected (also ensuring the .dll’s are updated), then you may have saved yourself a lot of time wasted rebooting servers and doing repairs on agents. 😉
The only caveat I have seen so far with this is that you may have a inconsistent patch list even after this because on the agent I repaired the patch list showed two CU3 patches (the one with and without the ENU Components) and on the ones that I repaird now showed just the CU3 patch without the ENU addition. If you are worried about that, just pick one you want displayed and disable the rest.