I’ve been working with Kevin Holman as well as spending some time working through my MP to see if we could determine the cause of the problems related to UR 3 and security monitoring. At this point, there is good news and bad news.
First, the good news:
It is related to a change that the product team made with UR3 affecting how event log filtering works in the security logs. Since Security Monitoring does A LOT of event long filtering in the security logs, it’s naturally affected, as is (from what I understand) any MP that’s doing any kind of security log monitoring with parameter filtering:
- Fixed the monitoring agent-related issue that affected formatted strings. These are now read from the provider DLLs to show a localized string
This is the crux of the issue: Due to the nature of the OS, the XML view of some logs differs from the friendly view of the same log. Here is an example as it affects Security Monitoring:
Note the 4624 XML view in the screenshot below:
Now take a look at the general view of the same event:
This has been a bit of a thorn in the flesh so to speak for SCOM administrators over the years. I know when I first started working with SCOM, I spent much more time than I’d have liked troubleshooting this issue and even filed a bug report with the Windows product team in my first year at MSFT. It was not fixed due to a considerable amount of re-writing code that would need to happen. Right or wrong, that’s what I was told. The SCOM product team recognized this problem and chose to fix it. Ultimately, I think that’s a good thing, though I will need to make some changes to Security Monitoring, as will anyone using these codes in custom MPs… As of UR3 in 2019 (even after this is fixed), the rules/monitors that they write will no longer flag on those codes but on the values in the event description.
With that out of the way, there were a couple of bugs introduced by this. One is relatively easy to fix as I have been told, while the other will take a bit more time. In terms of ETA, it’s likely that a hotfix will be released at some point, but UR4 as I understand it will hopefully have both of these addressed.
Continuing with good news, there’s more. First, the product team considers this a functional bug, which means it’s a high priority to fix.
Second, this has allowed the product team to revisit how they look at high volume logs and see if there’s a better way to handle high volume logs. This has been an ongoing issue so much so that there is general guidance out there on not doing monitoring in the security logs which of course I’ve ignored. Over the years, this hasn’t been too much of a problem for Security Monitoring, but as anyone reading this knows, Security Monitoring looks almost exclusively at the Security log, which in most environments is a high volume log. Performance, due to this, is always a concern in large environments. Hopefully, this leads to some improvements in how SCOM parses these logs allowing for better performance in general.
Now for the bad news:
SCOM 2019 with UR3 will not work with Security Monitoring without said hot fix. It’s quite possible given the nature of what was exposed that it may never work as well. I’m also going to need to go through the MP and look for any filtering I’m doing based off of those %% values and correct them. I’ve been told that a simply OR statement to look for both may not work either. I’m not quite clear on the why for this as of now, but I will likely have to do some re-writing to take this into account. I’m not sure how the final product will look (either making these overridable parameters, the OR statement, or simply publishing a 2019 UR3 and later version of this MP) until I have a better idea as to what the finished state will look like.
For now, I’d simply say that if you have to go to UR3, you should probably remove Security Monitoring until the hotfix is published.