Using SCOM to Capture Suspicious Process Creation Events

I recently had the privilege of chatting with Greg Cottingham on the Azure Security Center Analyst Team about process creation events and how to use them to detect anomalous events that need to be investigated.  It was a very interesting discussion and I was given a few real world examples of how the bad guys can move around in your networks.

To start with the basics, process creation is a fairly routine thing in the Windows world. If I execute a program, it can generate one or many processes which will remain open until the program completes.  This can be audited, as a process creation will generate event 4688 in the security event log.  With that said, it is disabled by default.  Server 2012 R2 added an additional feature, in that with an additional GPO setting, you can audit the command line that was executed.  So in short, you need to turn some things on to capture this functionality.

Before you run and do that though, you probably should review your environment. This will generate a lot of security events. If you have tools such as ArcSight, Splunk, OMS, or SCOM collecting these events, you’d be wise to do this incrementally to ensure that you aren’t overloading these tools, and I’d add that if you don’t have a plan in place to review and respond to what you find, then you should think about that before you start turning on auditing that won’t be looked at.  The other problem is that by turning on command line auditing, anyone that can read security events could read the contents, and potentially read something sensitive. So please, think this through carefully. A full write up on TechNet can be found here.

Turning this on is fairly easy.  A simple GPO will suffice.  Instructions are here.  In my lab, this was quite easy:

image

Enabling command line:

image

Within minutes of doing this, 4688 events start showing up in various event logs.

Next up, we need to determine what to look for.  These events contain a couple of useful parameter.  The full list can be seen in this screenshot:

image

Parameter 6 (New Process Name) and Parameter 9 (Command Line) will generally contain the items worth triggering on, but we can use SCOM to report or filter out on other things such as the user name (parameter 2).  Again, we don’t want to simply alert on 4688 events, as that will generate significant amounts of noise, but we can target this to specific events that should be investigated every time.  Here are some examples:

Applocker Bypass – an intruder uses javascript  or regsvr32 to work around applocker rules

image image

Command line FTP – an attacker uses command line to download an execute a payload.

image

Manipulating the PowerShell Window position – an attacker executes a powershell command, but manipulates the window position as such so that the logged on user cannot see it.

image

Command Execution in Folders with no executables – this folder list could be long when you think about it, but these are common destinations.

image

Manipulation of run key – an attacker adds a line to start a process at startup.  I’d note this is an example where you may need to add some “does not contain” options so as to avoid alerting for legitimate events.

image

Happy hunting.

I’m working on a management pack that contains these monitors along with additional security rules along with other security related items that SCOM can provide. My only test environment is my lab, which is hardly a production grade environment. I do welcome feedback. While I can only support this to the best of my ability, I can provide it if this is something you are interested in trying it out. My main goal is to keep the noise down to a minimum so that each alert is actionable. While that is not always easy to do, trying this out in other environments will go along way to getting it to that point.  If this is something you are interested in testing, please hit me up on linked in or have your TAM reach out to me internally.

Breaking apart the GPO Modification Process and Using SCOM to Detect GPO Changes – Part 2

Part 1 is here.  In it, I covered monitoring for GPO changes or the creation of new GPOs.  As I learned when working through this, deleting and unlinking GPOs were a bit more difficult to track.

For my first test, I decided to remove an old GPO from my environment, and to my surprise, there was no alert. I checked my domain controller logs for 5141 events as well, and that confirmed what SCOM was telling me:  none were generated.  I went back and checked my various audit policies, and these were also set correctly.  After some digging, I made this little discovery.  5141 events are not turned on by default, you need to add the ‘delete’ SACL to the advanced auditing tab for your security group that can delete GPOs.  This can be done at the root domain level using Active Directory Users and Computers, but if you have not explicitly turned this on, it will not work.  It must also be done against the context of users who have the rights to modify a GPO.  In my case, I assigned it to the domain administrators group, but if there are users who can modify GPOs who are not members of this group, then no 5141 will be generated. Tracking this will involving knowing which groups can make changes to GPOs.

image

Once I got this turned on, I was able to generate 5141 events when I deleted a GPO, but I was still not seeing alerts in SCOM.  As it turns out, this too is a function of the way 5141 events are generated in Windows.  The ObjectClass value that we are capturing looks for groupPolicyContainer.  The value of this parameter, however, is ‘container’.  The Windows team switched it up on us unfortunately.  This happened for two of the three 5141 events that were generated.  The third one, however, uses the correct parameter.  Looking at the XML, I see why it wasn’t collected, as it looks like 5141s are being excluded.

I’m guessing this was an oversight, likely for testing purposes, but not cleaned out of the solution when presented.  Correcting that, however, brings us back to SCOM generating an alert for a PowerShell script failing to run due to differences in the parameterization of the event.  That’s not a big deal, but for the purposes of my security management pack, I’d rather generate an alert specifically targeting GPO deletion.  Since this is a singular event, I can use the same parameters for an alert generating rule, with a description that allows me to notify the users that a GPO was removed.  It doesn’t tell you the exact GPO, but it does tell you who did it, allowing for a security monitor to confirm.

Linking and unlinking GPOs does not behave as desired.  This event generates two 5136 events, so I’d hope the correlation script gets them, but it does not.  The reason is the same as what we saw when we deleted a GPO.  The ObjectClass changes for these 5136 events.  Linking and unlinking a GPO changes that parameter to domainDNS.  As such, the script as designed does not collect that information.  That is easy enough to correct.  In this case, correcting it should not add additional noise.  I simply added an <Or> expression to collect both domainDNS as well as grouppolicyContainer for that parameter.  Now that the script collects both, a GPO change alert is generated when a GPO is linked as well as unlinked.

There does appear to be a draw back here as well:  namely the GPO Name and GUID are blank.  Turning on debug for the script, I can see why.  The object DN is not the same in the 5136 event generated by linking/unlinking as it is when a 5136 that tracks GPO change.  There’s no GUID present, just the domain name. As such, the script is failing because it “cannot index into a null array.”

image

Even with the failure, the results we get are acceptable. I can see that a changed happened and who made it.  At this point, we can reach out to the individual and confirm the change, at which point we can close the alert.

Windows Event Collector Discovery Management Pack

This is a management pack I wrote for the SCOM security MP.  If you have Windows Event Collector’s setup in your network, this will discover them and populate a class for you to monitor them.  It is worth noting that it discovers the running service.  That service is disabled by default, so I’m assuming if it is enabled that you are using it.  It does not submit blank discovery data if the service were to be stopped.  The main reason for that is that whether by accident or intentionally, someone could stop that service and you would lose visibility into it.  There are no health monitors targeted to it in this MP (that will be in the security MP).

I would note that any monitor or rule targeted towards this class will likely need to see some manual XML editing.  I have details of the specific issue at this blog article.

There isn’t much to the discovery, it’s a PowerShell script set to run once a day that looks at the wecsvc service and verifies if it is started. If started, it will populate the class.

You can find a downloadable copy of the sealed version, here.

Breaking apart the GPO Modification Process and Using SCOM to Detect GPO Changes – Part 1

A while back a colleague of mine introduce me to some work done by two other colleagues which involved using SCOM to alert against changes made to GPOs.  First, Jan Varšavský wrote the framework for SCOM 2007 which was later updated by Nathan OlmsteadAndres Naranjo took this a step further writing a composite data source that collected the key events along with step by step instructions to create a VSAE project with a nice demonstration as to how it works.  I’d love to say that I’m as smart as these guys, but in reality, they’ve done most of the heavy lifting.  A few weeks ago, I had the chance to chat with Andres about including this in the security management pack for SCOM that is currently in development. He gave his blessing to my request, but did note that there are a few issues with the rule as it is currently written. In particular, it generates multiple alerts for each change, which is something that I would personally like to dial back.

From a security perspective, I’m not sure how much the bad guys will mess with your GPOs once they are in your environment, but I’d imagine that this is not something that is potentially off limits to them.  I cannot envision many scenarios where they would outright delete a GPO, as that strikes me as the type of thing that someone might notice, but I could see them adding GPOs to deploy some of their malware or perhaps modifying an existing GPO to do the same, turn off auditing, or those types of things.  Keep in mind, that attackers have a much wider margin for error in poorly run environments, and as such have been known to be a bit more brazen with their activities.  Something like this could be beneficial, assuming of course the monitoring team has a good alert management process.

First, let’s start with the events that we are watching.  All of these come out of the Domain Controller security logs.  You will also need to have Directory Services Change Auditing configured for successes if you want this to work.

  • Event ID 5137 is fired when a new GPO is created.  That seems straight forward.
  • Event ID 5141 audits a GPO deletion.
  • Event ID 5136 audits the specific changes.

For the most part, Event ID 5136 is what we will be looking at, but as I dissected this process, it was clear that there’s a lot more to the 5136 events than I suspected at first. My first test was to change an existing GPO in my lab. As expected, I got 4 alerts. I filtered the security log on my domain controller and also confirmed there were four 5136 events.  This was in line with my expectation, so the next step was to configure suppression.  That wasn’t difficult to do as the gpoGUID was already being collected by the composite data source.  As such, I could add the following tag to the alert generating rule and drop the number of alerts from four to one.

            <Suppression>
               <SuppressionValue>$Data/Property[@Name=’gpoGUID’]$</SuppressionValue>
             </Suppression>

As is often the case, there is a draw back to this approach.  Those 5136 events are generated for each change, and as it happens in the case of my test, there were two writes and two deletes that were captured in the 5136 event. My suppression will let you know that a GPO changed, but if the action was nefarious in nature, you’re going to need to filter the security logs on the domain controller(s) in question to see what those changes were.  In my test, the alert mentioned the value add, and it still captured who made the change.  Personally, I’m fine with that draw back as I’m not a fan of using SCOM alerting as an inventory mechanism. Their purpose is to let you know that an event happened so you can investigate it.  Given that the user making the change is recorded, that can simply be verified and closed by contacting them and confirming (or if you have a good change management process in place, looking it up).

My second test did not behave as expected.  In this test, I created a new, empty GPO.  This generated 9 alerts in SCOM along with an “PowerShell script failed to run” warning as well.  After adding the suppression tag, this reduced to only one alert for the GPO change along with the PowerShell script failure alert.  Further investigation showed that on my domain controller, I observed eleven 5136 events and only one 5137.  That makes sense as logic would dictate that only one 5137 would be generated for a GPO creation. Unfortunately, this guy was also the reason for my script failure. The composite DataSource feeds parameter 15 as an input to the correlation script.  However, event ID 5137 has no parameter 15. As such, that blank value causes the script to fail when it tries to parse event 5137.  I could spend a lot of time reverse engineering the PowerShell and rewriting it, but I’m lazy at heart, and to top it off, not all that great with PS.

Instead, since 5137 generates only once, I removed it from the collection criteria and created a separate rule that looks for and alerts to the presence of a 5137 noting the creation of a GPO. In this scenario, I’ll get two alerts:  one for the creation of the GPO, and then a subsequent alert for the changes that naturally follow. It’s not ideal, but I can live with that.

Next up is Part 2, where I’ll cover detecting the deletion of a GPO as well as unlinking an existing GPO.