Windows event log api

Overview

This integration watches for Windows Event Logs and forwards them to Datadog.

Enable this integration to:

• Track system and application events in Datadog.
• Correlate system and application events with the rest of your application.

For more information, see the Windows Event Logging documentation.

Setup

Installation

The Windows Event Log check is included in the Datadog Agent package. There is no additional installation required.

Configuration

Windows Event Logs can be collected as one or both of the following methods.

• As Datadog Events
• As Datadog Logs

Both methods are configured in win32_event_log.d/conf.yaml in the conf.d/ folder at the root of your Agent’s configuration directory. See the sample win32_event_log.d/conf.yaml for all available configuration options. For a quickstart option to send Security event logs, see Send default Security logs.

This integration also comes with an out-of-the-box Windows Event Log Overview dashboard available in-app.

List Windows Event channels

First, identify the Windows Event Log channels you want to monitor.

Depending on collection method, the channel name can be used for the following configuration parameters:

• Datadog Logs: channel_path
• Datadog Events: path
• Datadog Events (legacy): log_file

PowerShell

To see a list of channels, run the following command in PowerShell:

To see the most active channels, run the following command in PowerShell:

Get-WinEvent -ListLog * | sort RecordCount -Descending

This command displays channels in the format LogMode MaximumSizeInBytes RecordCount LogName.

Example response:

LogMode  MaximumSizeInBytes RecordCount LogName 
Circular          134217728      249896 Security
Circular            5242880        2932 <CHANNEL_2>

The value under the column LogName is the name of the channel. In the example above, the channel name is Security.

Windows Event Viewer

To find the channel name for an Event Log in the Windows Event Viewer, open the Event Log Properties window and refer to the Full Name field. In the following example, the channel name is Microsoft-Windows-Windows Defender/Operational.

Картинка с сайта: docs.datadoghq.com

logs:
  - type: windows_event
    channel_path: Security
    source: windows.events
    service: Windows

  - type: windows_event
    channel_path: "<CHANNEL_2>"
    source: windows.events
    service: myservice

init_config:
instances:
  - # Event Log API 
    path: Security
    legacy_mode: false
    filters: {}

  - path: "<CHANNEL_2>" 
    legacy_mode: false
    filters: {}

init_config:
    legacy_mode: false
instances:
  - # Event Log API
    path: Security
    filters: {}

  - path: "<CHANNEL_2>"
    filters: {}

init_config:
instances:
  - # WMI (default)
    legacy_mode: true
    log_file:
      - Security
      
  - legacy_mode: true
    log_file:
      - "<CHANNEL_2>"

Edit the <CHANNEL_2> parameters with the Windows channel name you want to collect events from.

Finally, restart the Agent.

Note: For the Security logs channel, add your Datadog Agent user to the Event Log Readers user group.

Filtering events

Configure one or more filters for the event log. A filter allows you to choose what log events you want to get into Datadog.

You can use the query, as well as the log_processing_rules regex option, to filter event logs. Datadog recommends using the query option which is faster at high rates of Windows Event Log generation and requires less CPU and memory than the log_processing_rules filters. When using the log_processing_rules filters, the Agent is forced to process and format each event, even if it will be excluded by log_processing_rules regex. With the query option, these events are not reported to the Agent.

You can use the query option to filter events with an XPATH or structured XML query. The query option can reduce the number of events that are processed by log_processing_rules and improve performance. There is an expression limit on the syntax of XPath and XML queries. For additional filtering, use log_processing_rules filters.

Datadog recommends creating and testing the query in Event Viewer’s filter editor until the events shown in Event Viewer match what you want the Agent to collect.

Картинка с сайта: docs.datadoghq.com

Then, copy and paste the query into the Agent configuration.

  # collect Critical, Warning, and Error events
  - type: windows_event
    channel_path: Application
    source: windows.events
    service: Windows       
    query: '*[System[(Level=1 or Level=2 or Level=3)]]'
      
  - type: windows_event
    channel_path: Application
    source: windows.events
    service: Windows       
    query: |
      <QueryList>
        <Query Id="0" Path="Application">
          <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>
        </Query>
      </QueryList>      

Картинка с сайта: docs.datadoghq.com

In addition to the query option, events can be further filtered with log processing rules.

Some example filters include the following:

  - type: windows_event
    channel_path: Security
    source: windows.events
    service: Windows       
    log_processing_rules:
    - type: include_at_match
      name: relevant_security_events
      pattern: '"EventID":(?:{"value":)?"(1102|4624|4625|4634|4648|4728|4732|4735|4737|4740|4755|4756)"'

  - type: windows_event
    channel_path: Security
    source: windows.events
    service: Windows       
    log_processing_rules:
    - type: exclude_at_match
      name: relevant_security_events
      pattern: '"EventID":(?:{"value":)?"(1102|4624)"'

  - type: windows_event
    channel_path: System
    source: windows.events
    service: Windows       
    log_processing_rules:
    - type: include_at_match
      name: system_errors_and_warnings
      pattern: '"level":"((?i)warning|error)"'

  - type: windows_event
    channel_path: Application
    source: windows.events
    service: Windows       
    log_processing_rules:
    - type: include_at_match
      name: application_errors_and_warnings
      pattern: '"level":"((?i)warning|error)"'

Here is an example regex pattern to only collect Windows Events Logs from a certain EventID:

logs:
  - type: windows_event
    channel_path: Security
    source: windows.event
    service: Windows
    log_processing_rules:
      - type: include_at_match
        name: include_x01
        pattern: '"EventID":(?:{"value":)?"(101|201|301)"'

Note: The pattern may vary based on the format of the logs. The Agent stream-logs subcommand can be used to view this format.

For more examples of filtering logs, see the Advanced Log Collection documentation.

Legacy events

Applies to Agent versions < 7.41

Legacy Provider EventIDs have a Qualifiers attribute that changes the format of the log, as seen in the Windows Event Schema. These events have the following XML format, visible in Windows Event Viewer:

<EventID Qualifiers="16384">3</EventID>

The following regex must be used to match these EventIDs:

logs:
  - type: windows_event
    channel_path: Security
    source: windows.event
    service: Windows
    log_processing_rules:
      - type: include_at_match
        name: include_legacy_x01
        pattern: '"EventID":(?:{"value":)?"(101|201|301)"'

Agent versions 7.41 or later normalize the EventID field. This removes the need for the substring, (?:{"value":)?, from legacy pattern as it is no longer applicable. A shorter regex pattern can be used from versions 7.41 or later as seen below:

logs:
  - type: windows_event
    channel_path: Security
    source: windows.event
    service: Windows
    log_processing_rules:
      - type: include_at_match
        name: include_x01
        pattern: '"EventID":"(101|201|301)"'

Use the Windows Event Viewer GUI to list all the event logs available for capture with this integration.

To determine the exact values, set your filters to use the following PowerShell command:

Get-WmiObject -Class Win32_NTLogEvent

For example, to see the latest event logged in the Security log file, use the following:

Get-WmiObject -Class Win32_NTLogEvent -Filter "LogFile='Security'" | select -First 1

The values listed in the output of the command can be set in win32_event_log.d/conf.yaml to capture the same kind of events.

The information given by the Get-EventLog PowerShell command or the Windows Event ViewerGUI may slightly differ from Get-WmiObject.
Double check your filters’ values with Get-WmiObject if the integration does not capture the events you set up.

Filtering events using the Event Log API (Recommended)

The configuration option using the Event Log API includes the following filters:

• path: Application, System, Setup, Security
• type: Critical, Error, Warning, Information, Success Audit, Failure Audit
• source: Any available source name
• id: event_id: Windows EventLog ID

See the sample win32_event_log.d/conf.yaml for all available filter options.

This example filter uses Event Log API method.

instances:
  - legacy_mode: false
    path: System
    filters:
      source:
      - Microsoft-Windows-Ntfs
      - Service Control Manager
      type:
      - Error
      - Warning
      - Information
      - Success Audit
      - Failure Audit
      id:
      - 7036

You can use the query option to filter events with an XPATH or structured XML query. Datadog recommends creating the query in Event Viewer’s filter editor until the events shown in Event Viewer match what you want the Datadog Agent to collect. The filters option is ignored when the query option is used.

init_config:
instances:
  # collect Critical, Warning, and Error events
  - path: Application
    legacy_mode: false
    query: '*[System[(Level=1 or Level=2 or Level=3)]]'
    
  - path: Application
    legacy_mode: false
    query: |
      <QueryList>
        <Query Id="0" Path="Application">
          <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>
        </Query>
      </QueryList>      

Filtering events using Legacy Mode (Deprecated)

The configuration option using the Legacy Mode includes the following filters:

• log_file: Application, System, Setup, Security
• type: Critical, Error, Warning, Information, Audit Success, Audit Failure
• source_name: Any available source name
• event_id: Windows EventLog ID

This example filter uses the Legacy Mode method.

instances:
  # Legacy
  # The following captures errors and warnings from SQL Server which
  # puts all events under the MSSQLSERVER source and tag them with #sqlserver.
  - tags:
      - sqlserver
    type:
      - Warning
      - Error
    log_file:
      - Application
    source_name:
      - MSSQLSERVER

  # This instance captures all system errors and tags them with #system.
  - tags:
      - system
    type:
      - Error
    log_file:
      - System

The legacy method does not support the query option. Only the Event Log API method (setting legacy_mode: false) and the Logs Tailer supports the query option.

When you’re done setting up filters, restart the Agent using the Agent Manager, or restart the service.

Validation

Logs Agent
==========

  [...]

  win32_event_log
  ---------------
    - Type: windows_event
      ChannelPath: System
      Status: OK

Checks
======

  [...]

  win32_event_log
  ---------------
      - instance #0 [OK]
      - Collected 0 metrics, 2 events & 1 service check

Send Default Security logs

Starting with Agent 7.54, you can automatically send Security Events to Datadog as logs by using the dd_security_events flag. These logs can be used with Datadog’s Cloud SIEM to automatically detect threats and suspicious activity in real-time. These default security events are compatible with Datadog’s out-of-the-box Windows detection rules to create security signals when a user clears the Security logs, disables the Windows firewall, changes the Directory Services Restore Mode (DSRM) password, and more.

1. Enable collecting logs in your datadog.yaml file. It is disabled by default in the Datadog Agent.

2. In the integration configuration file, (win32_event_log.d/conf.yaml) set the dd_security_events flag to low or high to start sending Security Events to Datadog.

   init_config:
     legacy_mode: false
   instances:
     - dd_security_events: high
   

   • low: sends only the most important and critical Security events, including Audit log cleared (1102), Replay attack detected (4649), and System audit policy was changed (4719). For a full list of events collected on the low setting, see here.
   • high: sends a higher volume of Security events, including Encrypted data recovery policy was changed (4714), Domain policy was changed (4739), and Security-disabled group was deleted (4764). For a full list of events collected on the high setting, see here.

Teams can change which event IDs are associated with low or high settings by editing these profiles.

1. Restart the Agent.

Data Collected

Metrics

The Windows Event Log check does not include any metrics.

Events

All Windows events are forwarded to Datadog.

Service Checks

The Windows Event Log check does not include any service checks.

Troubleshooting

Need help? Contact Datadog support with an Agent Flare.

Log processing rules are not working

If you are using log processing rules to filter out logs, verify that the raw logs match the regular expression (regex) pattern you configured. In the configuration below, log levels must be either warning or error. Any other value is excluded.

    - type: windows_event
      channel_path: System
      source: windows.events
      service: Windows       
      log_processing_rules:
      - type: include_at_match
        name: system_errors_and_warnings
        pattern: '"level":"((?i)warning|error)"'

To troubleshoot your log processing rules:

1. Remove or comment out the log_processing_rules stanza.
2. Restart the Agent.
3. Send a test log that includes the values you’re attempting to catch. If the log appears in Datadog, there is probably an issue with your regex. Compare your regex against the log file to make sure you’re capturing the right phrases.

Further Reading

Additional helpful documentation, links, and articles:

• Advanced Log Collection
• Monitoring Windows Server 2012
• How to collect Windows Server 2012 metrics
• Monitoring Windows Server 2012 with Datadog
• Monitor Windows event logs with Datadog
• Monitor your Windows event logs with Datadog Cloud SIEM]

Request a personalized demo

Get Started with Datadog

Картинка с сайта: www.inversecos.com

Threat actors can leverage native Windows API calls to install malicious services without generating correlating entries in the event log. Using native APIs to install services instead of the standard API calls allow threat actors to bypass security controls and event logging. This technique was utilised in the infamous Stuxnet malware sample created by the alleged US and Israeli government to target the Iranian nuclear program. It also happens to be one of my favourite malware samples <3. I will not be covering what native API calls are – for more information please take a read of one of my previous tweets..

This technique is not new (as it was used by Stuxnet), however, a lot of analysts in the industry see the event logs as a single source of truth without realising that sometimes it’s not always the case. As such, it’s always prudent for a responder/hunter/detection engineer to correlate various streams of evidence per artefact versus relying on a single source of evidence. @HarlanCarvey calls this building an ‘artifact constellation’ and I think this is a fantastic phrase that perfectly describes this situation. 

In this blog post I’m going to highlight how services can be installed through native API calls to evade detection and then focus on detection mechanisms through memory forensics. It’s difficult to talk about this topic without covering some process internal information so the blog has been structured to cover:

• How the technique works and how it bypasses event logging
• Why the volatility plugins “svcscan” does not detect services installed via this method
• Detection methodology

If no “evasion” techniques are used, services are typically created through a standard Windows API call to “CreateServiceA” or “CreateService” – you can read the full documentation for this API here. This API is also called whenever you use the native Windows executable “sc.exe” to register a service on a machine. A call to this API subsequently generates entries in the event log which corresponds to the service creation event IDs that a responder would be used to seeing. 

Threat actors can create services without relying on the typical API call to “CreateServiceA” by directly interacting with the Windows native API calls. For example, the “StartService” API will a make a call to the native API “NdrClientCall2” along with registering the service start events in the event logs. However, by directly calling “NdrClientCall2” to start a service that’s had the registry keys manually created – the service is started and running and no correlating event logs are created – thus evading detection. This will result in a malicious service running and no correlating event log entry. To take this a step further, the threat actors can then remove the evidence in the registry and any correlating evidence loaded in the memory of “services.exe”. This leaves you with little to no registry artefacts and no event log information to analyse. 

Stuxnet utilises this technique to register two malicious services to run malicious kernel drivers using a call to the native API “NtLoadDriver”. However, the malware did not remove the artefacts hidden within the registry (because these registry values are required to call NtLoadDriver). This API is documented in the NTAPI Undocumented Functions website here. To call the API, you just pass one value – the driver name and ensure that the registry has had the path to the malicious driver set. 

Картинка с сайта: www.inversecos.com

Normally, when given a memory sample, analysts will run the “svcscan” plugin in Volatility to hunt for services. When confronted with Stuxnet – you will notice this does not return results showing the malicious services. The two malicious services that exist in Stuxnet are called “mrxnet” and “mrxcls”.  Running the “svcscan” module you cannot see evidence of these services existing either. 

Картинка с сайта: www.inversecos.com

However, referring to the documentation on NtLoadDriver where a registry value needs to be set. Iterating through the registry keys within the memory sample shows that these services indeed exist. So why is this the case? Why didn’t the svcscan plugin pick up on this service?

Картинка с сайта: www.inversecos.com

Картинка с сайта: www.inversecos.com

Картинка с сайта: www.inversecos.com

8. Correlate the malicious driver files with the loaded kernel modules to PROVE that the service was running on the system. This means these drivers were running. 

Картинка с сайта: www.inversecos.com

Event Log Explorer comes with 3 methods of opening event log files: Standard, New API and Direct. Such an abundance of options may confuse users when choosing the method.

“What
opening method should I use?” – a very common question of our customers.

The answer is very simple:

In most cases use New API method.

But what the
other methods for?

First, I
should remind that Event Log Explorer supports 2 formats of Event Log Files: classic
 EVT and modern EVTX. Windows NT, 2000
and XP/2003 log events into EVT files. To access these files Windows introduced
a special Event Logging API which we call StandardAPI). Starting from Windows
Vista/2008, Windows uses EVTX format. In the same time a new Windows Event Log
API was introduced. For compatibility reasons, Windows allowed to open any file
with any API although since Windows 7 support of EVT files was terminated.

In addition
to Windows API, we developed our own way to access event files without using
any API (direct access).

So if you have a modern Windows PC and you want to read classic EVT files, you have no choice except Direct method. More details are available in this article.

And if you want to open EVTX files, always use New API method. Using Standard method would also work, but for some events it may give you incorrect details. You can use Standard method e.g. if you are a software developer and make your own logging. In this case you may check your log to verify how it will look in legacy event log viewers. Direct method for EVTX files was designed to access these files on old Windows XP machines or to access corrupted EVTX log files.

Download Event Log Explorer and open your event files right now!

1. Introduce

Application can use the Event logging API to report and view events.
For details on reporting events, see Reporting Events.
To view events that have been written to log files, see Querying for Event Source Messages
and Receiving Event Notification.

You can find widows event log files(binary file) on C:\WINDOWS\system32\config\, list as below:

C:\WINDOWS\system32\config\AppEvent.Evt
C:\WINDOWS\system32\config\SecEvent.Evt
C:\WINDOWS\system32\config\SysEvent.Evt

2. Program

2.1 Prepare work

2.1.1 Write a message text file, name provider.mc, the follow show it

; // MyEventProvider.mc 

; // This is the header section.


SeverityNames=(Success=0x0:STATUS_SEVERITY_SUCCESS
               Informational=0x1:STATUS_SEVERITY_INFORMATIONAL
               Warning=0x2:STATUS_SEVERITY_WARNING
               Error=0x3:STATUS_SEVERITY_ERROR
              )


FacilityNames=(System=0x0:FACILITY_SYSTEM
               Runtime=0x2:FACILITY_RUNTIME
               Stubs=0x3:FACILITY_STUBS
               Io=0x4:FACILITY_IO_ERROR_CODE
              )

LanguageNames=(English=0x409:MSG00409)


; // The following are the categories of events.

MessageIdTypedef=WORD

MessageId=0x1
SymbolicName=NETWORK_CATEGORY
Language=English
Network Events
.

MessageId=0x2
SymbolicName=DATABASE_CATEGORY
Language=English
Database Events
.

MessageId=0x3
SymbolicName=UI_CATEGORY
Language=English
UI Events
.


; // The following are the message definitions.

MessageIdTypedef=DWORD

MessageId=0x100
Severity=Error
Facility=Runtime
SymbolicName=MSG_INVALID_COMMAND
Language=English
The command is not valid.
.


MessageId=0x101
Severity=Error
Facility=System
SymbolicName=MSG_BAD_FILE_CONTENTS
Language=English
File %1 contains content that is not valid.
.

MessageId=0x102
Severity=Warning
Facility=System
SymbolicName=MSG_RETRIES
Language=English
There have been %1 retries with %2 success! Disconnect from
the server and try again later.
.

MessageId=0x103
Severity=Informational
Facility=System
SymbolicName=MSG_COMPUTE_CONVERSION
Language=English
%1 %%4096 = %2 %%4097. 
.


; // The following are the parameter strings */


MessageId=0x1000
Severity=Success
Facility=System
SymbolicName=QUARTS_UNITS
Language=English
quarts%0
.

MessageId=0x1001
Severity=Success
Facility=System
SymbolicName=GALLONS_UNITS
Language=English
gallons%0
.

Use the following command to compile the message text file:

mc -U provider.mc

It will generate 3 files: (the report event project use provider.h)

provider.h
provider.rc
MSG0409.bin

Compile the resources file:

rc provider.rc

Generate only one file, provider.res

Last use link to create resource-dll file:

link -dll -noentry provider.res

Output file, provider.dll, with these library, event log viewer can show the
the detail description by ID, Type … etc.(at the follow, Receiving event
notification use thid library)

2.2 Report event log

An example show how to report event.
Create a win32 console project, with empty file.
Then, add a reportevent.cpp to it, and add the follow code.

#ifndef UNICODE
#define UNICODE
#endif
#include <windows.h>
#include <stdio.h>
#include "provider.h"

#pragma comment(lib, "advapi32.lib")

#define PROVIDER_NAME L"MyEventProvider"

// Hardcoded insert string for the event messages.
CONST LPWSTR pBadCommand = L"The command that was not valid";
CONST LPWSTR pFilename = L"c:\\folder\\file.ext";
CONST LPWSTR pNumberOfRetries = L"3";
CONST LPWSTR pSuccessfulRetries = L"0";
CONST LPWSTR pQuarts = L"8";
CONST LPWSTR pGallons = L"2";

void wmain(void)
{
    HANDLE hEventLog = NULL;
    LPWSTR pInsertStrings[2] = {NULL, NULL};
    DWORD dwEventDataSize = 0;

    // The source name (provider) must exist as a subkey of Application.
    hEventLog = RegisterEventSource(NULL, PROVIDER_NAME);
    if (NULL == hEventLog)
    {
        wprintf(L"RegisterEventSource failed with 0x%x.\n", GetLastError());
        goto cleanup;
    }

    // This event includes user-defined data as part of the event. The event message
    // does not use insert strings.
    dwEventDataSize = ((DWORD)wcslen(pBadCommand) + 1) * sizeof(WCHAR);
    if (!ReportEvent(hEventLog, EVENTLOG_ERROR_TYPE, UI_CATEGORY, MSG_INVALID_COMMAND, NULL, 0, dwEventDataSize, NULL, pBadCommand))
    {
        wprintf(L"ReportEvent failed with 0x%x for event 0x%x.\n", GetLastError(), MSG_INVALID_COMMAND);
        goto cleanup;
    }

    // This event uses insert strings.
    pInsertStrings[0] = pFilename;
    if (!ReportEvent(hEventLog, EVENTLOG_ERROR_TYPE, DATABASE_CATEGORY, MSG_BAD_FILE_CONTENTS, NULL, 1, 0, (LPCWSTR*)pInsertStrings, NULL))
    {
        wprintf(L"ReportEvent failed with 0x%x for event 0x%x.\n", GetLastError(), MSG_BAD_FILE_CONTENTS);
        goto cleanup;
    }

    // This event uses insert strings.
    pInsertStrings[0] = pNumberOfRetries;
    pInsertStrings[1] = pSuccessfulRetries;
    if (!ReportEvent(hEventLog, EVENTLOG_WARNING_TYPE, NETWORK_CATEGORY, MSG_RETRIES, NULL, 2, 0, (LPCWSTR*)pInsertStrings, NULL))
    {
        wprintf(L"ReportEvent failed with 0x%x for event 0x%x.\n", GetLastError(), MSG_RETRIES);
        goto cleanup;
    }

    // This event uses insert strings.
    pInsertStrings[0] = pQuarts;
    pInsertStrings[1] = pGallons;
    if (!ReportEvent(hEventLog, EVENTLOG_INFORMATION_TYPE, UI_CATEGORY, MSG_COMPUTE_CONVERSION, NULL, 2, 0, (LPCWSTR*)pInsertStrings, NULL))
    {
        wprintf(L"ReportEvent failed with 0x%x for event 0x%x.\n", GetLastError(), MSG_COMPUTE_CONVERSION);
        goto cleanup;
    }

    wprintf(L"All events successfully reported.\n");

cleanup:

    if (hEventLog)
        DeregisterEventSource(hEventLog);
}

2.3 View event log

go to http://msdn.microsoft.com/en-us/library/aa363677(v=vs.85).aspx
get the source code of “Receiving Event Notification”.

Notice that, change these two lines for your apply:

#define PROVIDER_NAME L"MyEventProvider"
#define RESOURCE_DLL  L"<path>\\Provider.dll"

3. Questions

How to monitor multi event logs? And make it to be a component.
I want to implement it, on this site you can find a first version project
Click here

But I think, it was a pool method to check each event name. Is there exist any
good method to do it? I meet the same problem with this guy,
Unexpected behavior of WaitForMultipleObjects for events used in NotifyChangeEventLog
I’m sad, why he don’t told the detail info.

4. Reference

• Using Event Logging
• Reporting Events: http://msdn.microsoft.com/en-us/library/aa363680(v=vs.85).aspx
• Receiving Event Notification: http://msdn.microsoft.com/en-us/library/aa363677(v=vs.85).aspx
• resource a freeware utility to view, modify, rename,
  add delete and extract resources in 32bit & 64bit windows executable and resource files
• Unexpected behavior of WaitForMultipleObjects for events used in NotifyChangeEventLog
• SNARE Auditing and EventLog Management: