No More Secrets: Logging Made Easy Through Graylog Part 3
Logging is a important but often overlooked part of an organization’s security posture. Logging without organization, searchability, or reporting leads to data being missed. This a continuation of a longer series that VDA Labs is writing on Graylog. This is part 3 of a multi-part series covering a variety of topics, including the following items:
- Installation, securing, and optimizing the setup part 1
- Installation, securing, and optimizing the setup part 2
- Domain Controller/DHCP log collection and alerts
- File/print server log collection and alerts
- Exchange server log collection
- IIS log collection
- Firewall log collection
- Setting up Threat Intelligence
- SQL Server
This week focuses on gathering logs from from a Windows Domain Controller/DHCP server. These logs are some of the most important logs a blue team can gather. They can help track and monitor users and devices across the organization and greatly enhance troubleshooting and incident response event due to the correlation that can be completed.
Domain Controller and DHCP Logging
Before gathering logs from our systems, there are a few prerequisites to take care of. First, enable logging for the events the domain controllers will encounter such as password changes, account lock-outs, and login failures, etc. Enable DHCP logging in order to see a historical account of which IP address were handed out to clients. This will allow the sysadmin to trace activity through the firewall once log collection begins.
Enabling Domain Controller Logging
To enable logging on Domain Controllers, edit the “Default Domain Controller Policy” GPO. After opening the “Default Domain Controller Policy” GPO, browse to Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> Audit Policies and enable Success and Failure tracking for all events.
Enabling Audit Logging For DC Events
Enabling DHCP Logging
To enable DHCP logging, use the DHCP administration tool, right-click on the DHCP Server that needs modification, select properties, and then enable DHCP audit logging from the General tab. The log files created will be located in %systemroot%\system32\dhcp.
Enabling DHCP Logging
Creating Our First Graylog Inputs
Now that Windows is configured to create logs, configure Graylog to accept the logs from the services. To start this process, browse to System -> Inputs.
Our empty input section
From the “Select Input” drop down, select “GELF TCP” and launch that new input. Let’s go over the settings we need to configure.
- Enable Global Input
- This will ensure that, if we create a cluster of Graylog servers, this input is created on all of them.
- Bind address
- Set this to the IP address currently being used by the Graylog Server.
- TLS cert file
- Make sure this points at the GraylogWeb.pem we created
- TLS private key file
- Make sure this points at the GraylogWeb.key we created
Everything else can be left on the default settings. If the configuration is successful, there will be one running Global Input.
Our new SSL input waiting for logs
Next is configuring our Domain Controller to start shipping the DC logs to the Graylog server.
Installing our Log Shipping Agent
To ship the DC logs from our domain, lets use NXlog Community Edition. This is a tool which can be configured to read just about any file or system input. There is also an enterprise support option available if required by your organization.
The Windows .MSI install will be used for this guide. Both the GraylogWeb.pem and GraylogWeb.key from part 2 of this series and the graylogca.pem from part one will be needed.
Installing NXLog is as simple as running through our download .msi installer. After installation is finished, the configuration needs to be written.
The first step of this configuration is to copy the GraylogWeb.pem, GraylogWeb.key, and graylogca.pem to C:\Program Files(x86)\nxlog\certs. Without these files, the SSL connection to the Graylog input that was created in the last step will fail. With those files moved over, let’s start going over the configuration we need to create. Here is the complete configuration file, located in C:\Program Files(x86)\nxlog\conf\nxlog.conf:
## This is a sample configuration file. See the nxlog reference manual about the ## configuration options. It should be installed locally and is also available ## online at http://nxlog.org/docs/ ## Please set the ROOT to the folder your nxlog was installed into, ## otherwise it will not start. #define ROOT C:\Program Files\nxlog define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf ShortMessageLength 500 </Extension> <Extension ParseDHCP> Module xm_csv Fields $ID ,$Date ,$Time ,$Description ,$IPAddress ,$ReportedHostname ,$MACAddress ,$UserName ,$TransactionID ,$QResult ,$Probationtime ,$CorrelationID ,$Dhcid Delimiter ',' </Extension> <Input EventLog_In> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ <Select Path="DNS Server">*</Select>\ <Select Path="Directory Service">*</Select>\ <Select Path="DFS Replication">*</Select>\ </Query>\ </QueryList> </Input> <Input DHCP_IN> Module im_file File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log" SavePos TRUE InputType LineBased Exec $Message = $raw_event; </Input> <Output DHCP_Out> Module om_ssl OutputType GELF_TCP Host <FQDN> CertFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.pem CertKeyFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.key CAFile C:\Program Files (x86)\nxlog\cert\graylogca.pem Port 12201 </Output> <Output EventLog_Out> Module om_ssl OutputType GELF_TCP Host <FQDN> CertFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.pem CertKeyFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.key CAFile C:\Program Files (x86)\nxlog\cert\graylogca.pem Port 12201 </Output> <Route DHCP> Path DHCP_IN => DHCP_OUT </Route> <Route EventLog> Path EventLog_In => EventLog_Out </Route>
Breaking Down The Configuration File
Important sections of this configuration file:
<Extension gelf> Module xm_gelf ShortMessageLength 500 </Extension>
Extending the “Short Message” field of the GELF message type from 64 characters to 500 characters can prevent some logs from creating errors.
<Extension ParseDHCP> Module xm_csv Fields $ID ,$Date ,$Time ,$Description ,$IPAddress ,$ReportedHostname ,$MACAddress ,$UserName ,$TransactionID ,$QResult ,$Probationtime ,$CorrelationID ,$Dhcid, $VendorClass(Hex), $VendorClass(ASCII), $UserClass(Hex), $UserClass(ASCII), $RelayAgentInformation, $DnsRegError Delimiter ',' </Extension>
All the logs created by our DHCP server are comma separated values. Browsing to %systemroot%/System32/dhcp/, will show the log files created for each day of the week. Inside these files will be a listing for each field created. For each field in the DHCP log file, a corresponding field in the CSV parser is required.
<Input EventLog_In> # Use 'im_mseventlog' for Windows XP, 2000 and 2003 Module im_msvistalog # Uncomment the following to collect specific event logs only Query <QueryList>\ <Query Id="0">\ <Select Path="Application">*</Select>\ <Select Path="System">*</Select>\ <Select Path="Security">*</Select>\ <Select Path="DNS Server">*</Select>\ <Select Path="Directory Service">*</Select>\ <Select Path="DFS Replication">*</Select>\ </Query>\ </QueryList> </Input>
Next, lets select which event logs are worth gathering. Starting with Server 2016, NXLog can no longer absorb all the Windows logs (too many sources). Because of this issue, a few logs sources will be need to be picked that will have the highest impact on security and troubleshooting. To add new event log paths, append a new select path which points to the new event log. The limit is 256 sources.
<Input DHCP_IN> Module im_file File "C:\\Windows\\Sysnative\\dhcp\\DhcpSrvLog-*.log" SavePos TRUE InputType LineBased Exec $Message = $raw_event; </Input>
With the <Input DHCP_IN> section, NXLog is being told where to look for the DHCP logs and what the file name formatting should be.
<Output DHCP_Out> Module om_ssl OutputType GELF_TCP Host <FQDN> CertFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.pem CertKeyFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.key CAFile C:\Program Files (x86)\nxlog\cert\graylogca.pem Port 12201 </Output> <Output EventLog_Out> Module om_ssl OutputType GELF_TCP Host <FQDN> CertFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.pem CertKeyFile C:\Program Files (x86)\nxlog\cert\GraylogWeb.key CAFile C:\Program Files (x86)\nxlog\cert\graylogca.pem Port 12201 </Output>
The next two sections are dedicated to configuring where to send the logs. The system to needs to know how to use SSL key, cert, and CA that were created earlier and copied to C:\Program Files (x86)\nxlog\cert\.
<Route DHCP> Path DHCP_IN => DHCP_OUT </Route> <Route EventLog> Path EventLog_In => EventLog_Out </Route>
In this last section, NXLog is notified where the outputs need to route the created inputs to. Lets go directly from EventLog_In and DHCP_IN to EventLog_Out and DHCP_OUT making for a very simple routing configuration.
With the configuration file and SSL certs copied over, restart the NXlog service and browse to C:\Program Files (x86)\nxlog\data\nxlog.log. The service should restart without errors.
Back to Graylog!
Browsing to the Graylog server, logs should be coming in. FINALLY! THE FRUITS OF OUR LABOR!
Organizing our Data
Creating our Indexes
Now that data is coming into Graylog lets start building and configuring a index to store our data. This data can be seen by browsing to “System / Indices” A index is group of data in Graylog that has a a common configuration for retention, sharding, and replication of the stored data. Typically I create a different index for various types of data. For example I would have a index for SQL logs, a index for Domain Controller Logs, a index for DHCP logs, etc. Before we do anything lets modify our default index set.
It is typical use a rotation period of P1D or 1 day. By using this retention period it is easy to calculate data set size for each type of index we create. For the default index I use 90 days but you may want to change this based on your organizations needs and how much storage you want to dedicate to logs. The number of shards and replicas can be left default unless a multi-node Graylog setup is being run.
Next create two new indexes “DHCP” and “Event Log”. Feel free to name these index sets however the name will be used to label the index a log is in so it is useful to have descriptive names. Here is what it should look like when completed.
Our created indexes
Routing our Data Into our New Indexes
With the indexes created data needs to be routed into these indexes. To do this browse over to Streams. Select “Create Stream” and build a stream for a DHCP logs. Remove logs that are moved to the new index from the default index. This will help prevent duplicate messages and keep our default index clean.
With the stream created, rules need to be added to help Graylog know what messages route into this index. To do this find a DHCP message that we can use as a template. Open Search and search “DHCP_IN”, using this as this is the module name reported by NXLog to Graylog allowing us to only show data from the DHCP_IN input. Expand one of the DHCP logs gathered and select “Copy ID” also take note of the index this log is stored in.
With the copied ID and index head back to the Stream interface. Select “Manage Rules” on the DHCP index that was created. Inside of this section select the tab “Message ID” and paste the copied message ID and input the index this message is part of. With the message now loaded fields available to work with will become visible.
With the message loaded, lets add the stream rule to help route these messages. For this use the field “SourceModuleName” which is reported to Graylog from NXLog. Match the value DHCP_IN so that all messages with the Source Module of DHCP_IN get added to the DHCP Index.
If successful you should start that our field rule matches the message loaded.
Start the stream now that it is configured it and data should flow into the DHCP index.
Complete the same steps for Event Logs but use a “SourceModuleName” of “EventLog_In”. After completion there should be a total of five streams.
Cleaning Up Our DHCP Logs
Now that logs are in their correct index lets work on cleaning up the logs we are receiving from the DHCP server. Right now the logs are coming over as one large string making searching for individual terms more difficult. Instead break each each field in the CSV document into their own Elasticsearch field. To do this head over to “System/Inputs” and manage the extractors of our GELF TCP SSL input that was created earlier. Select “Get started” and load up a DHCP message to evaluate.
With the message loaded lets select a extractor type of “Copy Input” on the field “full_message”. Make sure to add a regular expression that will only attempt to extract if a message matches the requirements. This ensures Graylog doesn’t process every message through every extractor. Use the following Regex to only extract our DHCP data.
Store this as the field “DHCP_Extract” and cut the message being modified so a second copy of the data is not made. Next add our converter, the converter type will be “CSV to Fields” and will use the following fields as described in the extension ParseDHCP but make sure to remove all spaces and leading “$”.
If everything is configured properly the new extractor should look like the following image.
If everything is successful next time a DHCP log is viewed individual fields we can search and alert on are being created for each message.
Using the Data We Have Created
Now that data is flowing into the system the data can be used to help relay information quickly to users. In this section we will describe how to create dashboards and our first alerts.
Creating Our First Dashboard Widgets
With the data in our system lets create two new dashboard widget. One widget for accounts that have been locked out and one widget for failed network logons. Start by making a blank dashboard to modify. To do this select “Dashboards” at the top and then select “Create New Dashboard”. Create a dashboard called “Domain Contollers”.
Return to our primary search page and search for EventID:4740 OR EventID:4771. After execution search use the drop down on them message count to select “Copy To Dashboard” and select the Domain Controller dashboard created earlier.
Edit the widget just created to only show failed logons, this can be done by using the search term EventID:4771. Select “Save” and we have created our first widget.
On the left side of the screen expand the menu and select “Aggregation”.
With the new menu open, fill in the search field with EventID:4740. Also select the metric of count() and a row of timestamp. Select save and now two widgets should be visible. One showing a 24 hour bar graph of account lockouts and another showing a 24 hour bar graph of failed logons.
The same process can be repeated for two message tables that will also show the pertinent logs from our search queries. In the end there should domain controller page with 4 widgets.
Creating our First Alerts
With data incoming and dashboards created, create some alerts so to use the new logging capabilities Graylog provides to be proactive instead of reactive to problems in the environment.
Configuring our SMTP Server
To start sending out alerts a SMTP server needs to be configured for email alerts to be sent to. Edit our config file located on our Graylog server at /etc/gray/server/server.conf. Uncomment and configure the settings based on the organization email server FQDN and ports.
# Email transport #transport_email_enabled = true #transport_email_hostname = mail.example.com #transport_email_port = 587 or 25 #transport_email_use_auth = true #transport_email_auth_username = email@example.com #transport_email_auth_password = secret #transport_email_subject_prefix = [graylog] #transport_email_from_email = firstname.lastname@example.org # Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLS #transport_email_use_tls = true
With the SMTP settings configured restart Graylog one more time with the following command.
sudo service graylog-server restart
Once services have come back online login to Graylog and browse to “Alerts”. To start test the email setup just configured in previous steps. In the upper right corner select “Notifications” and then “Get Started”.
From there lets configure and test the new notification. Make sure to fill out the sender information and recipients the newly created notification. After everything is configured test the new SMTP setup. Ensure a green “Notification was executed successfully.” appears, If it doesn’t work make sure you look at your email server configurations and ensure the proper FQDN and ports have been set.
With email notifications tested as working save the new recipient configuration and lets go configure our Alert. Start by selecting “Alerts & Events” in the top right corner and then selecting “Get Started”.
Name our event and set its priority. For excessive failed AD logins use a priority of high.
Alerts are created by creating a search query that our system counts the total number of messages from. For alerting on failed AD logons a search query of EventID:4771. This is the EventID of failed network logons created on our domain controller(s). Make sure to also set what streams to evaluate logs from. For the AD Failed Logon Alert use the Event Logs stream we created earlier. Tell the system how far to search and how often. The rules will be evaluated every time the rule is run so make sure values are set high enough that email storm of alerts are not created.
Next set the rule that our count of messages will be evaluated against. Configure the system to send an alert if there is one or more failed network logons on the domain. Set this number to something reasonable for your organization.
Custom event fields are not being used so these settings can be ignored for now.
In the next section attach the notification “Too Many Failed AD Logons” created earlier to this alert. This also lets administrators have the possibility to set a grace period to ensure that during a incident that creates alerts administrators don’t get overwhelmed by repeating messages. Also if wanted, the messages that caused the alert can also be sent in the email to allow a administrator to more quickly start diagnosing the problem causing alerts.
If everything is successful, when there are more failed logons than our threshold a e-mail like the following will be generated and sent to the configured email inbox.
Alerts that are being generated should also appear in the “Alerts” web interfaces in Graylog.
Complete the same process for EventID: 4740 to create an alert for account lockouts.
Its been a long post but at this point Graylog 3.2.x is completed configured with our first logs coming in from our Domain Controllers and DHCP Servers. Dashboards and alerts have also configured. Our next blog will be about gathering logs and creating alerts from File servers and Print servers.