Logging Made Easy Through Graylog Part 1
Logging is an important piece of an organization’s security posture. Logging without organization, searchability, or reporting leads to data being missed. This is the start of a long series that VDA Labs is writing on Graylog. This will be a multi-part series covering a variety of topics including the following items:
- Installation, securing, and optimizing the setup part 1
- Installation, securing, and optimizing the setup part 2
- Domain Controller/DHCP log collection and alerts
- File/print server log collection and alerts
- Exchange server log collection
- IIS log collection
- Firewall log collection
- Setting up Threat Intelligence
- SQL Server
Knowledge is power
Although many pieces of software provide logging, that logging can be modified or deleted from a system before it can be used for troubleshooting or post incident analysis. Having an external source of logging from all systems and services can help decrease time spent troubleshooting or correlating logs from separate systems. External logging can also provide insight into what an attacker may have done on systems even if they deleted or modified the local logs.
Graylog provides an answer to this problem by providing a way to export logs and put them in a separate system. That machine can then be used for: alerting, dashboards, reporting, and incident response.
Sizing Your System
For this blog we will be using the following system specs.
- 6 cores
- 24GB of memory
- 500GB of SSD Storage
Graylog is resource intensive, but with the following specs administrators can expect to process a few thousand messages per second. If administrators need more performance, they should start looking at a multi-node configuration. This process is outside the scope of this blog.
DNS
I CANNOT STRESS THIS ENOUGH. YOU MUST HAVE A VALID FQDN WITH A DNS RECORD.
If a valid DNS record does not exist and you use a IP for your configuration files within Graylog, you will have failures due to missing or invalid Subject Alternative Names (SAN) on your certificates.
Installing Prerequisites
Lets begin by installing and configuring the prerequisites for Graylog functionality.This installation will be done in an Ubuntu 18.04LTS environment. The following packages need to be installed before continuing:
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
MongoDB
Next lets install MongoDB.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
After the installation, make sure to configure MongoDB to start at boot:
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service
Elasticsearch
Graylog 3.x requires Elasticsearch 6.x. Using the following guide, lets install and configure Elasticsearch to work with the Graylog install. Starting with the install, lets run the following commands to install Elasticsearch:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install elasticsearch
Configure Elasticsearch for Graylog
Elasticsearch is now installed. It is time to name the cluster, this can be done by modifying the configuration file located in /etc/elasticsearch/elasticsearch.yml.
We want to uncomment and modify the following line to match the the configuration.
cluster.name: graylog
With Elasticsearch installed, tell the system to start the Elasticsearch service during boot process of the OS. This can be accomplished with the following commands:
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl restart elasticsearch.service
Securing Our Back End
Before installing Graylog, lets ensure our back end is secure by adding authentication and encryption to the MongoDB and Elasticsearch nodes. If these steps are skipped, there could be potential badness:
- Intercept data to and from MongoDB nodes
- Intercept data to and from Elasticsearch nodes
- Join rouge Elasticsearch nodes to a multi-node setup
It is especially important to complete these steps if the Graylog instance will be running in the cloud or on internet facing systems.
Securing MongoDB
Adding Authentication To MongoDB
Before we add TLS, lets include authentication to the MongoDB Graylog DB.
First, connect to the DB Instance:
mongo --port 27017
Next, lets connect to the Graylog DB and create an admin user with a strong password.
use graylog;
db.createUser(
{
user: "mongo_admin",
pwd: "password123",
roles: [ { role: "root", db: "admin" } ]
}
)
exit
A user is created, lets enable authentication for MongoDB. This can be done by modifying the following line in /etc/mongod.conf:
security:
authorization: enabled
Adding TLS
Before adding TLS, lets generate a Certificate of Authority (CA) certificate and a server certificate. We will be using self-signed certificates for this project. For production systems an engineer would want to get valid certs from either a trusted internal CA or a trusted external CA.
Generating our CA Certificate
Before we can do anything, lets generate a CA certificate that we can use to sign our server certificates. This can be done with the following command. These commands are ran from /opt/opensslkeys:
openssl req -out graylogca.pem -new -x509
After doing this we should have two files, graylogca.pem and privkey.pem. Make sure both of these keys are kept secure as they will be used for signing all other certificates on our system. See the output below to see what this process should look like when successful.
Generate our Server Certificate
Now that we have our CA certificate created, lets sign our server certs. To do this we need to run the following commands:
sudo openssl genrsa -out MongoDB.key 2048
sudo openssl req -key MongoDB.key -new -out mongodb.req
sudo openssl x509 -req -in mongodb.req -CA graylogca.pem -CAcreateserial -out mongodb.pem -CAkey privkey.pem
If everything is successful the engineer will receive this output:
Generating our MongoDB key
Generating our CSR
Generating our signed server certificate
Combine the private key and the Mongodb certificate with the following steps:
cat mongodb.pem
cp MongoDB.key mongoDB2.pem
Copy the data from mongodb.pem to append the data in mongoDB2.pem. The final product should have the following format.
-----BEGIN RSA PRIVATE KEY-----
Key
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
Certificate
-----END CERTIFICATE-----
Apply TLS Certificate to MongoDB
Now that we have generated our CA and server certificates we need to go modify our MongoDB config located at /etc/mongod.conf. The following lines need to be modified or created.
net:
ssl:
mode: requireSSL
PEMKeyFile: /opt/opensslkeys/mongoDB2.pem
CAFile: /opt/opensslkeys/graylogca.pem
allowConnectionsWithoutCertificates: true
With this completed we need to restart MongoDB and check the log file. If everything is working you should see that MongoDB has started and is running.
chmod 444 /opt/opensslkeys/*
sudo service mongod restart
sudo cat /var/log/mongodb/mongod.log
Securing Elasticsearch
Configuring TLS
For this project a self signed certificate will be used for the Elasticsearch node. This certificate can be generated using the certutil utility included with Elasticsearch. This tool is located in /usr/share/elasticsearch/bin/elasticsearch-certutil. Do not put a password on the file you create.
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --ca-dn CN=dontsquatme.com
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip <Server IP> --dns <Server FQDN>
If successful there will be two files:
elastic-certificates.p12
elastic-stack-ca.p12
Move the elastic-certificate to /etc/elasticsearch/certs:
sudo mkdir -p /etc/elasticsearch/certs
sudo cp /usr/share/elasticsearch/bin/elastic-certificates.p12 /etc/elasticsearch/certs
chmod 444 /etc/elasticsearch/certs/*
Next we want to modify our elasticsearch.yml located in /etc/elasticsearch/elasticsearch.yml. Add the following lines to ensure the new configuration is used:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: none
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
We also want to modify the following line to point to the IP address of our server:
network.host: $ServerIPGoesHere
Restart elasticsearch
sudo service elasticsearch restart
Setting up Authentication
Setting up authentication for Elasticsearch is extremely easy. All we need to do is run the following command.
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Keep track of the passwords you set here as they will be needed later.
Time to Install Graylog
Now all the prerequisites are completed, its time to install Graylog. Run the following commands to start the process:
wget https://packages.graylog2.org/repo/packages/graylog-3.2-repository_latest.deb
sudo dpkg -i graylog-3.2-repository_latest.deb
sudo apt-get update && sudo apt-get install graylog-server
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service
Adding our Self Signed Certificate to the Graylog JVM Trust Store
Without adding the newly generated self signed certificate to our Graylog JVM Trust Store, Graylog will throw errors and not connect to Elasticsearch. To prevent this, run the following commands:
sudo cp -a "${JAVA_HOME}/jre/lib/security/cacerts" /etc/graylog/server/cacerts.jks
The easiest way to find the location of the original JVM Trust Store is to use the following command:
sudo find / -name "cacerts"
As of this writing, the OpenJDK8 keystore is located at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts which means the following command will copy the data we need. The purpose of copying the keystore is to not modify the global official trust store and instead we want to make a copy for only Graylog to use:
sudo cp -a /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts /etc/graylog/server/cacerts.jks
Copy the previously created .p12 to the Graylog configuration folder and convert it to a format that Java can use:
sudo cp /etc/elasticsearch/certs/elastic-certificates.p12 /etc/graylog/server/elastic-certificates.p12
Convert to a .pem
sudo openssl pkcs12 -in /etc/graylog/server/elastic-certificates.p12 -out /etc/graylog/server/elastic.pem
Convert to a x509 certificate
sudo openssl x509 -outform der -in elastic.pem -out elastic.crt
sudo openssl x509 -outform der -in graylogca.pem -out graylogca.crt
Import it into the new Graylog keystore:
sudo keytool -importcert -keystore /etc/graylog/server/cacerts.jks -storepass changeit -alias elastic-cluster -file /etc/graylog/server/elastic.crt
sudo keytool -importcert -keystore /etc/graylog/server/cacerts.jks -storepass changeit -alias graylogca -file /opt/opensslkeys/graylogca.crt
Configuring Graylog
Now that we have Elasticsearch, MongoDB, and Graylog installed, we need to configure Graylog. The Graylog configuration file is server.conf, its located at /etc/graylog/server/server.conf. Here are the important lines:
- password_secret
- This should be a 64 character key used to salt other passwords
- root_password_sha2
- This is the default password for user “Admin” it should be set using the following command.
- echo -n yourpassword | shasum -a 256
- Copy the resulting answer into this line
- http_bind_address
- This is the address the web interface will bind to, this should be the private IP address of your server
- elasticearch_hosts
- Again this should point to your IP of your server. Don’t forget to include the username and password you created
- https://username:password@<Server FQDN>:9200
- mongodb_uri
- This will point at the IP address of your server make sure to include the username and password you created
- mongodb://username:password@Server FQDN:27017/graylog?ssl=true
After this is completed, Graylog should be operational. Reboot your server one more time. Run the following commands to ensure the services have started properly:
service graylog-server status
service mongod status
service elasticsearch status
sudo cat /var/log/elasticsearch
sudo cat /var/log/mongodb
sudo cat /var/log/graylog-server
If everything was setup correctly, browse to http://<Server IP>:9000 and log in to Graylog for the first time with the admin username and password that was created earlier.
VDA’s next blog post will cover securing the Graylog web interface and adding the first server to start parsing data from. If you have any questions about application, IoT, or system security please visit VDA’s home page to view our services.