Detection Engineering: ELK Stack with Fleet

Intro

Last month, my team and I participated in the CCDC competition – an exciting cyber defense challenge where teams defend their systems against simulated attacks from red team professionals. The goal? Keep your services running and score points for every system you successfully protect.

In competitions like this, success revolves around three key pillars: Prevent, Monitor, and Cure. I realized that implementing a robust SIEM (Security Information and Event Management) system would give us a huge advantage – providing real-time visibility into our systems and enabling quick responses to attacks.

Let’s walk through setting up a powerful ELK stack with Fleet Server – a setup that could be a game-changer in your security operations.

Elasticsearch Installation

Elasticsearch serves as the powerhouse database where all your security logs are indexed and made searchable in milliseconds.

Machine’s IP: 10.0.0.4

• Import Elasticsearch public GPG key and add the Elastic package source list in order to install Elasticsearch.

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch |sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg

• Next, add the Elastic source list to the sources.list.d directory, where APT will search for new sources:

echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update

• Install Elasticsearch

sudo apt install elasticsearch

• This generates password for the elastic built-in superuser. Make sure to note it.
• Start the Elasticsearch service with systemctl

sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

• Test whether the Elasticsearch service is responding properly or not.

dawadi@ubuntuMachine:~$ curl -X GET http://localhost:9200
curl: (52) Empty reply from server

• We received empty reply from the server because we are using HTTP instead of HTTPS.

dawadi@ubuntuMachine:~$ curl -X GET https://localhost:9200
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

• We’ll skip the certificate check and then provide it with authentication details for elastic user.

dawadi@ubuntuMachine:~$ curl -X GET -k https://elastic:<PASSWORD>@localhost:9200
{
  "name" : "ubuntuMachine",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "cHwoig-tSCORzBMPNo022A",
  "version" : {
    "number" : "8.17.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "d4b391d925c31d262eb767b8b2db8f398103f909",
    "build_date" : "2025-01-10T10:08:26.972230187Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

• Elasticsearch is up and running and ready to index our logs.

Kibana Installation

Kibana is your window into the data – providing a sleek, intuitive interface to visualize and analyze the information stored in Elasticsearch. We’ll install it on the same machine as Elasticsearch for simplicity.

• Install Kibana

sudo apt install kibana

• Enable the service and start it.

sudo systemctl enable kibana
sudo systemctl start kibana

• Generate an enrollment token for Kibana instance by using elasticsearch-create-enrollment-token method.

dawadi@ubuntuMachine:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
eyJ2ZXIiOiI4........xYnVUa1AxZyJ9

• Setup Kibana to use this enrollment token.

dawadi@ubuntuMachine:~$ sudo /usr/share/kibana/bin/kibana-setup 
Native global console methods have been overridden in production environment.
? Enter enrollment token: eyJ2ZXIiOiI4........xYnVUa1AxZyJ9

✔ Kibana configured successfully.

To start Kibana run:
  bin/kibana

• Then restart the kibana service to apply changes.

sudo systemctl restart kibana

• Kibana runs on port 5601. We can check the status using ss -tupln because it takes some time for the service to be fully up.

dawadi@ubuntuMachine:~$ ss -tupln
Netid             State              Recv-Q              Send-Q                                Local Address:Port                           Peer Address:Port             Process             
udp               UNCONN             0                   0                                             [::1]:323                                    [::]:*                                    
tcp               LISTEN             0                   128                                         0.0.0.0:22                                  0.0.0.0:*                                    
tcp               LISTEN             0                   511                                       127.0.0.1:5601                                0.0.0.0:*                                    
tcp               LISTEN             0                   4096                                  127.0.0.53%lo:53                                  0.0.0.0:*                                   

Setup Kibana behind Nginx

Setting up Nginx as a reverse proxy adds an extra layer of security and flexibility to your Kibana deployment.

• Install nginx using sudo apt install nginx
• We are trying to put the Kibana service (running on port 5601) behind a nginx web server. Lets edit /etc/nginx/sites-enabled/default file and use proxy_pass feature.

        server_name _;

        location / {
                proxy_pass http://127.0.0.1:5601
        }

• Restart nginx service with sudo systemctl restart nginx
• Now you can access Kibana dashboard on port 80 of the server.
• Login with user elastic and the password that was generated in the beginning.

Now our setup is ready to ingest the security logs from the agents and start making sense from it. For this, one way is to install the Elastic Agent on each machines and configure them to directly send logs to the Elastic database. However, in this installation we’ll be using Fleet Server which serves as a control plane for updating agent policies, collecting status information, and coordinating actions across Elastic Agents.

If you notice, a Fleet Server is essentially an Elastic Agent with additional responsibilities. While both are based on the Elastic Agent, the key difference is their role within an Elastic Stack deployment:

• Elastic Agent: A lightweight, unified agent that collects logs, metrics, security data, and executes endpoint security policies.
• Fleet Server: A special Elastic Agent that acts as a control plane for managing and orchestrating other Elastic Agents. It facilitates communication between the Fleet UI (in Kibana) and the enrolled agents.

Setting up Fleet Server

Think of Fleet Server as your command center for managing all your Elastic Agents. It’s the bridge between your Kibana dashboard and the agents deployed across your infrastructure.

Machine’s IP: 10.0.0.6

We will be installing Fleet Server in a separate machine than Elastic.

Add Fleet Server integration to Kibana dashboard. Fleet Server is a component of the Elastic Stack used to centrally manage Elastic Agents.

I went with default settings.

Click on Add Agent and Enroll in Fleet. Add your Fleet Server’s IP address with port 8220. You can use Fleet Server’s Public IP Address as well.

Install it using the following commands:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.1-linux-x86_64.tar.gz 
tar xzvf elastic-agent-8.17.1-linux-x86_64.tar.gz 
cd elastic-agent-8.17.1-linux-x86_64

• Run this command in order to install Fleet Server Agent with proper configuration. I have added --fleet-server-es-ca=/usr/local/etc/ssl/certs/elastic/http_ca.crt --insecure because I am using self-signed certificates.

sudo ./elastic-agent install \ 
--fleet-server-es=https://10.0.0.4:9200 \ 
--fleet-server-service-token=<TOKEN> \ 
--fleet-server-policy=ff12d36f-c348-479b-b82c-8b18dbffd786 \ 
--fleet-server-es-ca-trusted fingerprint=<YOUR_FINGERPRINT> \ 
--fleet-server-port=8220 \
--fleet-server-es-ca=/usr/local/etc/ssl/certs/elastic/http_ca.crt --insecure

• /usr/local/etc/ssl/certs/elastic/http_ca.crt is not present by default. I copied the same certificate from Elastic machine from /etc/elasticsearch/certs/http_ca.crt for the sake of this tutorial. Please refer to this link to create your own certificates (a must if you are running in production.)
• Finally it is successfully installed and you can verify this from UI as well.

Adding agents

Now comes the exciting part – deploying agents to monitor your systems. These agents will be your eyes and ears, collecting and forwarding crucial system data to Elasticsearch.

Windows Agent IP: 10.0.0.5

• Now we will install elastic-agent on those machines which we want to monitor. These agents will forward their system logs (and configured logs) to Elasticsearch.

$ProgressPreference = 'SilentlyContinue' 
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.17.1-windows-x86_64.zip -OutFile elastic-agent-8.17.1-windows-x86_64.zip 
Expand-Archive .\elastic-agent-8.17.1-windows-x86_64.zip -DestinationPath . 
cd elastic-agent-8.17.1-windows-x86_64 
.\elastic-agent.exe install --url=https://10.0.0.6:8220 --enrollment-token=SEFkYnVKUUJ3RVZCdmxFcGpDcHE6UkhHbk1EWmVRc2FJQjBLelVsb3AxUQ==

• The elastic agent is successfully installed on this windows machine.

Lets verify this from Kibana’s Discover tab.

General workflow to set up Agent Policies

Here’s a simple three-step process to get your agents up and running:

• Create an Agent Policy – Defines the configurations and settings that will be applied to enrolled agents.
• Add Integrations – Include necessary integrations like logs, metrics, or security tools based on your requirements to your policy.
• Enroll Agents – Assign agents to the policy so they apply the configurations and start collecting monitoring data.

Enable Endpoint Security

Elastic Defend is your Swiss Army knife for security – packed with powerful features to protect your systems.

• Lets add this integration to Agent Policy 3 under which 2 agents are currently enrolled: one is the Windows server that we installed it on and another is the Fleet-Server machine itself (Linux).

• Oops we received an error stating xpack.encryptedSavedObjects.encryptionKey needs to be set.

• Use this command to generate encryption keys.

root@elastic:~# /usr/share/kibana/bin/kibana-encryption-keys generate

• This will generate three settings that we need to paste at the end of /etc/kibana/kibana.yml file.

Settings:
xpack.encryptedSavedObjects.encryptionKey: af6e821ff234ad2ff2c1b3c809b548bd
xpack.reporting.encryptionKey: 11b3f337563635cfa5401b51729af1e9
xpack.security.encryptionKey: 209d18e6c7cceb1fe5ce6bee2ab31ee6

• Let’s restart the kibana service with systemctl restart kibana .
• With this, you should be able to save and deploy changes.

Enable Detection Rules

One of the most impressive features is the vast library of pre-built detection rules. These rules cover the entire MITRE ATT&CK framework, helping you detect sophisticated attack patterns across Windows, Linux, and cloud environments.

Lets enable all the detection rules.

It has a wide range of detection rules for different sources. We are particularly interested in Linux and Windows sources.

Triggering a sample detection rule

Let’s see these detection rules in action with a simple test.

sudo useradd -m -s /bin/bash username

We can see the alert Linux User Account Creation being fired. Everything is working as expected.

Summary

After setting up and testing the ELK Stack with Fleet Server and Elastic Agents, I’m thoroughly impressed by its capabilities. The combination of robust endpoint security and comprehensive detection rules makes it an invaluable tool for system monitoring and security visibility. Whether you’re participating in a cyber defense competition or securing production systems, this setup provides the visibility and control you need.

References

• DigitalOcean: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-22-04
• Ippsec: https://www.youtube.com/watch?v=Ts-ofIVRMo4