Torna al blog
Cybersecurity

How I built a SIEM with Elastic Stack: from theory to security dashboard in Docker

A corporate SIEM in Docker Compose with Elasticsearch, Kibana, Logstash and Filebeat. In this article I share how I built a complete log collection and analysis system with KQL, event generation scripts and security dashboards — all on local containers.

Why build a SIEM instead of using a commercial one

Building a SIEM from scratch with the Elastic Stack gives you something no commercial license can: a deep understanding of every component. You know how logs are collected, indexed, queried, and visualized. This knowledge makes you a much more effective security professional — even if you later use a commercial platform.

ELK Stack architecture: the four components

Elasticsearch — the search and analytics database, indexing all logs as JSON documents. Kibana — the visual interface for exploring data, writing KQL queries, building dashboards, and configuring alerts. Logstash — the processing engine that receives raw logs, transforms them, and sends them to Elasticsearch. Filebeat — the collection agent that runs near log sources and reads/ships them. The data flow is: Source → Filebeat → Logstash → Elasticsearch → Kibana.

Docker Compose: the entire environment in one file

Docker Compose elegantly solves the ELK configuration complexity. With docker compose up, the entire stack is operational in minutes. Elasticsearch is accessible internally at http://elasticsearch:9200, Kibana on port 5601 of the host.

Log generation script: simulating real events

Testing the SIEM requires realistic logs. I wrote a script that continuously generates simulated events: successful and failed SSH connections, application logins, HTTP errors, system events. The script generates events at a constant rate with some randomization: some failed SSH attempts per minute (simulating brute force), legitimate logins distributed throughout the day, sporadic application errors. This approach — simulating traffic before connecting real sources — is essential for developing and testing KQL queries without waiting for organic events.

Filebeat and the Dissect Processor: structuring raw logs

The Dissect Processor transforms unstructured text lines into separate fields. Instead of a single message field with all the text, you get distinct fields: timestamp, level, service, event_type, ip. This structuring is fundamental for writing effective KQL queries — on separate fields you can filter precisely for app.level:ERROR.

KQL: Kibana Query Language

KQL is simpler than Elasticsearch's native query DSL, optimized for interactive use in Kibana. I use queries like: app.level:ERROR for all errors, message:"Failed password" AND app.ip:192.168.1.* for failed SSH attempts from a subnet, (app.level:ERROR OR app.level:WARN) AND NOT service:monitoring for filtered alerts. Real-time event simulation is particularly useful during learning: write a query, observe results while new events are generated, adjust the query.

Kibana Lens: security monitoring dashboards

For daily monitoring I build: event counter by level (ERROR/WARN/INFO in last 24h), error timeline (bar chart by hour to see anomalous peaks), top SSH failure source IPs (table sorted by failed attempts to identify brute force), hourly heatmap (activity by hour of day and day of week to identify anomalous patterns). A real-time updated dashboard transforms the SIEM from a post-incident analysis tool to a preventive monitoring system.

What you learn building a SIEM from scratch

Building this system gave me practical understanding no theoretical certification could: why log structuring matters more than volume, how false positives are generated by overly permissive queries, why timestamp normalization is critical for correlating events from different sources, and how to calibrate alerts to be useful without creating alert fatigue. These aren't abstract concepts — they're real problems I encountered and solved.

Condividi