Why build a SIEM instead of using a commercial oneBuilding a SIEM from scratch with the Elastic Stack gives you something no commercial license can: a deep understanding of every component. You know how logs are collected, indexed, queried, and visualized. This knowledge makes you a much more effective security professional — even if you later use a commercial platform.ELK Stack architecture: the four componentsElasticsearch — the search and analytics database, indexing all logs as JSON documents. Kibana — the visual interface for exploring data, writing KQL queries, building dashboards, and configuring alerts. Logstash — the processing engine that receives raw logs, transforms them, and sends them to Elasticsearch. Filebeat — the collection agent that runs near log sources and reads/ships them. The data flow is: Source → Filebeat → Logstash → Elasticsearch → Kibana.Docker Compose: the entire environment in one fileDocker Compose elegantly solves the ELK configuration complexity. With docker compose up, the entire stack is operational in minutes. Elasticsearch is accessible internally at http://elasticsearch:9200, Kibana on port 5601 of the host.Log generation script: simulating real eventsTesting the SIEM requires realistic logs. I wrote a script that continuously generates simulated events: successful and failed SSH connections, application logins, HTTP errors, system events. The script generates events at a constant rate with some randomization: some failed SSH attempts per minute (simulating brute force), legitimate logins distributed throughout the day, sporadic application errors. This approach — simulating traffic before connecting real sources — is essential for developing and testing KQL queries without waiting for organic events.Filebeat and the Dissect Processor: structuring raw logsThe Dissect Processor transforms unstructured text lines into separate fields. Instead of a single message field with all the text, you get distinct fields: timestamp, level, service, event_type, ip. This structuring is fundamental for writing effective KQL queries — on separate fields you can filter precisely for app.level:ERROR.KQL: Kibana Query LanguageKQL is simpler than Elasticsearch's native query DSL, optimized for interactive use in Kibana. I use queries like: app.level:ERROR for all errors, message:"Failed password" AND app.ip:192.168.1.* for failed SSH attempts from a subnet, (app.level:ERROR OR app.level:WARN) AND NOT service:monitoring for filtered alerts. Real-time event simulation is particularly useful during learning: write a query, observe results while new events are generated, adjust the query.Kibana Lens: security monitoring dashboardsFor daily monitoring I build: event counter by level (ERROR/WARN/INFO in last 24h), error timeline (bar chart by hour to see anomalous peaks), top SSH failure source IPs (table sorted by failed attempts to identify brute force), hourly heatmap (activity by hour of day and day of week to identify anomalous patterns). A real-time updated dashboard transforms the SIEM from a post-incident analysis tool to a preventive monitoring system.What you learn building a SIEM from scratchBuilding this system gave me practical understanding no theoretical certification could: why log structuring matters more than volume, how false positives are generated by overly permissive queries, why timestamp normalization is critical for correlating events from different sources, and how to calibrate alerts to be useful without creating alert fatigue. These aren't abstract concepts — they're real problems I encountered and solved.