Designing a security data platform
You design a platform that collects, stores, processes and serves all of an organisation's security data, from requirements and architecture to cost and governance, and build it at small scale.
Level 501 · 17 lessons · 15 practice labs
Lessons
Requirements
- Requirements: sources, volume, latency, retention
- A use-case catalogue: detection, hunting, compliance
- Build or buy
- Quick task: a platform requirements document
Architecture
- Reference architecture: ingestion, streaming, lake, serving
- Choosing a common event schema
- Hot, warm and cold tiers
- Query engines and access paths
- Quick task: an architecture diagram and decision records
Building
- Building the ingestion layer
- The processing and enrichment layer
- Serving: queries and dashboards
- Quick task: an end-to-end data test
Governance and cost
- Platform governance: ownership, access, retention
- Capacity and cost model
- Resilience and disaster recovery
- Capstone: a security data platform for a mid-sized organisation
Practice labs
- Draw up the source inventory: Write a mid-sized organisation's ten log sources with volume, latency need and retention to sources.csv.
- Turn use cases into requirements: Turn six given use cases for detection, hunting and compliance into latency, retention and query-type requirements.
- A build-or-buy decision: Score two options on five-year cost, team skills and lock-in risk and write a justified decision record.
- Map an event to the common schema: Map login events from three different sources to the chosen common schema's fields and point out the required fields left empty.
- A hot, warm and cold tier plan: Place each data type into one of three tiers by query frequency and write the transition times between tiers to tiers.csv.
- Choose the access path by query: Run five typical queries against the stream, lake and serving layers and prove the best engine for each with time and cost.
- Connect a new source to ingestion: Connect a firewall source to a Kafka topic and verify end to end that events are also written to the raw bucket.
- Add an enrichment step: Write an enrichment job in the processing layer that adds asset owner and network zone and report the share of events with no match.
- An end-to-end data test: Send marked test events at the source, measure count and delay at every layer and find where the loss occurs.
- A data ownership table: Assign an owner, responsible team and sensitivity class to every data set on the platform and list those left unowned.
- A role-based access model: Define and test bucket, topic and table permissions for analyst, hunter, engineer and auditor roles.
- Enforce retention and deletion across layers: Enforce a data type's retention consistently on topic, bucket and table and prove expired data remains in no layer.
- A storage cost model: Build a table that computes three-year storage cost from daily volume, compression ratio and tier prices.
- Remove a single point of failure: Find the single-copy component in the architecture, make it redundant and show the flow continues when it is shut down.
- A disaster recovery drill: Delete the serving layer, rebuild it from the raw layer, measure the recovery time and compare it with the target.
About this course
You design a platform that collects, stores, processes and serves all of an organisation's security data, from requirements and architecture to cost and governance, and build it at small scale.
What you will learn
- Requirements
- Architecture
- Building
- Governance and cost
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.