Detection engineering with Python: Sigma and YARA
You write a small engine that runs Sigma-like rules, automate file scanning with yara-python and protect your detections with tests.
Level 301 · 16 lessons · 15 practice labs
Lessons
Detection as rules
- The detection lifecycle: idea, rule, test, upkeep
- Reading the Sigma rule format
- Writing a small engine that runs YAML rules
- Field mapping: fitting a rule to different logs
- Short task: write and run a rule for a suspicious PowerShell command
File detection with YARA
- YARA rules: strings, conditions, modules
- Scanning folders and process memory with yara-python
- Fast YARA rules with few false positives
- Short task: catch a malware family’s samples with one rule
Testing detections
- Building test data from good and bad samples
- A unit test for every rule
- Mapping coverage with MITRE ATT&CK
- Validating rules automatically on every change
- Short task: fix a noisy rule without breaking its tests
Wrap-up
- Producing alerts in a format the SIEM understands
- Capstone: ship a tested pack of detection rules
Practice labs
- A rule for encoded PowerShell: Write a Sigma rule that catches PowerShell started with an encoded command, and show your small engine flags the right lines in the given sample events.
- One rule, two different logs: Run one Sigma rule on both Sysmon and Linux auditd logs through a field-mapping table, and show it finds the expected event in both sources.
- The rule engine ignores modifiers: Fix a rule engine that ignores the contains, startswith and all modifiers, and show with tests that three previously wrong rules now match correctly.
- One rule for a family: Find the common strings in the given family of training samples and write one YARA rule that catches all of them without touching clean files.
- A folder scan report: Write a yara-python script that scans a copy of the Kuzey Logistics file server and reports matching rule, file path and SHA-256 as JSON.
- A slow, noisy YARA rule: Rewrite a rule that is slow and noisy because of broad regexes and short strings, measure the scan time and show zero matches on clean samples.
- A test for every rule: Write pytest tests with one good and one bad sample for every rule in the pack, and add a check showing no rule is left untested.
- Fix a noisy rule without breaking its tests: Narrow a rule that fires hundreds of times a day because of admin scripts, and show by measurement that all existing tests still pass and noise has dropped.
- Keep broken rules out of release: Write a validation script that checks syntax, required fields and tests on every change to the rule repository, and show a deliberately broken rule is rejected.
- An ATT&CK coverage table: Write a script that builds a MITRE ATT&CK coverage table from the rule pack’s tags, and list tactics with no rules at all.
- Alerts the SIEM cannot read: Convert the alerts your rule engine produces into the given common event schema, and show the sample SIEM import script accepts them without errors.
- Release a tested rule pack: Prepare a pack of five Sigma and two YARA rules for Copperleaf Insurance, each tested, versioned and with change notes, and run it through the validation script.
- Prepare good and bad samples for a rule: From a raw log file, pick five events that should match and five that should not for a suspicious command-line rule, write them to a labelled test file and show the rule classifies all ten correctly.
- Does the rule miss small variations?: Write a script that produces variations of the bad samples with changed letter case, parameter order and extra spaces, test the rule pack against them and give a table of which rule misses which variation.
- How many alerts does an ordinary week produce?: Run a week of clean lab logs from Yelkovan Bankası through the rule pack and measure each rule’s daily alert count, rank the noisy rules above the threshold and add a check that makes every new rule pass this measurement before release.
About this course
You write a small engine that runs Sigma-like rules, automate file scanning with yara-python and protect your detections with tests.
What you will learn
- Detection as rules
- File detection with YARA
- Testing detections
- Wrap-up
Prerequisites
You should be comfortable with the level 101 and 201 topics of the same path.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 301; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.