Binary data and protocol parsing with Python
You think at the byte level, parse binary file formats and network protocols safely and harden your own parser with fuzzing.
Level 501 · 16 lessons · 15 practice labs
Lessons
The byte level
- bytes, bytearray and hex notation
- Byte order and struct formats
- Bit fields and flags
- Variable-length integers and TLV encoding
- Short task: parse a file header
File formats
- Reading PNG chunks
- Reading an ELF header with Python
- Parsing the pcap file format by hand
- Declarative parsing: construct
- Short task: a file type detector based on magic bytes
Protocols and robustness
- Decoding a DNS message from its bytes
- Writing a client for a documented custom protocol
- Defensive parsing: lengths, bounds and resource limits
- Fuzzing a parser with atheris
- Short task: fix the crashes the fuzzer found
Capstone
- Capstone: a complete parser and test suite for a custom protocol
Practice labs
- Producing hex dump lines: Write a function that dumps a byte string into sixteen-byte lines with offset, hex and printable-character columns.
- Decoding a file header with the right byte order: Decode a header holding version, record count and timestamp with struct and fix the existing code that yields nonsense through the wrong byte order.
- Reading a truncated record file safely: Harden a tool that reads fixed-size records with iter_unpack so it does not crash on files whose last record is cut short and reports the missing byte count.
- Extracting permissions from a flag byte: Write functions that decode and re-encode read, write, delete and share flags packed in one byte using IntFlag.
- A variable-length integer codec: Write encode and decode functions for variable-length integers in seven-bit groups and pass the round-trip and overlong-input tests.
- Parsing nested TLV with limits: Parse nested type-length-value records and stop with a clear error when a declared length exceeds the remaining data or the depth limit is passed.
- Recognising file types from magic bytes: Write a function that recognises PNG, PDF, ZIP, GZIP and ELF files from their first bytes without trusting the extension.
- Verifying PNG chunks and their CRCs: Walk the chunks of a PNG file in order, verify each chunk's CRC and report the type and offset of a corrupt one.
- Reading a pcap file by hand: Determine byte order from the global header of a prepared capture file, walk the packet records and count the five busiest IPv4 pairs.
- Decoding a DNS header from bytes: Write a function that decodes the id, flags and record counts from the twelve-byte DNS header and maps the response code to a name.
- Decoding compressed DNS names: Decode domain names compressed with pointers and raise an error instead of looping forever on a pointer that points back at itself.
- Writing a client for a documented custom protocol: Write a client for the binary protocol the lab target speaks that performs the handshake, requests and checksum verification from the specification.
- Checking a length field against the data: Add a bounds check to a message parser that blindly trusts the declared length and return a domain error instead of IndexError on short input.
- Describing a header declaratively with construct: Convert hand-written struct calls to a construct definition and show by test that parsing and rebuilding gives the same bytes.
- Closing the crashes the fuzzer found: Write an atheris harness for your own parser, turn the crashing inputs into regression tests and close them all by adding memory and time limits.
About this course
You think at the byte level, parse binary file formats and network protocols safely and harden your own parser with fuzzing.
What you will learn
- The byte level
- File formats
- Protocols and robustness
- Capstone
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal in the browser, connected to a private lab network. Nothing to install.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.