Inside CPython: bytecode, memory and the interpreter
Through your own experiments you see how Python code turns into bytecode, how objects live in memory and how the interpreter runs.
Level 501 · 17 lessons · 21 practice labs
Lessons
The compilation pipeline
- Tokens and the abstract syntax tree
- Inspecting and transforming code with ast
- Bytecode: reading it with dis
- Code objects and .pyc files
- Short task: write your own lint rule with ast
The runtime
- The evaluation loop and frames
- Reference counting and the garbage collector
- How objects are laid out in memory
- Small integers and string interning
- The GIL and the free-threaded build
- Short task: find a reference-cycle leak
Extending the interpreter
- Import hooks
- Audit hooks: sys.addaudithook
- The limits of sandboxing Python
- Building CPython from source and debugging it
- Short task: an audit hook that records suspicious calls
Capstone
- Capstone: an analysis tool that inspects .pyc files
Practice labs
- Counting functions in a source file with ast: Write a tool that parses a source file with ast and lists function and class names with their line numbers.
- A lint rule for mutable defaults: Write a rule that finds parameters whose default is a list, dict or set with ast and pass the nested-function and lambda tests.
- Rewriting old calls with NodeTransformer: Write a transformer that converts deprecated function calls to a replacement with reordered arguments and show the output compiles and gives the same result.
- Comparing the bytecode of two loops: Extract instruction counts from the dis output of two equivalent functions and write a script reporting which operation is done extra.
- Extracting constants and names from a code object: Write a recursive tool that walks the nested code objects of a compiled function and collects string constants and global names.
- Validating a pyc header and listing its contents: Complete a tool that decodes a .pyc file's magic number, flags and source timestamp, reports a version mismatch and lists imported modules without running the code.
- Observing reference count changes: Complete tests that measure how an object's reference count changes on assignment, container insertion and deletion and compare with expected values.
- Curing a cache leak with weak references: Convert a registry that keeps objects alive forever to a WeakValueDictionary and verify with gc that the objects are collected.
- Hunting a reference-cycle leak: Trace accumulating objects in a long-running job with gc.get_referrers, find the callback forming the cycle, break it and measure that memory stays flat.
- Fixing an is versus == mix-up: Change an identity comparison that works for small numbers and breaks for large ones into a value comparison and write the test showing why.
- Reading the call chain from frames: Write a light tracer that collects caller names and lines with sys._getframe while releasing frame references so nothing leaks.
- Measuring the GIL's effect and picking the right tool: Measure a CPU-bound and an I/O-bound job with threads and processes, move the one threads slow down to a process pool and report the speed-up.
- Recording file opens with an audit hook: Write a hook with sys.addaudithook that captures open events and records path and mode, and show it does not loop on its own log file.
- Restricting modules with an import hook: Write a meta path finder that stops imports of modules on a deny list with a clear error and test that the hook can be removed.
- Retiring a restricted-eval sandbox: Document with tests why a formula evaluator assumed safe because it strips builtins falls short and replace it with an ast-based allow-list interpreter.
- Observing shared immutable objects: Write a report function that uses id to measure whether small integers, short strings and strings built at run time are the same object, and explain the results.
- Measuring the memory saved by sys.intern: Share strings with sys.intern in a record load where the same labels repeat hundreds of thousands of times, measure the memory difference and show with tests that comparisons stay correct using equality.
- Reading the running function's name from its frame: Write a helper that returns the name and line number of the function it is in using sys._getframe, and use it in log messages.
- A small coverage meter with sys.settrace: Collect the line numbers executed while a function runs using sys.settrace and report the lines that never ran, restoring the previous trace function afterwards.
- Measuring that threads do not speed up computation: Run the same computation with one and four threads, measure the times and write the result into a table.
- Observing the effect of the switch interval: Measure the response time of short jobs running beside a computing thread while changing sys.setswitchinterval, then restore the old value.
About this course
Through your own experiments you see how Python code turns into bytecode, how objects live in memory and how the interpreter runs.
What you will learn
- The compilation pipeline
- The runtime
- Extending the interpreter
- Capstone
Prerequisites
You should be experienced in the topics of the same path up to level 401.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 501; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.