Python performance: measure, speed up, extend with C
You find slow code by measuring, not guessing, and speed up Python programs with algorithms, data structures, caching and C extensions.
Level 401 · 18 lessons · 18 practice labs
Lessons
Measuring
- Fair benchmarks with timeit
- Profiling with cProfile and pstats
- Line profiling and live sampling with py-spy
- Memory profiling: tracemalloc and object size
- Short task: find the hotspot of a slow report
Speeding up within Python
- Complexity: getting rid of O(n²)
- How the choice of data structure affects speed
- Built-ins and letting C run the loop
- Caching and memoisation
- Copy-free data with array, bytes and memoryview
- Short task: make a script ten times faster
Extending with C
- Calling a C library with ctypes
- Writing bindings with cffi
- Compiling a hot loop with Cython
- A small extension module with the C API
- The risks of native code: memory errors and the GIL
- Short task: move a checksum to C and measure it
Capstone
- Capstone: measure, speed up and report on a data pipeline
Practice labs
- Measuring two joining approaches fairly: Measure two ways of joining strings under the same conditions with timeit and write a small script reporting repeat counts and best times.
- Finding the hotspot of a slow report: Profile Kestane Market's minutes-long stock report with cProfile, find the function eating most of the time and present the pstats evidence.
- Catching a hidden cost with a line profile: Find the expensive line with line_profiler in a loop that looks innocent at function level, hoist it out and document the speed-up with measurements while passing the tests.
- Comparing the memory footprint of containers: Write a function that deep-measures the memory of holding the same data in a list, tuple, set and dict and produce the table.
- Finding a growing cache with tracemalloc: Compare two snapshots in a service whose memory grows with every request, find the growing line and put an upper bound on the cache.
- Turning a load-everything job into a stream: Convert a summariser that loads a large CSV into a list to a generator stream and prove with tracemalloc that peak memory no longer depends on file size.
- Turning list membership into a set: Replace the list with a set in a filter that slows down while searching a blocklist and beat the time limit on a hundred thousand records.
- Reducing quadratic matching to one pass: Reduce code that searches same-day same-amount transactions with nested loops to one pass with a dictionary and show with random tests the results are unchanged.
- Making a script ten times faster: Measure Sarıkız Kargo's shipment matching script, make it at least ten times faster with data-structure and built-in changes and prove the output is byte-identical.
- Caching a postcode lookup with lru_cache: Wrap a slow district lookup function called repeatedly for the same postcode in lru_cache and show by test that the call count drops.
- Using memoryview for copy-free slicing: Make a parser that copies on every slice while splitting a large buffer into frames copy-free with memoryview and measure the allocation count.
- Repairing a cache that returns stale data: Add expiry and per-key invalidation to a cache that returns the old value after a price update and pass both the freshness and hit-rate tests.
- Calling a C function with ctypes: Call the summing function in a small prebuilt C library through ctypes with correct argtypes and restype declarations.
- Compiling a hot loop with Cython: Convert a pure-Python distance loop to typed Cython, build it and measure at least a fivefold speed-up with identical results.
- Closing a buffer overrun in a native call: Add the length check and fix the buffer lifetime in a wrapper that passes a too-small buffer to C through ctypes and corrupts memory, then show the crash test passes cleanly.
- Caching a function that takes a list: Derive a stable key for a report function that cannot be cached because it takes list and dictionary arguments, add a bounded cache and show the hit rate in the test.
- Accumulating chunks with bytearray: Change a slow loop that joins incoming chunks by repeatedly adding bytes objects to use a bytearray, and measure the difference.
- Processing a large file with one buffer: Rewrite a file processor that allocates a new bytes object per read to reuse a single buffer with readinto and memoryview; tests check the allocation count and the result.
About this course
You find slow code by measuring, not guessing, and speed up Python programs with algorithms, data structures, caching and C extensions.
What you will learn
- Measuring
- Speeding up within Python
- Extending with C
- Capstone
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.