Python iterators, generators and functional tools
You process large data without filling memory, in short readable code, using the iteration protocol, generators, itertools and functools.
Level 201 · 17 lessons · 15 practice labs
Lessons
Iteration
- The iteration protocol: iter and next
- Writing your own iterator class
- Generators and yield
- Generator expressions and lazy evaluation
- Generator pipelines with yield from
- Short task: filter a 1 GB log without filling memory
itertools
- Infinite iterators: count, cycle, repeat
- Combinations and permutations: product, combinations
- groupby, chain, islice and batched
- Short task: group session records by user
Functional tools
- First-class functions and closures
- functools: partial and reduce
- Storing results with lru_cache and cache
- The operator module and sort keys
- Behaviour by type with singledispatch
- Short task: build a report pipeline from a chain of functions
Capstone
- Capstone: a streaming CSV summariser
Practice labs
- Merging sorted logs lazily: Merge several time-sorted log files with heapq.merge without loading them into memory and produce the ten busiest sources per minute.
- Writing a countdown iterator: Write an iterator class with __iter__ and __next__ that counts down from a number to zero and raises StopIteration at the end.
- Walking paged results one by one: Wrap a fake client that returns records page by page in an iterable class yielding single records, stopping cleanly after the last page.
- Finding an exhausted-iterator bug: Find why a report comes back empty the second time it is iterated and fix the class to return a fresh iterator each time.
- Reading a readings file with a generator: Write a generator function that reads a large readings file line by line and converts to numbers, skipping blank lines.
- Turning a list into a generator expression: Rewrite a total that collects every record in a list as a generator expression and show the memory difference with tracemalloc.
- A generator pipeline for an order stream: Build a four-generator pipeline that reads, parses, filters and totals Kestane Market's orders without any step building a list.
- Flattening a nested menu with yield from: Lazily yield every leaf of an arbitrarily nested menu structure with yield from, working for deep structures too.
- Dealing out shifts with cycle: Deal Narçiçeği Hospital's nurse list out over days with cycle and islice and return the first fourteen days.
- Generating fixtures with combinations: Generate a fixture list where every pair of teams meets once using combinations and verify the number of matches.
- Remembering to sort before groupby: Fix a groupby report that lists the same city several times and produce one total line per city.
- Sending records in batches of a hundred: Split a possibly endless record stream into batches of a hundred with batched and chain, sending the last short batch too.
- Making ready converters with partial: Produce km-to-mile and kg-to-pound converters from one general conversion function with partial.
- Caching a rate lookup with lru_cache: Add lru_cache to a slow rate lookup, fix the error caused by a list argument and prove the call count with a test.
- Splitting export by type with singledispatch: Split an export function written as an isinstance chain with singledispatch and register a handler for a new date type.
About this course
You process large data without filling memory, in short readable code, using the iteration protocol, generators, itertools and functools.
What you will learn
- Iteration
- itertools
- Functional tools
- Capstone
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.