Debugging and logging in Python
You read tracebacks, step through code with pdb, produce structured, leak-free logs with the logging module and diagnose problems in production.
Level 201 · 17 lessons · 18 practice labs
Lessons
Finding bugs
- Reading tracebacks: chained exceptions
- print or a debugger? breakpoint()
- pdb: breakpoints, stepping, moving through the stack
- Post-mortem debugging: pdb.pm and faulthandler
- The warnings module and deprecation warnings
- Short task: fix a script that gives the wrong total
Logging
- logging: levels, handlers and formatters
- Configuring logging with dictConfig
- Rotating log files
- Structured JSON logs
- Keeping passwords and personal data out of logs
- Short task: tidy up the logs of a noisy service
Diagnosing in production
- Tracing calls with sys.settrace and the trace module
- Finding memory growth with tracemalloc
- A hung process: stack dumps and timeouts
- Short task: find from the logs why a job slowed down
Capstone
- Capstone: diagnose and fix a crashing background job
Practice labs
- Tracing a job that fails now and then: Add targeted DEBUG records to a batch job that produces a wrong total only for some inputs, catch the bad record with pdb and prove the fix with a test.
- Finding the root cause in a chained traceback: Read a three-level chained traceback, find the line where the original error occurred and fix it.
- Tracing a wrong discount with breakpoint(): Inspect variables with breakpoint() in basket code that applies a discount twice and fix the faulty line.
- Catching a bad record with a conditional breakpoint: Catch a conversion that breaks on only one of ten thousand records with a pdb conditional breakpoint and fix the cause.
- Examining a crashed job post-mortem: Examine a job that crashes from deep recursion with faulthandler and pdb.pm and fix it by turning recursion into a loop.
- Moving print calls to logging: Turn a script's print calls into logging calls at fitting levels, with warnings going to a file and errors to stderr too.
- Setting log levels per module: Write a dictConfig that sets a noisy module to WARNING and your own to DEBUG without duplicated lines.
- Rotating a log file by size: Set up logging with RotatingFileHandler that rolls at one megabyte and keeps three backups, never exceeding the file limit in the test.
- A structured JSON log formatter: Write a Formatter that emits time, level, message and extra fields as one-line JSON, including exception traces as a field.
- Filtering passwords out of logs: Write a logging.Filter that masks password= and token= values and apply it to every handler.
- Finding lines that log personal data: Find the log calls in Defne Finans's payment module that write id and card numbers, mask them and verify with a test.
- Thinning out repeated error lines: Add a filter to a service that logs the same error hundreds of times a second so repeats are counted and summarised once a minute.
- Finding the source of growing memory: Compare tracemalloc snapshots to find the list that grows on every request and close the leak.
- Taking a stack dump of a hung process: Add a signal-triggered stack dump with faulthandler to an unresponsive job and find where it is waiting.
- Finding the slow stage from step timings: Extract step timings from a data transfer job's logs, identify the stage that slowed down and remove the redundant repeated query.
- Bringing back a swallowed traceback: In a job where a broad except block prints only a short message, record the full traceback and re-raise the exception, then use the traceback to find and fix the faulty line.
- Rotating a log file daily: Add a timed rotating handler to an application that writes to a single file so it rotates at midnight and keeps seven days of backups.
- Compressing rotated logs and deleting old ones: Customise the rotating handler to gzip rotated files and delete the oldest archives beyond a total size limit, without losing records while writing.
About this course
You read tracebacks, step through code with pdb, produce structured, leak-free logs with the logging module and diagnose problems in production.
What you will learn
- Finding bugs
- Logging
- Diagnosing in production
- Capstone
Prerequisites
Finishing the level 101 courses of the same path, or knowing those topics, is enough.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 201; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.