Python and Jupyter for data analysis
You learn to work in a Jupyter notebook, read records with Python's csv, json and datetime modules and turn your analysis into a repeatable notebook.
Level 101 · 17 lessons · 15 practice labs
Lessons
The working environment
- Why Python is common in data analysis
- Virtual environments and packages
- Jupyter: first notebook, cells and kernel
- Taking notes in Markdown cells
- Quick task: an analysis notebook skeleton
Reading records
- Reading row by row with the csv module
- Reading events with the json module
- Parsing timestamps with datetime
- Counting with Counter and defaultdict
- Quick task: count event IDs
Tidy analysis
- Splitting analysis into functions
- Bad rows and error handling
- Writing results as CSV and JSON
- A reproducible notebook: order, seeds, versions
- Quick task: make a messy notebook run from the top
Sharing
- Secrets and personal data when sharing notebooks
- Capstone: an analysis notebook for VPN session logs
Practice labs
- Tidy up a notebook skeleton: Add title, purpose and data-source Markdown cells to a messy analysis notebook and put its cells in a sensible order.
- A virtual environment with pinned versions: Create a new virtual environment, pin the packages the analysis needs in requirements.txt and run the notebook on that kernel.
- A notebook with hidden state: Find the hidden state in a notebook that only works in a certain order and make it give the same result from a clean kernel, top to bottom.
- Sum row by row with the csv module: Read Defne University's library loan file with csv.DictReader and print the number of loans per faculty.
- Counting events in JSON lines: Use Counter to count event types in a JSON-lines file and print the five most common types and the number of broken lines.
- A per-user timeline with defaultdict: Build a time-ordered event list per user from shuffled events with defaultdict and write it to timeline.json.
- Turn date text into objects: Convert day/month/year record dates into datetime objects and find the oldest and newest record.
- Three different time formats: Write a function that converts ISO 8601, Unix time and Apache-format timestamps in the same file into one UTC format.
- The gap at a daylight-saving change: Find the missing and repeated hour in local-time records that span a daylight-saving change and compute event durations correctly.
- Turn repeated code into a function: Turn filter code copied three times into one function that takes parameters and show the results have not changed.
- A reader that survives bad rows: Write a reader that survives a file with broken rows in it and writes each bad row, with its line number, to a separate file.
- Write verifiable CSV and JSON results: Write the analysis result as both CSV and JSON and add a check that both carry the same record count and total.
- Find the password in a notebook: List every embedded password and key in a notebook about to be shared, including those in cell outputs.
- Move secrets into environment variables: Move the database password in a notebook into an environment variable, clear the outputs and make the notebook ready to share.
- Strip personal data from outputs: Mask the e-mail addresses and phone numbers visible in a notebook's output cells and prove the masking survives a re-run.
About this course
You learn to work in a Jupyter notebook, read records with Python's csv, json and datetime modules and turn your analysis into a repeatable notebook.
What you will learn
- The working environment
- Reading records
- Tidy analysis
- Sharing
Prerequisites
No prior knowledge is needed; the course starts from zero.
Lab environment
Lessons run on your own Kali Linux desktop in the browser, with the tools ready in their windowed interfaces. Nothing to install; a computer screen is recommended.
Frequently asked questions
- Is this course right for my level?
- The course is at beginner level; every term is explained where it first appears.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.