Structured data on the command line: jq, yq, CSV and XML
Instead of hacking at JSON, YAML, CSV and XML with grep, you query, convert and validate them with the right tools and handle API output and configs safely.
Level 401 · 17 lessons · 15 practice labs
Lessons
JSON and jq
- The shape of JSON and why grep is not enough
- jq filters: fields, arrays and paths
- select, map and conditional picking
- Transforming and grouping: group_by and reduce
- Editing a JSON file safely with jq
- Processing JSON-per-line logs
- Short task: list users about to expire from an API response
YAML
- YAML pitfalls: indentation, types and the "Norway problem"
- Querying and editing with yq
- Short task: update the same key in twenty YAML files
CSV and XML
- Reading CSV with the right tool: quotes and embedded commas
- Querying and converting with Miller and csvkit
- Querying XML with xmllint and XPath
- Converting between formats: JSON, YAML, CSV
- Short task: pull servers on old versions out of an XML inventory
Wrap-up
- Validating against a schema and rejecting bad input
- Capstone: merge an inventory held in three formats into one report
Practice labs
- List users about to expire from an API response: Use jq on a saved API response to extract the name and end date of users whose accounts expire next week.
- Group errors by service in a JSON-lines log: In a log with one JSON object per line, group error-level records by service and give their counts.
- Update a JSON config without breaking it: Change two fields of the array element with a given name using jq and write the file atomically via a temp file.
- Pull one value from nested JSON: Find each service's listening port in a nested config JSON with jq paths and print service-port pairs.
- Turn a JSON array into CSV with jq: Select name, IP and operating system from a server inventory JSON and produce a CSV with a header, using empty values for missing fields.
- Total usage per tenant with reduce: Compute total storage and request counts per tenant from a billing dump with reduce and print in descending order.
- Fix a YAML value read as the wrong type: Find the lines where values like a country code or version number are read as a boolean or number and quote them.
- Update the same key in twenty YAML files: Update the image version key in twenty service definitions with yq, leaving comments and other fields unchanged.
- Find a setting ignored because of indentation: Prove with yq that a setting the application ignores sits at the wrong level in otherwise valid YAML, and fix it.
- Read a CSV with quoted commas correctly: Extract the address column from a CSV that cut splits wrongly using the right tool, and confirm the row count is kept.
- Summarise departments with Miller: Compute device count and average age per department from an asset CSV with Miller and print it as a table.
- Pull servers on old versions from an XML inventory: Use XPath to list the names of servers in an XML inventory whose OS version is below the threshold.
- Convert a JSON inventory to CSV: Convert a JSON server list to CSV with a header row, leaving missing fields empty.
- Reject records that break the schema: Check incoming JSON records against required-field and type rules and split failures into a separate file with a reason.
- Merge an inventory in three formats into one report: Merge inventory pieces held as JSON, YAML and CSV by host name and report the conflicting fields.
About this course
Instead of hacking at JSON, YAML, CSV and XML with grep, you query, convert and validate them with the right tools and handle API output and configs safely.
What you will learn
- JSON and jq
- YAML
- CSV and XML
- Wrap-up
Prerequisites
You should have practised the topics of the same path up to level 301.
Lab environment
Lessons run in your own Kali Linux terminal that opens in the browser. Nothing to install; the lab starts clean in every lesson.
Frequently asked questions
- Is this course right for my level?
- The course is at level 401; it assumes you know the topics of the lower-level courses.
- Is there anything to install?
- No. Every lesson runs in your own lab environment that opens in the browser.
- How do lessons work?
- Each lesson has 8 to 10 steps and a short quiz. Early steps show the command to type, later steps give less help; if two attempts fail, the solution opens.
- Can I try it for free?
- Yes. The first three lessons of every course are free.