Data analysis and big data

38 courses, 642 lessons. A real Kali Linux lab in the browser.

Introduction to data analysis (Level 101, 18 lessons): What data is, how to ask questions of it and how to read a table; you run your first analysis on security records at the command line and turn it into a short report.
Data analysis at the command line (Level 101, 17 lessons): You filter, count, join and summarise CSV and JSON data with cut, sort, uniq, awk, jq and csvkit, without installing anything heavy.
Statistics basics for analysts (Level 101, 17 lessons): You compute means, medians, spread, percentiles and rates on real event data and learn when numbers mislead.
Python and Jupyter for data analysis (Level 101, 17 lessons): You learn to work in a Jupyter notebook, read records with Python's csv, json and datetime modules and turn your analysis into a repeatable notebook.
Introduction to big data (Level 101, 17 lessons): You learn why data outgrows one machine, the core ideas of distributed storage and processing, batch versus streaming, and the parts of a modern data architecture.
Numerical computing with NumPy (Level 201, 17 lessons): You learn to process large sets of numbers quickly and compute simple statistics over event counts.
Data analysis and log review with Pandas (Level 201, 22 lessons): You read, clean and summarise tabular data and turn security logs into meaningful reports.
Data visualisation with Matplotlib (Level 201, 17 lessons): You turn data into readable charts and prepare visuals that explain security events to managers.
Hadoop and HDFS: distributed storage and batch processing (Level 201, 19 lessons): You store log data too big for one machine in HDFS, protect it with permissions and process it in batch jobs.
SQL over big data with Hive (Level 201, 18 lessons): You query files in HDFS with SQL, partition them and turn them into report tables.
Data cleaning and preparation (Level 201, 18 lessons): You find dirty, missing and inconsistent data, standardise formats, times and identifiers, document the cleaning and write a repeatable preparation script.
Parsing logs and text data (Level 201, 19 lessons): You split syslog, web server, JSON and Windows event logs into fields with regular expressions and Python, map them to a common schema and get them ready for analysis.
Statistics 201: distributions, correlation and tests (Level 201, 16 lessons): You answer "is this difference real?" with probability, distributions, confidence intervals, hypothesis tests and correlation.
Analytical SQL with DuckDB (Level 201, 19 lessons): With server-free DuckDB you query CSV, JSON and Parquet files directly and analyse logs with window functions, CTEs and time bucketing.
Reporting and storytelling with data (Level 201, 16 lessons): You summarise analysis for its audience, choose the right chart, write reports in a finding–evidence–recommendation structure and produce an automated weekly report.
Machine learning and anomaly detection with Scikit-learn (Level 301, 20 lessons): You build and evaluate simple models that learn from data and use them to find unusual behaviour in security data.
Event streaming with Apache Kafka (Level 301, 18 lessons): You collect log and event streams with Kafka, replay them and consume them securely.
Big data processing with Apache Spark (Level 301, 21 lessons): You read, clean and join large event data with PySpark and turn it into reusable batch jobs.
Advanced Pandas: reshaping and performance (Level 301, 17 lessons): You analyse millions of log rows with multi-level indexes, reshaping, window calculations, chunked processing and memory and speed tuning.
Advanced visualisation: seaborn, Plotly and dashboards (Level 301, 18 lessons): You make security data explorable with statistical charts in seaborn, interactive visuals in Plotly and a small dashboard app.
Time series analysis of event data (Level 301, 16 lessons): You turn event counts into time series, find seasonality, trends, spikes and regular-interval communication (beaconing), and set expected levels with simple forecasts.
Data pipelines with Airflow (Level 301, 17 lessons): Starting from ETL and ELT, you build scheduled, retryable, observable and secure data pipelines with Apache Airflow.
Data quality and validation (Level 301, 16 lessons): You measure the dimensions of data quality, write schema and business-rule tests, put quality gates into pipelines and trace quality problems to their root.
Security log analytics (Level 301, 18 lessons): You analyse authentication, endpoint, network, DNS, proxy and web records with Pandas and SQL, and find traces of brute force, impossible travel, lateral movement and exfiltration.
Data lake design (Level 301, 16 lessons): You design a data lake with raw, clean and serving layers and store security data neatly and cheaply with object storage, Parquet, partitioning, a catalogue and access policies.
Hadoop cluster operations and security (Level 301, 16 lessons): You operate HDFS and YARN and keep the cluster secure and running with high availability, resource queues, Kerberos, Ranger policies, audit logs and backups.
Case studies: investigating incidents with data (Level 301, 15 lessons): You investigate six realistic incident cases using data sets alone: frame the question, query the evidence, build a timeline and report your findings.
Advanced Spark: performance and stream processing (Level 401, 17 lessons): You read Spark's execution model in depth, tune shuffles, skew, memory and join strategies, and build stateful, windowed real-time detection with structured streaming.
Advanced Kafka: Connect, Streams and operations (Level 401, 16 lessons): You connect sources and sinks with Kafka Connect, produce contract-bound events with a schema registry, enrich with Kafka Streams, and run the cluster securely, observably and resiliently.
Advanced anomaly detection in security data (Level 401, 15 lessons): You turn user and entity behaviour into features, compare statistical, density and tree-based methods, and build an explainable detection pipeline that reduces alert fatigue.
Data analysis for threat hunting (Level 401, 17 lessons): With hypothesis-driven hunting you run stacking, frequency, time and relationship analysis over large log sets and turn findings into reusable detection rules and hunt reports.
Privacy and security in analytics (Level 401, 16 lessons): You minimise analysis data, apply masking, pseudonymisation, k-anonymity and differential privacy, measure re-identification risk and make access auditable.
Data pipeline security and operations (Level 401, 15 lessons): You threat-model data pipelines and protect, monitor and respond to incidents in them with secrets, identity, network, integrity and lineage controls.
Open table formats: Apache Iceberg (Level 401, 15 lessons): With Apache Iceberg tables you add transactional guarantees, schema and partition evolution, time travel and rollback to a data lake, and use table history for deletion requests and audits.
Job role capstone: security data analyst (Level 401, 12 lessons): You live a week as a security data analyst: a data request, cleaning, analysis, rule tuning, a dashboard, a report and peer review.
Designing a security data platform (Level 501, 17 lessons): You design a platform that collects, stores, processes and serves all of an organisation's security data, from requirements and architecture to cost and governance, and build it at small scale.
Detection at scale: streaming analytics and correlation (Level 501, 15 lessons): You run detection rules on streams of millions of events and build a scalable detection engine with stateful correlation, enrichment, scoring and detection as code.
Job role capstone: security data engineer (Level 501, 12 lessons): As a security data engineer you onboard a new log source, build the pipeline, add quality and privacy checks, fix a midnight failure and document the handover.

Projects

A store sales summary at the command line: You combine, filter and group the sales CSV files from Turnalı Perakende's stores with command-line tools, and produce a weekly summary table per store.
Clinic waiting times analysis in a notebook: You explore Lavinya Klinikleri's appointment records in a Jupyter notebook, work out the mean, median and spread per branch, and explain the findings with charts.
Convert a large CSV to Parquet and measure the difference: You convert Kuzeyyıldızı Kargo's shipment file of millions of rows to Parquet, measure the difference in size and query time, and write a short note on which format to choose and why.
Clean up messy customer records with Pandas: You clean Karaçam Mobilya's duplicated, incomplete and inconsistently formatted customer records with Pandas, and document each transformation and the number of rows affected in a cleaning log.
A traffic report from web server logs: You turn Selvikent Belediyesi's web server access logs into a table, work out the most requested pages, error rates and hourly load, and prepare a traffic report with charts.
Meter consumption analytics with DuckDB: You query Demirsu Enerji's meter readings in Parquet files with DuckDB, find daily consumption and unusual jumps using window functions, and export the result tables.
Compare two campaigns with a statistical test: You compare the results of Ayçiçeği Gıda's two campaign trials, choose and apply a suitable test, work out the confidence interval, and write a decision note for the managers.
Load call-centre tickets into HDFS and query them with Hive: You load Yakamoz Telekom's call-centre tickets into HDFS, define partitioned tables in Hive, and write queries giving the most common fault types and resolution times.
Prepare a monthly occupancy report for management: You prepare a monthly occupancy report from Mercanköşk Otelcilik's booking data, choose the right chart types, lead with the main message, and make the report reproducible with one command.
Anomaly detection in sign-in logs: You extract features from Yelkovan Bankası's prepared sign-in logs, train an anomaly model with Scikit-learn, evaluate the results against labelled examples, and report the suspicious sessions with reasons.
A daily data pipeline in Airflow with quality gates: You write an Airflow workflow that ingests, validates and loads Mavikaya Lojistik's daily shipment files, quarantine data that fails schema and value rules, and re-run a broken day.
A production line event stream with Kafka: You design Kafka topics for events from Ilgıntepe Tekstil's looms, write producer and consumer programs, scale with a consumer group, and show no data is lost when a consumer goes down.
A timeline and interactive dashboard from incident logs: You examine prepared logs from an outage at Saklıvadi Üniversitesi as a time series, find when the incident started and how it spread, and build an interactive dashboard that shows it.
Derive app usage sessions with Spark: You process Göktaşı Yazılım's app usage logs with PySpark in local mode, split clicks into sessions, work out session metrics, and write the result as partitioned Parquet.
A threat hunt over prepared organisation logs: You set up three hunt hypotheses over Lodoskaya Denizcilik's prepared authentication, proxy and endpoint logs, test them with DuckDB and graph analysis, and write a hunt report with the evidence.
Produce a privacy-preserving analysis set from student data: Before opening Gelincik Okulları's student achievement data for analysis, you pseudonymise and generalise it, measure the re-identification risk, and explain the measures in a privacy note.
Security audit and hardening of a data pipeline: You audit Yakamoz Telekom's data pipeline in the lab, made up of a workflow scheduler, object store and message queue, fix the exposed secrets, over-broad permissions and unencrypted connections, and write the audit report.
Detection with correlation rules on streaming data: You process Yelkovan Bankası's authentication and transaction events streaming from Kafka with Spark Structured Streaming, write two correlation rules, handle late events, and measure the delay under load.