Big Data Processing

What it covers

• "Small data first": Unix pipelines (grep, sort, uniq, pipes) as a mental model before scaling up
• Functional programming fundamentals in Scala — immutability, pure functions, higher-order functions
• Distributed systems theory: replication, partitioning, consistency models, CAP theorem, consensus (Raft), vector clocks
• Batch processing with Apache Spark — RDDs, lineage-based fault tolerance, narrow vs. wide transformations, Spark SQL
• Stream processing with Apache Flink — event-at-a-time vs. micro-batch, windowing, event time vs. processing time
• Graph processing — Bulk Synchronous Parallel model, Pregel/GraphX

Assessment

Written exam (60%) + four mandatory lab assignments in pairs (40%): Functional Programming, Unix, Spark, Flink — via Weblab, strict deadlines, no late submissions. Exam is closed-book: multiple-choice (negative marking), open questions, and code-analysis questions. Both exam and lab grades need a minimum of 5.0 independently; a 4.0–5.7 lab average allows repairing one assignment.

Worth knowing

• No lecture slides or past exams are archived here — only assignment solutions and notes. For exam flavour, the LLM bundle's Exams.md is the best source.
• Reference text is Kleppmann's Designing Data-Intensive Applications — not included in this folder, but the course leans on it heavily.
• Spark internals (RDDs, lineage, shuffle) and distributed-systems theory dominate the exam; graph processing and Unix are comparatively minor.
• Assignments are pair work with strict, non-negotiable deadlines — late submissions score zero.