Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Surveying the Big Data landscape and the pivotal role of Spark in contemporary data platforms
• Decoding Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the most effective approach for specific use cases
• Establishing and configuring SparkSession instances while grasping the fundamentals of application configuration

Module 2: PySpark DataFrames

• Reading from and writing to enterprise data sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames through transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing complex operations such as window functions, timestamp management, and nested data handling
• Integrating data quality checks and structuring reusable, maintainable PySpark codebases

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Mapping the relationship between Docker images and Kubernetes pods to Spark driver and executor processes, providing a conceptual view suitable for cluster users rather than administrators
• Tracing the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests versus limits, and the translation of session settings into cluster resources
• Differentiating between parameters controlled via Spark session settings and those managed by the platform team through cluster provisioning, and understanding why this distinction is critical for tuning
• Practical Exercise: Examine a live Spark-on-Kubernetes job in the sandbox environment to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing the trade-offs between executor count and size: evaluating memory versus cores and developing a rational framework for decision-making
• Allocating driver and executor memory, accounting for memory overhead, and mapping these configurations to Kubernetes pod resources
• Evaluating dynamic allocation: assessing its behaviour on Kubernetes, identifying scenarios where it conserves resources and those where it may not
• Practical Exercise: Execute identical jobs under varying executor and core configurations to benchmark runtime efficiency and resource usage

Module 5: Scaling Behaviour & Cost Optimisation

• Understanding how node scaling impacts job behaviour, completion latency, and overall resource consumption
• Comparing the performance and cost implications of using many small executors versus fewer, larger ones
• Optimising shuffle behaviour and partition tuning, including methods to estimate the cost impact of specific configuration choices
• Practical Exercise: Expand the sandbox cluster from five to ten nodes to observe and measure the effects on job completion time and resource efficiency

Module 6: Efficient Data Ingestion & Partitioning

• Analyzing the small-files problem: explaining why sources fragmented into numerous 1–5 MB Parquet files hinder performance and disrupt partitioning
• Implementing repartitioning and coalescing strategies to optimise data flow
• Regulating partition sizes during both read and write operations
• Writing Parquet files efficiently to prevent the recurrence of small-file issues in downstream processes
• Practical Exercise: Ingest a dataset consisting of many small Parquet partitions, apply distinct repartitioning strategies, and analyse performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognizing early warning signs
• Applying memory-efficient conversion patterns when transitioning data between Spark and Pandas
• Preventing memory spikes when serializing large datasets to CSV
• Utilizing chunked processing and dtype optimisation to manage resources in constrained environments
• Practical Exercise: Simulate a typical Pandas out-of-memory failure and resolve it through the application of chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Defining the positioning of Polars relative to Pandas: analyzing performance profiles, lazy evaluation, and memory characteristics
• Determining where Polars integrates with PySpark and Pandas within modern cloud data stacks and migration strategies, including AWS environments
• Practical Exercise: Refactor a Pandas-intensive transformation using Polars to compare memory efficiency and processing velocity

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles across a comprehensive, realistic ETL pipeline
• Addressing optimisation considerations unique to machine-learning workloads sharing the same cluster resources
• Executing a systematic tuning workflow to diagnose and resolve cost and performance issues
• Practical Exercise: Complete an end-to-end mini-project that includes data loading, transformation, and a basic model-training step, with participants independently tuning the Spark configuration