Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction, Learning Objectives, and Migration Strategy
- Defining course goals, aligning participant profiles, and establishing success metrics
- High-level migration methodologies and associated risk assessments
- Preparing workspaces, repositories, and lab datasets
Day 1 — Core Migration Concepts and Architecture
- Foundations of the Lakehouse, an overview of Delta Lake, and Databricks architecture
- Differences between SMP and MPP architectures and their impact on migration
- Designing the Medallion (Bronze→Silver→Gold) structure and an introduction to Unity Catalog
Day 1 Lab — Converting a Stored Procedure
- Practical exercise: migrating a sample stored procedure to a notebook
- Mapping temporary tables and cursors to DataFrame transformations
- Validating and comparing results against the original output
Day 2 — Advanced Delta Lake Features & Incremental Loading
- ACID transactions, commit logs, versioning, and time travel capabilities
- Auto Loader, MERGE INTO patterns, upserts, and schema evolution
- OPTIMIZE, VACUUM, Z-ORDER, partitioning strategies, and storage optimization
Day 2 Lab — Incremental Ingestion & Performance Optimization
- Implementing Auto Loader ingestion and MERGE workflows
- Applying OPTIMIZE, Z-ORDER, and VACUUM techniques; verifying outcomes
- Evaluating improvements in read/write performance
Day 3 — SQL in Databricks, Performance Analysis & Debugging
- Advanced SQL features: window functions, higher-order functions, and JSON/array manipulation
- Interpreting Spark UI, DAGs, shuffles, stages, tasks, and diagnosing bottlenecks
- Query optimization patterns: broadcast joins, hints, caching, and reducing spills
Day 3 Lab — SQL Refactoring & Performance Tuning
- Refactoring complex SQL processes into optimized Spark SQL
- Utilizing Spark UI traces to detect and resolve skew and shuffle issues
- Running before/after benchmarks and documenting tuning procedures
Day 4 — Applied PySpark: Replacing Procedural Logic
- Spark execution model: driver, executors, lazy evaluation, and partitioning strategies
- Converting loops and cursors into vectorized DataFrame operations
- Modularization, UDFs/pandas UDFs, widgets, and building reusable libraries
Day 4 Lab — Refactoring Procedural Scripts
- Refactoring procedural ETL scripts into modular PySpark notebooks
- Incorporating parametrization, unit-style testing, and reusable functions
- Conducting code reviews and applying best-practice checklists
Day 5 — Orchestration, End-to-End Pipelines & Best Practices
- Databricks Workflows: job design, task dependencies, triggers, and error handling
- Designing incremental Medallion pipelines with quality rules and schema validation
- Integrating with Git (GitHub/Azure DevOps), CI pipelines, and testing strategies for PySpark logic
Day 5 Lab — Constructing a Complete End-to-End Pipeline
- Assembling the Bronze→Silver→Gold pipeline orchestrated via Workflows
- Implementing logging, auditing, retry mechanisms, and automated validations
- Executing the full pipeline, verifying outputs, and preparing deployment documentation
Operationalization, Governance, and Production Readiness
- Unity Catalog governance, data lineage, and access control best practices
- Managing costs, cluster sizing, autoscaling, and job concurrency patterns
- Creating deployment checklists, rollback strategies, and runbooks
Final Review, Knowledge Transfer, and Next Steps
- Participant presentations showcasing migration work and key takeaways
- Gap analysis, recommended follow-up actions, and handover of training materials
- Providing references, further learning pathways, and support options
Requirements
- A solid grasp of core data engineering principles.
- Practical experience with SQL and stored procedures (e.g., Synapse / SQL Server).
- Familiarity with ETL orchestration concepts (such as ADF or similar tools).
Target Audience
- Technology managers with a background in data engineering.
- Data engineers looking to shift from procedural OLAP logic to Lakehouse patterns.
- Platform engineers responsible for leading Databricks adoption initiatives.
35 Hours
