Azure Data Bricks(intermediate) Syllabus

1. Spark Internals & Execution Architecture

• Spark Fundamentals: Spark Architecture, Driver, Executors & Cluster Manager
• Distributed Processing: Partitions, Repartition vs Coalesce & Shuffle Operations
• Execution Engine: Lazy Evaluation, DAG Flow, Catalyst Optimizer & Tungsten Engine
• Monitoring: Spark UI Analysis & Execution Insights

2. Spark Performance Optimization

• Memory Optimization: Cache, Persist, Unpersist & Storage Levels
• Performance Tuning: Partition Pruning, Data Skew & AQE
• Join Optimization: Broadcast Join Strategies
• Troubleshooting: Small File Problems & Performance Analysis

3. Python for ETL Development

• Core Programming: Control Flow, Conditional Logic & Loops
• File Operations: File Handling & Utility Functions
• Error Management: Exception Handling, Try-Except-Finally & Logging

4. Advanced Data Formats Processing

• File Processing: CSV, JSON, XML & Excel Files
• Complex Data Handling: Nested JSON & Schema Evolution
• Transformations: explode(), arrays_zip() & Structure Flattening

5. Delta Lake Fundamentals

• Delta Architecture: Delta Lake, ACID Transactions & Transaction Logs
• Table Management: Managed vs External Tables
• Data Operations: INSERT, UPDATE, DELETE & MERGE Commands

6. Advanced Delta Lake Optimization

• Data Management: Time Travel, Restore & Vacuum Operations
• Optimization Techniques: Partitioning, Z-Ordering & Liquid Clustering
• Performance Tuning: Data Skipping, OPTIMIZE & Auto Compaction

7. Slowly Changing Dimensions & Data Modeling

• Warehouse Design: Fact Tables, Dimension Tables & Star Schema
• Data Modeling: Snowflake Schema Design
• Historical Tracking: SCD Type 1, SCD Type 2 & MERGE-Based Frameworks
• Advanced Concepts: Hash Key Implementation & History Management

8. Auto Loader & Incremental Processing

• File Ingestion: Auto Loader Fundamentals & Incremental Processing
• Schema Management: Schema Inference & Evolution
• Processing Modes: Notification Mode & Directory Listing Mode
• Pipeline Design: COPY INTO, MERGE Strategy & Idempotent Pipelines

9. Databricks Architecture & Cluster Optimization

• Compute Resources: Job Clusters, All-Purpose Clusters & Serverless Compute
• Infrastructure: ADLS Integration & Cluster Sizing Strategies
• Performance Optimization: Memory Tuning & Core Allocation
• Architecture Design: Medallion Architecture & Bronze-Silver-Gold Layers

10. Logging, Monitoring & Audit Framework

• Monitoring Frameworks: ETL Logging & Audit Table Design
• Error Management: Error Handling Frameworks & Monitoring Strategies
• Production Support: Alerting Mechanisms & Real-Time Support Scenarios

11. CI/CD & Deployment Automation

• Version Control: Git Integration & Azure DevOps Fundamentals
• Deployment Frameworks: Databricks Asset Bundles & CI/CD Pipelines
• Release Management: Multi-Environment Deployments & Best Practices

12. Real-Time Industry Projects

• Project 1: Enterprise Delta Lake Optimization Project
• Project 2: Auto Loader Incremental Processing Framework
• Project 3: SCD Type 2 Historical Data Pipeline
• Project 4: Medallion Architecture Implementation
• Project 5: Production Logging & Audit Framework
• Project 6: End-to-End Azure Databricks Optimization Project

13. Advanced Interview Preparation

• Architecture Questions: Databricks & Spark Architecture Scenarios
• Optimization Topics: AQE, Broadcast Joins & Performance Tuning
• Delta & Auto Loader: Delta Lake & Auto Loader Interview Questions
• Career Readiness: Mock Interviews & Production Support Discussions

14. Course Highlights

• Spark Internals & Execution Plan Analysis
• Delta Lake Optimization & Performance Tuning
• Auto Loader Frameworks & Incremental Processing
• SCD Type 1 & Type 2 Implementation
• Medallion Architecture & Enterprise Data Modeling
• CI/CD with Azure DevOps & Deployment Automation
• Production Support Knowledge & Real-Time Projects
• Interview Preparation, Resume Building & Career Guidance