MODULE LEAD - Data Modelling

Data Modeling: Strong proficiency in dimensional data modeling, including Star Schema and Snowflake Schema, with hands-on experience designing Retail data models for sales, transactions, products, stores, customers, inventory, pricing, promotions, and supply chain. Strong understanding of Delta Lake and Lakehouse architecture in Databricks.

Databricks & Lakehouse: Strong hands-on experience with Azure/AWS Databricks, Delta Lake, Unity Catalog, Databricks Workflows, notebooks, SQL Warehouses, and Medallion Architecture (Bronze, Silver, Gold). Experience implementing scalable and governed enterprise data platforms.

Distributed Computing & Spark: Strong understanding of distributed computing concepts using Apache Spark and PySpark, including DAGs, stages, tasks, transformations, actions, shuffling, serialization, caching, partitioning, broadcast joins, and partition pruning. Ability to analyze and optimize Spark execution plans.

SQL & Data Engineering: Advanced proficiency in SQL and Spark SQL, with strong experience developing complex transformations, joins, aggregations, window functions, CTEs, MERGE operations, SCD Type 1/Type 2, data quality validations, and incremental data processing for large-scale Retail datasets.

Data Engineering: Strong experience implementing large-scale, complex data engineering pipelines using Databricks, PySpark, SQL, Delta Lake, and AWS services. Experience integrating data from sources such as ERP, POS, e-commerce, inventory, product, pricing, and customer systems.

Performance Tuning: Proven ability to troubleshoot and optimize Databricks/Spark workloads by analyzing Spark UI, query execution plans, slow stages, data skew, shuffle, spill-to-disk, excessive file counts, small files, and partitioning issues. Strong knowledge of optimization techniques such as partition pruning, broadcast joins, AQE, OPTIMIZE, Z-ORDER, caching, cluster sizing, autoscaling, and Photon.

Delta Lake: Strong understanding of Delta Lake architecture and features, including ACID transactions, schema enforcement/evolution, time travel, MERGE, Change Data Feed (CDF), OPTIMIZE, VACUUM, and handling concurrent read/write operations.

Programming: Strong proficiency in Advanced SQL, Python, PySpark, and Spark SQL. Experience developing reusable frameworks, utilities, data quality frameworks, and scalable ETL/ELT components. Scala knowledge is an advantage.

Retail Data Domain: Good understanding of Retail business processes and data, including Store Sales, POS Transactions, Product/UPC, Store, Customer, Inventory, Purchase Orders, Pricing, Promotions, Assortments, and Supplier data. Ability to translate Retail business requirements into scalable analytical data models and Databricks pipelines.

Data Quality & Governance: Experience implementing data quality rules, validation frameworks, reconciliation, error handling, audit frameworks, and data lineage, with strong understanding of Unity Catalog, RBAC/ABAC, row-level and column-level security.

Production Engineering: Strong experience with Databricks Workflows, job orchestration, dependency management, monitoring, logging, retry mechanisms, CI/CD, Git, DAB, and production deployment practices.