Senior Consultant Databricks Pyspark Mumbai Engineering Data Modernization & Migration
Deloitte India (South Asia) · Consulting & Professional Services
- Mumbai, India
- Hybrid
- Posted today
- Data & AI
About the job
Job Title: Databricks - Sr. Consultant
Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.
Experience:
5–7 years of experience in data engineering, with deep expertise in Azure Databricks and Apache Spark.
Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).
Strong experience with Unity Catalog for data governance and security.
Key Responsibilities: Technical Skills:
Design and implement scalable Medallion Architectures using Delta Live Tables (DLT) and Databricks.
Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
Implement enterprise-grade data governance and access control using Unity Catalog.
Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.
Soft Skills:
Strong leadership and ability to drive technical delivery.
Advanced problem-solving skills for complex architectural challenges.
Excellent communication skills for interacting with stakeholders and technical leads.
Preferred Qualifications:
Bachelor’s or Master’s degree in a technical field.
Databricks Certified Data Engineer Professional is highly desirable.
Experience with DataOps and Infrastructure as Code (Terraform).
Job Details:
Location: Mumbai (Hybrid)