ConsultantSenior Consultant Databricks Coimbatore Engineering Data Modernization & Migration

Deloitte India (South Asia) · Consulting & Professional Services

Delhi, India
On-site
Posted today
Closes today
Data & AI

Apply on Deloitte India (South Asia)

About the job

Consultant/Senior Consultant | Databricks |Coimbatore | Engineering | Data Modernization & Migration

Job requisition ID : 111044

Location : Delhi

Entity : Deloitte Touche Tohmatsu India LLP

Job Title: Senior Databricks Engineer

Experience: 6-9 Years

Location: Mumbai

Employment Type: Full-Time

Job Summary

We are seeking an experienced Databricks Developer with 6-9 years of expertise in designing, building, and supporting modern Data Lakehouse solutions. The ideal candidate will have strong hands-on experience in Databricks, PySpark, Delta Lake, cloud platforms, and large-scale data processing. The role involves building scalable data pipelines, optimizing distributed data workloads, and enabling advanced analytics and AI-driven initiatives.

Key Responsibilities

Design, develop, and maintain scalable data engineering solutions using Databricks.

Build and optimize batch and real-time data processing pipelines.

Develop ETL/ELT frameworks using PySpark and Spark SQL.

Implement Delta Lake architecture and data governance standards.

Process large-scale structured, semi-structured, and streaming datasets.

Optimize Spark jobs and cluster performance for maximum efficiency.

Develop reusable frameworks, notebooks, workflows, and data products.

Collaborate with business teams, data scientists, analysts, and architects to deliver enterprise-grade solutions.

Implement monitoring, alerting, security, and compliance requirements.

Participate in code reviews, deployment activities, and production support.

Develop technical documentation and mentor junior engineers.

Required Technical Skills

Databricks & Big Data

Azure Databricks

Databricks Lakehouse Platform

Delta Lake

Spark SQL

PySpark

Databricks Workflows

Unity Catalog

Delta Live Tables (DLT)

Structured Streaming

Data Engineering

Data Lake Architecture

Lakehouse Architecture

ETL/ELT Development

Data Modeling

Data Integration

Performance Tuning

Data Quality Frameworks

Programming Languages

Python

PySpark

SQL

Scala

Java

Shell Scripting

Cloud Platforms

Microsoft Azure

Azure Data Factory (ADF)

Azure Data Lake Storage (ADLS)

AWS

Amazon S3

AWS EMR

Google Cloud Platform (GCP)

Databases

SQL Server

Oracle

PostgreSQL

MySQL

MongoDB

Cassandra

Streaming & Messaging

Kafka

Event Hubs

Spark Streaming

Azure Stream Analytics

DevOps & Tools

Git

Jenkins

Azure DevOps

CI/CD Pipelines

Terraform

Unix/Linux

Required Qualifications

Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.

6-9 years of IT experience with at least 4+ years of hands-on Databricks and Spark development experience.

Strong expertise in distributed data processing and Lakehouse architecture.

Experience building enterprise-scale data engineering solutions.

Strong knowledge of PySpark, Spark SQL, and performance optimization.

Experience with Agile development methodologies.

Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications

Experience with cloud-native analytics platforms on Azure, AWS, or GCP.

Knowledge of Machine Learning workflows within Databricks.

Experience implementing Delta Lake and Unity Catalog.

Exposure to Data Governance and Data Security frameworks.

Relevant Databricks, Azure, or AWS certifications.

Key Competencies

Data Engineering Excellence

Performance Optimization

Solution Design

Stakeholder Management

Collaboration and Leadership

Ownership and Accountability

Nice to Have

Snowflake

Azure Data Factory (ADF)

Delta Live Tables (DLT)

Unity Catalog

MLflow

Airflow

Kafka

Data Mesh Architecture

Generative AI / AI Engineering exposure

dbt

Apply on Deloitte India (South Asia)