Databricks Data Engineer (Spark AND Unity)
Unison Group · IT Services & Consulting
- Bengaluru, Karnataka, India
- On-site
- Posted today
- Data & AI
About the job
Interview - Face to face
Position Overview:
Build and maintain scalable data pipelines using Azure Databricks to support DATA & AI ACDP projects. Apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights.
Key Responsibilities:
Design, build, and optimize data pipelines in Azure Databricks for ingestion, ETL/ELT, and transformations, implementing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake for data quality and versioning.
Utilize Databricks Spark (PySpark, SQL), Delta Live Tables, and Unity Catalog for pipeline development, governance, and basic streaming.
Integrate data from ADLS Gen2, Azure SQL Database, and SQL Pools; support orchestration via Databricks workflows.
Implement CI/CD in Azure DevOps for Databricks deployments, including Bicep templates and ADF integration.
Apply data quality measures (expectations, constraints), security (RBAC, encryption), and monitoring for performance/cost efficiency using auto-scaling and Photon.
Work with analysts and stakeholders to refine requirements into functional workflows; handle data cleansing, modeling, and Python/SQL scripting for structured/unstructured data.
Troubleshoot pipelines and contribute to documentation/best practices.
Required Qualifications:
3-4 years in data engineering, focused on Azure Databricks pipelines and ETL
Proficiency in PySpark, SQL, Python; experience with cloud storage and basic orchestration
Familiarity with Azure fundamentals and collaborative tools