Lead II - Data Engineering

We are looking for strong Lead Databricks Developer with streaming expertise. Key Responsibilities: • Design, develop, and lead enterprise-scale data platforms and pipelines using Databricks, PySpark, Delta Lake, and Spark SQL. • Build and optimize scalable batch and real-time data ingestion solutions using Kafka, Event Hubs, Auto Loader, Structured Streaming, and Delta Live Tables (DLT). • Develop and manage Lakehouse architectures leveraging Medallion (Bronze, Silver, Gold) design patterns and data engineering best practices. • Implement and govern enterprise data assets using Unity Catalog, including data lineage, metadata management, security policies, and access controls. • Collaborate with architects and cloud teams to build and deploy data solutions on AWS and Azure, leveraging services such as S3, ADLS, Event Hubs, IAM, and cloud-native security capabilities. • Drive data quality, observability, monitoring, and validation frameworks to ensure trusted, reliable, and business-ready data products. • Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency through workload tuning and FinOps best practices. • Lead cloud modernization and migration initiatives from legacy/on-premises platforms to modern Databricks Lakehouse architectures. • Implement and support GenAI, RAG, AI Agents, Vector Search, Mosaic AI, MLflow, and Databricks Genie use cases to accelerate analytics and business insights. • Establish CI/CD, DataOps, Infrastructure as Code (Terraform), and DevOps best practices while providing technical leadership, mentoring team members, and ensuring delivery excellence. Domain Experience: • Experience in Healthcare Payer domain (Claims, Membership, Coverage) • Exposure to FHIR / healthcare data standards preferred • Understanding of data privacy, security, and governance requirements (PII/HIPAA) • Leadership Expectations • Operate at a Lead level driving design, development, and delivery • Mentor team members and ensure delivery quality • Own modules independently and collaborate across distributed teams Experience: • 8+ years in Data Engineering • 5+ years hands-on experience with Databricks • Prior experience in Tech Lead/Senior roles