Lead I - Data Engineering

Job Description: Databricks Lead Role: Databricks Lead / Technical Lead Experience: 8+ years overall, with strong hands-on data engineering experience Role Summary We are looking for an experienced Databricks Lead to lead the design, development, and delivery of scalable data engineering solutions. The candidate should have strong hands-on expertise in Apache Spark, Databricks, SQL, and Python, along with excellent analytical and problem-solving skills. The role requires a combination of technical leadership and hands-on engineering, including solution design, code reviews, performance optimization, troubleshooting, and mentoring the development team. Key Responsibilities • Lead the design and development of scalable data pipelines and data-processing solutions using Databricks and Apache Spark. • Develop and optimize complex data transformations using PySpark/Python and SQL. • Design efficient ETL/ELT workflows for high-volume datasets. • Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines. • Provide technical leadership, design guidance, and hands-on support to the engineering team. • Conduct code reviews and ensure adherence to coding and engineering best practices. • Troubleshoot complex data, performance, and production issues. • Collaborate with business analysts, architects, data engineers, and other stakeholders to translate business requirements into technical solutions. • Mentor team members and drive effective technical problem-solving. • Ensure data quality, reliability, maintainability, and scalability of implemented solutions. • Mandatory Skills • Apache Spark / PySpark: Mandatory • Databricks: Strong hands-on experience • SQL: Strong, including complex queries, joins, aggregations, query optimization, and data analysis • Python: Strong hands-on programming skills • Strong experience in data engineering and ETL/ELT development • Strong analytical, debugging, and problem-solving skills • Experience working with large-scale datasets and distributed data-processing systems • Strong communication and technical leadership skills Good to Have • Experience with Delta Lake / Delta tables • Databricks workflows and job orchestration • Cloud experience with Azure/AWS • CI/CD and DevOps practices for data engineering • Data modeling and data warehouse concepts • Git/version control and automated deployment practices