•5–7 years of relevant experience in ETL, data integration, or data engineering roles, with demonstrated ownership of complex technical deliverables.
•Lead ETL and data engineering initiatives or projects from planning through delivery with minimal guidance, ensuring scope, timelines, quality, dependencies, and risks are effectively managed.
•Design, develop, test, deploy, and support scalable batch and near-real-time data pipelines.
•Build and optimize ETL workflows using AWS Glue, Lambda, S3, and Redshift.
•Develop data transformation logic using Python, PySpark, and SQL, with a focus on performance, reusability, maintainability, and engineering best practices.
•Integrate data from databases, files, REST APIs, and other source systems into curated data layers.
•Implement data quality checks, reconciliation, monitoring, error handling, and operational alerting across data pipelines.
•Design and optimize data models, schemas, and complex SQL queries for relational and data warehouse environments.
•Contribute to and lead workstreams within the planned migration from the AWS-based data platform to Databricks, including pipeline assessment, redesign, validation, and implementation planning.
•Implement and manage CI/CD pipelines using Azure DevOps or similar tools for source control, automated testing, and deployment.
•Provide technical direction, mentorship, and day-to-day guidance to team members; review deliverables and help the team resolve blockers to achieve successful outcomes.
•Apply critical thinking and sound judgment to analyze ambiguous problems, evaluate options, make recommendations, and drive issues to resolution.
•Collaborate with cross-functional teams to translate business and data requirements into reliable technical solutions that meet quality and delivery expectations.
•Manage project plans, priorities, risks, dependencies, and action items, and provide clear, regular progress updates to stakeholders and leadership.
•Communicate complex technical topics clearly in meetings, presentations, and written updates, adapting the message for technical and business audiences.
•Demonstrate accountability, collaboration, and the ability to work independently while managing multiple priorities in a dynamic environment.
Education/Background: Bachelor / Master degree in Computer Science Engineering, or a related field.
Location: Hyderabad
Shift Timings: 11:00 AM to 8:00 PM
Key skills:
•5–7 years of hands-on experience designing, developing, and supporting ETL and data integration pipelines.
•Strong hands-on experience with AWS Glue, Lambda, S3, and Redshift.
•Strong Python and PySpark development experience for data ingestion and transformation.
•Strong SQL skills, including writing complex queries, performance tuning, and working with relational databases and data warehouses.
•Hands-on experience with Kafka or comparable event-driven technologies for real-time or near-real-time data transmission and processing.
•Hands-on experience building traditional bulk data ingestion processes from relational databases and CSV/flat files, including full and incremental loads.
•Strong understanding of data modeling, data quality, reconciliation, orchestration, logging, monitoring, and error-handling practices.
•Demonstrated ability to lead technical initiatives or projects with minimal guidance and deliver outcomes through effective planning, prioritization, risk management, and follow-through.
•Experience guiding team members, coordinating work across contributors, reviewing technical deliverables, and helping teams resolve blockers and achieve delivery goals.
•Strong critical thinking, analytical, troubleshooting, and problem-solving abilities, including the ability to navigate ambiguity and make sound recommendations.
•Effective project management skills, including tracking milestones, risks, dependencies, and actions and reporting progress clearly to stakeholders and leadership.
•Excellent written and verbal communication, presentation, and stakeholder management skills, with the confidence to lead discussions and explain complex technical topics to technical and business audiences.
•Familiarity with CI/CD processes and tools, with Azure DevOps preferred.
•Hands-on experience with Databricks, including notebooks, workflows/jobs, Delta Lake, and lakehouse concepts.
•Experience migrating ETL pipelines and data workloads from AWS services or legacy platforms to Databricks.
•Knowledge of Databricks governance and security capabilities, such as Unity Catalog.
•Experience with containerization technologies such as Docker, Kubernetes, or AWS EKS.
•Experience with legacy ETL and database technologies, including SSIS, SQL Server, and Oracle, and modernizing or migrating those solutions.
•High-level familiarity with reporting and dashboarding tools such as SSRS/Bold Reports, QlikView, or Qlik Sense, to support downstream data consumers.
•Understanding of Agile development methodologies.
