Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.
Primary Responsibilities:
•Design, build and maintain scalable batch and streaming data pipelines on Azure and Databricks using the Medallion Architecture (Bronze, Silver, Gold) on Delta Lake
•Ingest and process structured, semi-structured and unstructured data in formats such as CSV, JSON, XML, Parquet, Avro, ORC, Excel and Delta, using Azure Data Factory, Auto Loader and Lakeflow Spark Declarative Pipelines
•Develop complex transformations in PySpark and Spark SQL, including CDC, SCD Type 1/2, deduplication and handling of nested or schema-evolving data
•Design data models and build the right consumption objects for each use case: managed/external tables, views, temporary views, materialized views and streaming tables
•Implement data governance, lineage and security using Unity Catalog, including row/column-level security and PHI/PII protection
•Build data quality, validation and reconciliation frameworks to keep data accurate and reliable
•Tune Spark jobs and Delta tables for performance and cost (partitioning, liquid clustering, Z-ordering, OPTIMIZE/VACUUM, compute sizing)
•Drive CI/CD and DataOps practices using Azure DevOps or GitHub, Declarative Automation Bundles and automated testing
•Troubleshoot production issues, do root cause analysis and put preventive fixes in place
•Work with architects, analysts, data scientists and business stakeholders, and mentor junior engineers
•Use AI-assisted engineering tools to improve development, testing and documentation
•Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Required Qualifications:
•B.E./B.Tech./MCA or equivalent degree in Computer Science, IT or Engineering
•6+ years of data engineering experience, including 3+ years hands-on with Azure Databricks
•Hands-on experience with Delta Lake, Lakehouse architecture and Unity Catalog
•Experience working with file formats such as CSV, JSON, XML, Parquet, Avro and Delta, including schema inference, schema evolution and nested data handling
•Experience with Azure services: ADF, ADLS Gen2, Key Vault, Azure SQL , Power Bi
•Solid knowledge of data modeling, ETL/ELT design, Git, CI/CD and Agile delivery
•Good understanding of views, temporary/global temporary views, materialized views, streaming tables and when to use each
•Solid proficiency in Python, PySpark and SQL, with a solid understanding of Spark internals and performance tuning
•Proven solid problem-solving and communication skills
