Principal Data Engineer (PySpark, AI)

Autodesk · Software & Internet

Bengaluru, India
On-site
Posted today
Data & AI

Apply on Autodesk Create resume for this job

About the job

Job Requisition ID #

26WD101628

Position Overview

Autodesk is looking for a highly skilled Principal Data Engineer to join our Advanced Compliance Products (ACP) team. ACP is a strategic initiative focused on detecting non-compliant usage of Autodesk software through large-scale data processing and analytics.

As a Principal Data Engineer, you will drive the technical architecture, scalability, and modernization of ACP's data engineering platform, leveraging modern cloud technologies and AI-driven engineering practices. You will solve complex engineering challenges, establish best practices, and lead cross-functional technical initiatives while remaining hands-on.

Responsibilities

Define and drive the architecture of scalable, reliable, and cost-efficient data platforms processing billions of records.

Lead the design, development, and optimization of large-scale data pipelines using Spark, PySpark, Python, and SQL.

Drive platform modernization initiatives, including incremental processing, reusable frameworks, and modern data lakehouse architectures.

Leverage AI/GenAI technologies and AI-assisted development tools to improve engineering productivity, automate workflows, enhance data quality, and optimize pipeline operations.

Identify and resolve complex performance, scalability, reliability, and data quality challenges.

Establish engineering best practices for CI/CD, automated testing, observability, and cloud infrastructure.

Lead technical design discussions, influence architectural decisions, and mentor engineers across teams.

Collaborate with product managers, data scientists, and business stakeholders to translate business requirements into scalable data solutions.

Take end-to-end technical ownership of complex, cross-team engineering initiatives.

Minimum Qualifications

10+ years of experience in data engineering, distributed systems, or large-scale data processing.

Strong hands-on expertise in Apache Spark, PySpark, Python, Spark SQL, and SQL, including performance tuning and optimization.

Proven experience architecting and scaling complex data pipelines and distributed data processing systems.

Strong knowledge of AWS, Hive, Snowflake, and modern data lakehouse architectures.

Experience with data modeling, ETL/ELT frameworks, orchestration tools (such as Airflow), and CI/CD practices.

Hands-on experience with AI/GenAI technologies, including LLMs, prompt engineering, and AI-assisted development tools such as GitHub Copilot, Cursor, or equivalent.

Practical experience applying AI to data engineering workflows, such as code generation, automated testing, data quality validation, anomaly detection, pipeline optimization, or intelligent troubleshooting.

Strong understanding of data quality, observability, reliability, and cloud infrastructure.

Demonstrated technical leadership, architectural decision-making, and mentoring experience.

Excellent problem-solving, communication, and cross-functional collaboration skills.

Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Preferred Qualifications

Experience with Apache Iceberg, incremental data processing, and modern orchestration frameworks.

Experience modernizing legacy data platforms and optimizing large-scale Spark workloads.

Familiarity with BI tools such as Power BI or Looker.

Experience with AI agents or LLM-powered automation.

Experience building reusable data engineering frameworks and driving engineering standards across teams.

#LI-SJ1

Learn More / Plus d'information

About Autodesk / À propos d’Autodesk

Welcome to Autodesk! Amazing things are created every day with our software – from the greenest buildings and cleanest cars to the smartest factories and biggest hit movies. We help innovators turn their ideas into reality, transforming not only how things are made, but what can be made.

We take great pride in our culture here at Autodesk – it’s at the core of everything we do. Our culture guides the way we work and treat each other, informs how we connect with customers and partners, and defines how we show up in the world.

When you’re an Autodesker, you can do meaningful work that helps build a better world designed and made for all. Ready to shape the world and your future? Join us!

Bienvenue chez Autodesk ! Nos logiciels créent chaque jour des choses extraordinaires : des bâtiments les plus écologiques aux voitures les plus propres, en passant par les usines les plus intelligentes et les films à succès. Nous aidons les innovateurs à concrétiser leurs idées, transformant non seulement la façon dont les choses sont fabriquées, mais aussi ce qui peut l’être.

Nous sommes très fiers de notre culture chez Autodesk ; elle est au cœur de tout ce que nous faisons. Notre culture guide notre façon de travailler et de nous comporter les uns envers les autres, influence nos interactions avec nos clients et nos partenaires, et définit notre image au monde.

En tant qu’Autodeskien, vous pouvez accomplir un travail significatif qui contribue à bâtir un monde meilleur, conçu et réalisé pour tous. Prêt à façonner le monde et votre avenir ? Rejoignez-nous !

Salary transparency / Transparence Salariale

Salary is one part of Autodesk’s competitive compensation package. Offers are based on the candidate’s experience and geographic location. In addition to base salaries, our compensation package may include annual cash bonuses, commissions for sales roles, stock grants, and a comprehensive benefits package.

Belonging

We take pride in cultivating a culture of belonging where everyone can thrive. Learn more here: https://www.autodesk.com/company/global-belonging

In-Person Onboarding and Identity Verification

This role may require in-person onboarding and/or in-person ID verification.

Apply on Autodesk