Senior / Lead Data Engineer – Service Layer / QTC Integration Platform
TALPRO INDIA PRIVATE LIMITED · Staffing & Recruitment
- Bengaluru, India
- On-site
- Posted today
- Data & AI
- Contract
About the job
Senior / Lead Data Engineer – Service Layer / QTC Integration Platform
Role Details
Role: Senior / Lead Data Engineer
Primary Skills: Go / Golang, Python, AWS, Kafka / MSK, Microservices, Distributed Systems, Data Engineering
Platform Exposure: AWS Serverless, Kafka, Snowflake, PostgreSQL, ClickHouse, Event-Driven Architecture
Domain Exposure: Quote-to-Cash, CRM, ERP, Customer Lifecycle, Enterprise Data preferred
Experience: 8+ years
Location/ Mode: Bengaluru/Hybrid
Budget : 27 LPA
Contract Duration : 12 months
Role Overview
We are looking for a strong, hands-on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote-to-Cash (QTC) , customer lifecycle, finance, and enterprise application integrations.
This role requires deep expertise in distributed systems, event-driven architecture, microservices, data engineering, cloud-native development, and platform reliability . The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.
The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud-native architecture patterns , processing high-volume, business-critical events with strong focus on reliability, security, scalability, observability, and operational readiness.
Key Responsibilities
Technical Delivery & Engineering Leadership
Own end-to-end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.
Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.
Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud-native architecture.
Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.
Architecture & Platform Design
Drive architecture and design decisions for event-driven integrations , canonical data models, orchestration frameworks, and platform capabilities.
Define and enforce engineering standards, secure-by-design principles, cloud best practices, data governance, and non-functional requirements.
Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.
Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.
Event-Driven Integration & Microservices
Design, develop, and operate real-time event-driven integration services using Go / Golang and Python .
Build and enhance the Service Layer platform to orchestrate Quote-to-Cash workflows across enterprise applications.
Design and implement scalable Kafka / AWS MSK-based event processing with support for: - Retries
Idempotency
Sequencing
Message replay
Fault recovery
Dead-letter queues
Eventual consistency
Build secure and scalable RESTful APIs , integration services, and enterprise system interfaces.
Data Engineering, Migration & Reconciliation
Design and implement large-scale ETL, migration, reconciliation, and data-quality solutions using: - AWS Glue
Step Functions
Snowflake
PostgreSQL
ClickHouse
Build and enforce canonical data models , data contracts, transformation layers, and validation frameworks.
Support enterprise integrations by standardizing communication across multiple business systems.
Ensure data accuracy, traceability, consistency, and operational reliability across the platform.
AWS Cloud-Native Engineering
Design and build cloud-native applications on AWS using: - AWS Lambda
API Gateway
S3
RDS / Aurora PostgreSQL
DynamoDB
SQS
SNS
EventBridge
IAM
CloudWatch
VPC
Build high-availability, business-critical applications with strong focus on performance, security, and operational supportability.
Observability, Monitoring & Operations
Design and implement observability, monitoring, alerting, and operational dashboards for end-to-end workflow visibility.
Work with tools such as: - CloudWatch
Grafana
Datadog
Prometheus
OpenTelemetry
Support production operations including incident management, root-cause analysis, troubleshooting, and performance optimization.
Drive metrics-driven improvements for reliability, scalability, and platform performance.
Collaboration & Stakeholder Management
Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.
Define technical roadmaps, execution plans, dependencies, and delivery milestones.
Communicate technical risks, trade-offs, design decisions, and mitigation plans clearly to stakeholders.
Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.
Must-Have Skills
Leadership & Delivery
Proven experience leading and mentoring engineering teams.
Strong experience in task planning, technical guidance, code reviews, and delivery management.
Ability to drive architecture decisions for large-scale distributed systems and enterprise integration platforms.
Experience managing multiple parallel technical initiatives.
Programming & Backend Engineering
Strong hands-on experience in: - Go / Golang
Python
Experience building scalable backend services, event-driven integrations, orchestration workflows, and data processing solutions.
Strong understanding of REST API development and secure API design.
AWS Cloud & Serverless
Strong experience designing and building cloud-native applications on AWS.
Hands-on experience with: - Lambda
API Gateway
S3
RDS / Aurora PostgreSQL
DynamoDB
SQS / SNS
EventBridge
IAM
CloudWatch
VPC
Strong understanding of AWS serverless architecture patterns.
Kafka / Event-Driven Systems
Strong experience designing and operating event-driven and asynchronous systems using: - Kafka
AWS MSK
RabbitMQ or equivalent messaging technologies
Strong understanding of: - Fault tolerance
Idempotency
Retry patterns
Dead-letter queues
Message replay
Eventual consistency
Sequencing
Event orchestration
Databases & Data Engineering
Strong experience with: - PostgreSQL
ClickHouse
Relational databases
DynamoDB / NoSQL databases
Strong knowledge of: - Data modelling
Query optimization
Performance tuning
Data validation
Reconciliation frameworks
Experience with AWS Glue, Step Functions, Snowflake, and large-scale data workflows.
CI/CD, Security & Observability
Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as: - GitLab CI
Jenkins
SonarQube
Trivy
Semgrep
Strong understanding of cloud security and secure-by-design principles including: - IAM
OAuth2
JWT authentication
Encryption at rest and in transit
Secrets Manager
Vulnerability management
Audit logging
Compliance controls
Required Qualifications
Bachelor’s degree in Computer Science, Engineering, Information Technology , or related discipline.
8+ years of software engineering experience.
At least 4+ years focused on cloud-native, data engineering, integration, or distributed platforms.
4+ years building cloud-native solutions on AWS.
4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.
Proven experience designing, building, and operating highly available enterprise integration and event-driven platforms.
Strong experience in production support, incident management, root-cause analysis, and performance optimization.
Nice to Have
AWS certifications such as: - AWS Certified Data Engineer – Associate
AWS Certified Solutions Architect – Associate
AWS Certified Developer – Associate
Equivalent AWS certifications
Experience in: - Quote-to-Cash
CRM
ERP
Customer Lifecycle Management
Enterprise Data domains
Exposure to large-scale finance, billing, or enterprise integration transformation programmes.
Experience building operational dashboards and platform-level reporting for engineering and business teams.