Data Engineer
Techdome · IT Services & Consulting
- Hyderabad, India; Indore, India
- On-site
- Posted today
- Data & AI
About the job
Experience: 3-6
Role summary: The Data Engineer builds the data foundation that AI agents, scorecards, and dashboards run on: ingestion pipelines, connectors, metadata and search indexes, vector stores, data marts, and quality instrumentation.
Experience: 3 to 6 years in data engineering. Senior Data Engineer: 2+ years, including ownership of data foundation architecture and connector frameworks.
Key responsibilities
Design and build ingestion pipelines for structured data, unstructured content (documents, PDFs), and metadata.
Build reusable connectors to enterprise systems, catalogs, content repositories, and third-party or licensed sources via APIs.
Model and build raw-to-mart data layers that serve analytics and AI use cases.
Implement metadata extraction, enrichment, and search indexing, including semantic and vector indexes for RAG.
Set up and manage vector databases and knowledge repositories used by LLM agents.
Implement data quality rules, profiling, scoring outputs, and exception handling.
Register lineage and maintain source registries, version tracking, and refresh controls.
Design data stores for signals, findings, audit trails, and user feedback loops.
Apply security and governance controls: RBAC, PII/sensitivity flagging, and approved data handling.
Support SIT/UAT data validation, defect fixes, and production release activities.
Required skills
Strong Python and SQL; solid data modeling (dimensional and normalized).
Hands-on experience with a modern data platform such as Databricks, Snowflake, or Azure/AWS data services.
Pipeline orchestration and transformation (Spark, Airflow, ADF, dbt, or similar).
API-based integration (REST), JSON handling, and incremental/CDC ingestion patterns.
Data quality frameworks and testing practices for pipelines.
Version control (Git) and CI/CD for data workloads.
Preferred skills
RAG data preparation: chunking, embeddings, vector databases (Azure AI Search, pgvector, Pinecone, or similar).
Unstructured content processing: text extraction, OCR, document parsing.
Metadata management, data catalogs, ontologies, or knowledge graphs (for example Neptune or other graph databases).
Experience supporting LLM or agentic applications with grounded, traceable data.
Life sciences data exposure (commercial, medical, regulatory, or launch data) and regulated-data handling.
Cloud certification (Azure Data Engineer, Databricks, AWS, or Snowflake).