Data Engineer - AWS Data Pipelines - Data Engineer,AWS,Apache Airflow,SQL,Python,ETL,Terraform - Bangalore,Gurgaon,Hyderabad, 6.00 - 10.00 yrs | Sampoorna (J51018)

job opening

Job Summary


Experience:

6.00 - 10.00  Years 

Industrial Type:

IT-Software/Software Services

Location:

Bangalore,Gurgaon,Hyderabad

Functional Area:

IT Software - DBA / Datawarehousing

Designation:

Data Engineer - AWS Data Pipelines

Key Skills:

Data Engineer,AWS,Apache Airflow,SQL,Python,ETL,Terraform

Educational Level:

Graduate/Bachelors

Job Post Date:

Stream of Study:

Degree:

BA, BBA, BBA/BMS, BBI, BCA, BCom, BCS, BDES, BE-Comp/IT, BEd, BE-Other, BFA, BFM, BIS, BIT, BMS, BSc-Comp/IT, BSc-Other, BTech-Comp/IT, BTech-Other

Company Description


Our client is committed to combining cutting-edge innovation with exceptional product, service, and operation performance. We specialize in Artificial Intelligence, Metaverse, Blockchain, Machine Learning, AR/VR, Big data, IoT, and DevOps software development among others willingness to invest in new ideas, technologies, and approaches to solve problems or create better products and services. Our unwavering dedication to harnessing the power of innovation and forward-thinking solutions to shape what lies ahead. As we move forward, we recognize the pivotal role that cutting-edge programming technologies play in crafting the future. Our commitment extends beyond the mere development of reliable technology. when harnessed with precision and purpose, becomes a potent force for change. With a relentless focus on-time delivery, continuous testing, attentive response to feedback, and an unceasing drive for improvement. The route our organization is committed to following while also delivering leadership in digital transformation to our clients.

Job Description


1.Mandatory Technical Skills – Must Have
A. Apache Airflow – Self-Managed (Critical Requirement)
• Hands-on experience installing, configuring, deploying, and operating Apache Airflow.
• Experience managing DAGs, scheduling, dependencies, retries, monitoring, troubleshooting, and production failures.
• Understanding of Airflow infrastructure, workers, schedulers, executors, and operational maintenance.
• Experience running Airflow independently, not exclusively through managed platforms such as AWS MWAA or Astronomer.
Important: Candidates who have only created or maintained DAGs within an existing managed Airflow environment are not an exact match.
B. AWS Data Engineering
Hands-on experience building and maintaining data pipelines using:
• Amazon S3: Data storage, ingestion, and organization.
• AWS Glue: ETL jobs, Glue Data Catalog, and data transformations.
• Amazon Athena: SQL-based querying, validation, and analysis of data stored in S3.
Candidates should demonstrate how these services were used together in actual production pipelines.
C. SQL & Python – Strong Hands-On Expertise
• Advanced SQL skills, including joins, CTEs, window functions, aggregations, and query optimization.
• Strong Python programming skills for building ETL/ELT pipelines, transformations, automation, and data validation.
• Experience troubleshooting data issues and improving pipeline reliability.
D. Data Quality & Reconciliation – Critical Requirement
Must have experience ensuring the accuracy and integrity of data processed through pipelines.
Relevant experience includes:
• Source-to-target data reconciliation.
• Record count and aggregate validation.
• Identifying missing, duplicate, inconsistent, or incorrect records.
• Investigating data mismatches and transformation errors.
• Implementing automated data quality checks.
• Ensuring business metrics and reporting outputs are accurate.
Important: Candidates focused only on pipeline execution, scheduling, or infrastructure monitoring without data validation experience are not a strong match.
E. dbt (Data Build Tool)
• Hands-on experience developing SQL transformation models using dbt.
• Experience with dbt tests, dependencies, documentation, and data transformation workflows.
• Ability to demonstrate dbt usage in actual projects.
F. Terraform – Infrastructure as Code
• Practical experience using Terraform to provision and manage cloud infrastructure.
• Experience maintaining Terraform configurations and deploying AWS resources.
• Understanding of infrastructure version control and repeatable deployments.
3. Preferred / Good-to-Have Experience
• Experience working with healthcare, health-tech, or personal/sensitive data.
• Familiarity with data privacy, security, and compliance requirements.
• Experience handling regulated or confidential datasets.
• Experience troubleshooting and maintaining production data pipelines.
NOTE : Profiles to Avoid / Deprioritize
Please avoid submitting candidates whose primary experience is:
• AI/GenAI/LLM Engineering: Primarily focused on AI agents, machine learning, or GenAI applications rather than traditional data engineering.
• Large-Scale Big Data Engineering: Primarily focused on Spark, PySpark, Kafka, Hadoop, or streaming platforms without strong experience in the required AWS stack.
• Managed Airflow Only: Experience limited to using AWS MWAA, Astronomer, or an already configured Airflow environment.
• ETL Tool Operators: Experience limited to running or monitoring pipelines without developing, troubleshooting, or validating data.
• Skills-Only Resumes: Candidates listing the required technologies without demonstrating hands-on project experience.
Spark, Kafka, and AI experience are not disqualifying, but they should not substitute for the mandatory skills.

2. Mandatory Resume Submission Guidelines :
Before sharing a profile with the client, please ensure:
1. Employment and Project Dates: Include start and end dates (month and year) for each role and relevant project.
2. Project-Based Technical Evidence: Clearly describe how Airflow, AWS S3, Athena, Glue, SQL, Python, dbt, and Terraform were used in actual projects.
3. Airflow Ownership: Explicitly mention whether the candidate independently deployed and managed Airflow or only worked with an existing managed environment.
4. Data Quality Experience: Highlight specific examples of data reconciliation, validation, accuracy checks, and issue resolution.
5. Previous Application Status: Confirm whether the candidate has previously applied, interviewed, or been submitted to this client.

announce
Did not find a matching job? You can still send your CV to jobs@sampoorna.com or Register Here