facilita.etc
← Todas as vagas

Data Engineer – Python, PySpark & AWS

Vagas Externas — Remotar · Remoto · CLT · pleno
🌐 Entre para traduzir esta vaga

Descrição

- Design, develop, test, and maintain scalable ETL/ELT pipelines to support data integration, transformation, and processing across multiple data sources. - Build and optimize data-processing solutions using Python, SQL, and PySpark. - Develop and maintain cloud-native data solutions using AWS services, including S3, Glue, Athena, Lambda, and Redshift. - Design, implement, and support Data Lakes and Data Warehouses, ensuring efficient data storage, organization, accessibility, and processing. - Develop data models and transformation workflows to support analytics, reporting, and business intelligence requirements. - Integrate structured and semi-structured data from multiple sources, ensuring consistency, accuracy, and reliability. - Implement data-quality checks, validation processes, and monitoring mechanisms to maintain data integrity across pipelines. - Optimize data pipelines, SQL queries, and distributed processing workloads to improve performance, scalability, and cost efficiency. - Collaborate with business stakeholders, analysts, and engineering teams to understand data requirements and translate them into effective technical solutions. - Participate in technical design discussions, contributing to data architecture, integration patterns, and engineering best practices. - Develop and maintain reusable, well-structured, and maintainable code following data engineering best practices. - Work with Git, CI/CD pipelines, and Terraform to support version control, automated deployments, and infrastructure management. - Troubleshoot complex data-processing issues, investigate pipeline failures, and implement reliable, long-term solutions. - Perform testing, validation, and debugging to ensure the quality and reliability of data solutions. - Produce technical documentation covering pipeline architecture, data transformations, implementation decisions, and operational procedures. - Take end-to-end ownership of data engineering deliverables, from requirements analysis and development through testing, implementation, and technical handoff. Required Qualifications - Solid professional experience in Data Engineering, with a strong background in developing and maintaining production data pipelines. - Strong proficiency in Python for data processing, transformation, and automation. - Advanced SQL skills, including complex queries, data transformations, performance optimization, and troubleshooting. - Hands-on experience with PySpark and distributed data processing. - Strong experience designing, developing, and maintaining ETL/ELT pipelines. - Practical experience with AWS cloud services, particularly Amazon S3, AWS Glue, Amazon Athena, AWS Lambda, and Amazon Redshift. - Solid understanding of Data Lake and Data Warehouse architectures, including data storage, processing, and integration patterns. - Experience with data modeling, data transformation, and data integration across multiple sources. - Familiarity with data-quality validation, monitoring, and pipeline troubleshooting. - Experience with Git and collaborative development workflows. - Familiarity with CI/CD practices and Terraform for automated deployments and infrastructure management. - Strong understanding of data engineering principles, code quality, performance optimization, and maintainability. - Experience working across the full data engineering lifecycle, including requirements gathering, implementation, testing, troubleshooting, documentation, and technical handoff. - Strong analytical and problem-solving skills, with the ability to independently investigate and resolve complex technical issues. - Excellent communication and collaboration skills. - Ability to work independently, manage priorities, and deliver high-quality solutions in a collaborative environment

Candidatar-se

Esta vaga é agregada de Vagas Externas — Remotar. A candidatura é feita no site da fonte.

Candidatar-se no site da fonte