Job descriptionYour Mission
We are seeking a highly skilled Senior Data Engineer to join our team and play a crucial role in developing and managing critical components of our data platform like Data API (Rest API), Apache Airflow and data engineering artifacts like data ingestion and data curation pipelines. This role demands a strong proficiency in Python, Apache Airflow, REST APIs, PySpark and Azure cloud services.
For this opportunity, fluent English is essential, as the professional will work daily with people from all over the world. We're looking for people who live in Europe, in locations that share the same time zone as the UK. Professionals from other locations can apply as long as they are interested and available to work in the client's time zone. You will need to have a Brazilian CNPJ and be available to issue an invoice for payment.
Main responsibilitiesActivities
Work closely with the Data Engineering Lead to devise effective data ingestion strategies aligned with business objectives.Define and implement robust data ingestion patterns and processes to ensure efficient and reliable data flow into the organization's data platform.Develop and maintain reusable data engineering or ETL pipelines and codebase using Python, PySpark, Databricks, Airflow, REST APIs, and the Azure cloud platform.Design and implement robust data APIs using Python frameworks such as FastAPI or Flask and deploy them on Azure App Service.Collaborate with cross-functional teams to understand data requirements and provide scalable data engineering solutions.Design and implement batch and streaming data architectures leveraging Azure cloud services like Azure Data Factory and Azure Databricks.Contribute to the development and deployment of Apache Airflow on Azure Kubernetes Service (AKS).Ensure adherence to software engineering best practices, including version control, testing, and continuous integration/continuous deployment (CI/CD) processes.Participate in code reviews, technical discussions, and knowledge-sharing sessions within the team. Requirements and skillsTechnical Requirements:
Good experience in Python programming language.Proven experience with Apache Airflow and its deployments on Azure Kubernetes Service.Hands-on experience with Python frameworks like FastAPI or Flask and deploying REST APIs on Azure App Service.Strong understanding of core data engineering concepts and principles.Solid experience with Azure cloud services, including Azure Data Factory and Azure Databricks.Proficiency in writing PySpark code for data processing and transformation.Ability to design and implement reusable and scalable data engineering pipelines and codebase.Sound understanding of software engineering development practices, including version control, testing, and continuous integration/continuous deployment (CI/CD).Experience in designing batch and streaming data architectures using Azure cloud platform services.Effective problem-solving skills and the ability to troubleshoot complex data engineering issues.Commitment to continuous learning and staying updated with industry trends and best practices in data engineering.Strong communication and collaboration skills with English language proficiency. Ability to work effectively in a team environment, provide training, and document processes.Differentiator: Familiarity with Python tools like Pytest, tox, poetry.Familiarity with Python testing frameworks like pytest and build tools like tox and poetry.Experience in building CI/CD pipelines and working with Git-based version control systems.Knowledge of containerization technologies such as Docker and Kubernetes.Behavioral Requirements: Effective problem-solving skills and the ability to troubleshoot complex data engineering issues.Commitment to continuous learning and staying updated with industry trends and best practices in data engineering.Strong communication and collaboration skills with English language proficiency. Ability to work effectively in a team environment, provide training, and document processes. Additional informationAdvantages of service provision:
Anywhere office culture: we like to have the freedom to work with autonomy, flexibility, and trust;
Psychological support: for taking care of our mental health;
Certification assistance FIAP: Undergraduate and postgraduate courses - 10% OFF on undergraduate courses and 20% OFF on MBA, POSTECH, and SHIFT courses, both in-person and online.
#J-18808-Ljbffr