We are seeking a highly skilled and experienced Senior Data Engineer to join our client team. The ideal candidate will play a key role in designing, building, and maintaining scalable and efficient data processing pipelines, as well as supporting the deployment of advanced data solutions.
Responsibilities
* Pipeline Development: Design, develop, and maintain scalable data processing pipelines for handling large volumes of structured and unstructured data in a distributed environment.
* Microservices Development: Build and maintain production-ready microservices in Python to serve data and features at scale.
* Tooling and Automation: Create and enhance internal tools to support CI/CD pipelines, experiment tracking, and data versioning workflows.
* Data Quality Assurance: Ensure data integrity and accuracy by implementing robust data quality processes and monitoring solutions.
* System Design: Architect and optimize distributed software systems to meet high performance, scalability, and reliability requirements.
* Collaboration: Work closely with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data solutions that drive business insights.
Qualifications
* Extensive experience in software development with Python in high-performance, large-scale production environments.
* Hands-on expertise with Apache Spark and PySpark for distributed data processing.
* Strong knowledge of data modeling and handling both structured and unstructured data.
* Significant experience working with Apache Kafka for real-time data streaming and messaging.
* Proven experience with cloud environments (e.g., AWS, GCP, Azure).
* Expertise in designing and implementing distributed software systems.
* Strong problem-solving abilities and excellent communication skills.
Nice-to-Have Skills
* Experience with the Databricks Platform.
* Familiarity with streaming data and frameworks for real-time processing (e.g., Flink, Spark Streaming).
* Knowledge of NoSQL databases, such as Redis and Neo4j.
* Understanding of ML algorithms and the ML lifecycle.
Area: Data and BI
Location: Lisboa
Contract Type: FULL_TIME
Specialization: Tecnologias de informação
Industry: TI
Salary: Negotiable
Work Type: Híbrido
Experience Level: Gerente
Job Reference: 2310342/001
Posted Date: 25 de Novembro de 2024
Consultant: Cátia Carlos
#J-18808-Ljbffr