About the Role
We are seeking a skilled DataOps Engineer to join our global team, focusing on optimizing and automating data workflows and cloud infrastructure. You will collaborate closely with Data Scientists, Data Engineers, and DevOps Engineers to ensure smooth and reliable data operations from ingestion to deployment.
Your primary responsibility will be managing and optimizing data pipelines, while maintaining performance and scalability across our cloud-based infrastructure.
Key Responsibilities
* 1. Develop and Manage Data Pipelines: Build, maintain, and optimize scalable data pipelines to ensure smooth data flow and integration across systems.
* 2. Implement Data Streaming and Processing Solutions: Manage and optimize real-time data streaming and processing systems to handle large-scale, low-latency data operations.
* 3. API Development and Integration: Create and manage APIs to facilitate integration with data systems, enabling smooth data services for internal stakeholders.
* 4. Monitor and Automate Data Workflows: Ensure the automation and continuous monitoring of data workflows, implementing performance improvements to enhance system efficiency.
* 5. Collaboration Across Teams: Collaborate closely with Data Scientists, Data Engineers, and DevOps Engineers to enhance data workflows and optimize processes for operational efficiency.
* 6. Data Visualization and Reporting: Utilize data visualization techniques to monitor pipeline performance and ensure data processes meet business and operational requirements.
Your Qualifications – Must Have
* 1. Education: Bachelor's Degree in Computer Science, Data Engineering, Software Engineering, Information Technology, or a related field. Equivalent experience in DataOps, Cloud Computing, or Data Engineering will also be considered.
* 2. Experience: 3+ years of experience in data engineering, DevOps, or a related DataOps role, with practical knowledge of cloud-based data systems and workflow automation.
* 3. Programming Skills: Strong proficiency in Python and SQL for building and optimizing data workflows.
* 4. Containerization Knowledge: Solid experience with Docker for managing and deploying data services.
* 5. Cloud Expertise: Experience with core AWS services such as S3, Lambda, and RDS for cloud-based data operations.
* 6. Data Streaming: Strong experience with Apache Kafka for real-time streaming and low-latency data processing.
* 7. Automation & Monitoring: Proficiency in automating workflows using AWS Wrangler, Boto3, and monitoring tools like Grafana and AWS CloudWatch.
Nice to Have Qualifications
* 1. API Development: Familiarity with FastAPI, Pydantic, or similar frameworks for building APIs.
* 2. Kafka Connect & KSQLDB: Hands-on experience with Kafka Connect and KSQLDB for advanced streaming functionality.
* 3. Kubernetes: Knowledge of Kubernetes for orchestrating containerized applications.
* 4. MLOps Frameworks: Familiarity with MLFlow or Kubeflow to support machine learning model deployment and operations.
* 5. Data Validation & Testing: Experience with tools like Great Expectations or dbt for data validation and testing.
About Siemens Mobility
You will be part of a global community that is passionate about shaping the future of transportation. We offer a competitive salary, comprehensive benefits, and opportunities for professional growth and development within an innovative and dynamic environment.