Data engineer

Job Description

Assignment description
For our client we are looking for a Data Engineer.

Our client is looking for a skilled Data Engineer to join the team and work on cutting-edge autonomous vehicle data solutions. In this role, you will design, develop, and optimize data pipelines, ensuring efficient data processing for autonomous vehicle applications. They exptect you to have a passion for clean code, scalable architecture, and high-performance data solutions.

Key Responsibilities

✔️ Develop and maintain data pipelines using Python, PySpark, and Airflow

✔️ Work with PostgreSQL, NoSQL databases, and on-premise solutions

✔️ Design and implement enhancements for the data lakehouse, with medallion architecture

✔️ Build and optimize ETL processes for upscaling towards large-scale data handling

✔️ Implement CI/CD processes for data pipelines and end-to-end solutions

✔️ Work with Docker, Linux, and Shell scripting

✔️ Collaborate in a GitLab/GitHub environment for version control

✔️ Ensure clean, maintainable, and high-quality code

✔️ Process and manage data in MCAP format for autonomous vehicle applications

✔️ Contribute to autonomous development in the agile team using Jira.

✔️ Handle CAN bus data and support automotive development

✔️ Close collaboration with the data architect contributing with data enginnering solutions

What They’re Looking For

🔹 Strong experience in Python and software development best practices

🔹 Background in data engineering with expertise in data processing

🔹 Experience with GitLab/GitHub, Docker, Linux, and Shell scripting

🔹 Knowledge of CI/CD pipelines and on-prem solutions

🔹 Understanding of ETL workflows, medallion architecture, and data lakehouses

🔹 Experience with MCAP format for data processing in autonomous vehicle systems

🔹 Automotive and autonomous vehicle industry experience or knowledge of CAN bus data is a plus