DESCRIPTION
Currently, we are looking for a Lead Data Engineer with Python and Databricks expertise to make our team even stronger.
Responsibilities
- Develop, monitor, and operate the most used and most critical curated data pipeline - Sales Order Data (incl. Post-order information, e.g. shipment, return, payment). This pipeline is processing hundreds of millions of records to provide high-quality datasets for analytical and machine learning use-cases
- Consul with analysts, data scientists, and product managers to build and continuously improve "Single Source of Truth" KPI for business steering such as the central Profit Contribution measurement (PC II)
- Redevelop old legacy pipelines to new, advanced and standard versions that are easy to maintain and scalable for future demands
- Leverage and improve a tech stack that includes Python, Databricks, Kubernetes, Spark, and Airflow
Want more jobs like this?
Get Software Engineering jobs in Entroncamento, Portugal delivered to your inbox every week.
- Fluency in Python programming language
- Good hands on experience with Databricks
- Expertise in Apache Spark along with Spark streaming
- Good understanding & hands-on experience with CI/CD
- Rich working experience with Github
- Upper-Intermediate level of English, both spoken and written (B2+)
- Team player (easy & respectful communications, shares responsibilities for the team overall success)
- Communication skills
- Organizational skills
- Delta-Lake
- Expertise in SQL
- Fluency working with AWS landscape
- Ability to build Apache Airflow pipelines
- Presto
- Superset
- Starburst
- Oracle & Exasol
- Competitive compensation depending on experience and skills
- Variety of projects within one company
- Being a part of a project following engineering excellence standards
- Individual career path and professional growth opportunities
- Internal events and communities
- Flexible work hours