Poste :

• We are looking for an experienced Data Engineer who will join our Data Analytics team and will assume the Dev tech Lead role to support the development activities of the client data platform.
• The core of the work delivered by our team is to provide data engineering, development, testing, platform support and application support services for the data platform.
• Focus of the data engineering and development team is build data pipelines to properly ingest data from source systems into the data platform, make the necessary data transformations (ELT), store and make the data available for the consumption (reporting, self-serve BI, AI/ML models)
• Data platform is fully based on Azure Cloud and Databricks is heavily used for processing and transforming the data.
• Data platform contains all kind of (business) data to build an advanced analytics platform aiming at delivering better insights and applications to the business.
• The platforms are continuously being enhanced to support (additional) CI/CD and validated learning environment for science, machine learning and AI capabilities for all areas customer-facing like digital omni-channel interaction and commerce, commerce relevance, personalization, loyalty and marketing and non-customer-facing like assortment optimization, supply chain optimization, external parties and IoT.
• We will be working on end to end functionality including architecture, data preparation, processing and consumption by systems.
• All development is done by agile teams that are working with SAFe framework.

•Responsibilities:
• Work in collaboration with each development team to;
o provide guidance, consultation, best practices and standard approach for the development
o when necessary, deepdive into the code and provide hands-on support to data engineering teams to overcome any roadblocks
o be the single point of contact across the development teams
• Work closely with platform architecture team to
o be the link between architectural design and day-to-day development
o define the technical backlog items to improve/enhance the platform
• Scope of the development teams can be summarized as
o Designing and implementing batch and real-time data ingestion pipelines for Azure Data Lake
o Implementing data quality checks
o Implementing data lineage and reconciliation mechanisms (including monitoring dashboards and alerts)
o Implementing data cataloguing, archiving and disaster recovery
o Working with data source and data consumption teams to align on data structures and schemas
• Participate in client sessions for business backlog and technical backlog refinement sessions and take part in PI planning sessions
 

Profil recherché :

• Mandatory Skills:
  • Mastery in modern, cloud based data/BI technologies (data lakes, delta lakes, data warehouses, ETL/ELT, Spark)
  • 7 years Experience with implementing data ingestion pipelines / ETLs / ELTs
  • In depth knowledge of Azure platform and ecosystem - Storage Account, CosmosDB (SQL & Gremlin API), Event Hub, App Services
  • Experience with at following Azure data technologies
  o Azure SQL
  o Azure Data Lake
  o Azure Databricks (SQL or Python, Delta Lake)
  o Azure Data Factory
  o Azure Analysis Services
  o Experience with SQL
  • Good communication
  • Client Facing
• Nice-to-Have Skills:
  • Python, C#/.NET
  • Azure Stream Analytics
•