Architectural Data Engineering (m/w/d)

Overview

Description

– You will design, implement and manage data flows that integrate information from multiple sources (including e.IoT) based on ELT and ETL models.

– Design, implement, and operate data analytics and AI platform services, as well as business solutions that run on the platform

– Manage and operate cloud-based (Azure and AWS) platform services using state-of-the-art tools and design patterns

– You will take responsibility for core orchestration pipelines as well as data integration architecture and technology stack.

– You consult with architecture, data platform, user interaction and solutions teams and understand their unique needs and challenges.

– You coordinate with other teams to troubleshoot data exchange issues between disparate systems.

– Consulting with internal customers

– Advise local business teams on solution design, data integration, and data engineering.

Our Requirements:

Technologies:

– Azure: Solid experience with Azure services, e.g. Synapse, Data Lake, Functions, CosmosDB, DataBricks, Data Factory, Events Hub, IoT Hub.

– Terrform, Terragrunt (nice to have)

– Python, PySpark

– CI/CD (GitLab)

– SQL

Skills & Experience:

– Proven hands-on DevOps experience (5+ years) in cloud-based environments for Data Analytics and Data Science applications.

– Relevant experience in architectural design of (streaming) data integration solutions with focus on (Confluent) Kafka.

– Sound practical knowledge of cloud services, their setup and integration.

– Practical knowledge of management and configuration services and tools.

– Programming/Scripting Skills. Python preferred, more languages welcome.

– Hands-on experience with streaming data processing, including real-time analytics and data transformation.

– Working knowledge of data processing and integration patterns, such as ETL and ELT.

– Working knowledge of ActiveDirectory / LDAP.

– Monitoring of cloud services and log analysis.

– Practical knowledge of network configuration and connectivity troubleshooting.

– Knowledge and hands-on experience around cloud security.