Technical Lead - Data Engineer (Data&AI)
Job description
Overview
We are looking for a Lead Data Engineer who combines hands-on multi-platform expertise with strong leadership in data architecture, pipelines, and CI/CD. This role requires a versatile engineer with deep technical skills across modern data platforms (such as Databricks, Snowflake, AWS, and Azure), an understanding of MLOps/DevOps practices, and the ability to guide a high-performing team in building scalable, production-ready data solutions. You will not be limited to a single platform but will leverage a diverse toolkit to solve complex data challenges.
We are looking for a Lead Data Engineer who combines hands-on multi-platform expertise with strong leadership in data architecture, pipelines, and CI/CD. This role requires a versatile engineer with deep technical skills across modern data platforms (such as Databricks, Snowflake, AWS, and Azure), an understanding of MLOps/DevOps practices, and the ability to guide a high-performing team in building scalable, production-ready data solutions. You will not be limited to a single platform but will leverage a diverse toolkit to solve complex data challenges.
Key Responsibilities
Key Responsibilities
· Pipeline & Architecture: Lead hands-on development of scalable ETL/ELT pipelines, data models, and integration frameworks to process high-volume (billions of records) structured and unstructured retail data.
·
·
Pipeline & Architecture:
Pipeline & Architecture:
Lead hands-on development of scalable ETL/ELT pipelines, data models, and integration frameworks to process high-volume (billions of records) structured and unstructured retail data.
· Multi-Platform Engineering: Design, develop, and optimize data processing applications across multiple platforms, including Databricks (Spark/Delta Lake), Snowflake, AWS, or Azure.
·
·
Multi-Platform Engineering:
Multi-Platform Engineering:
Design, develop, and optimize data processing applications across multiple platforms, including Databricks (Spark/Delta Lake), Snowflake, AWS, or Azure.
· Data Integration & Orchestration: Build and manage robust data pipelines using Apache Airflow for orchestration and Airbyte for seamless data integration and movement.
·
·
Data Integration & Orchestration:
Data Integration & Orchestration:
Build and manage robust data pipelines using Apache Airflow for orchestration and Airbyte for seamless data integration and movement.
· Data Processing: Architect and implement robust solutions for Change Data Capture (CDC), large-scale batch processing, and low-latency real-time/streaming data processing.
·
·
Data Processing:
Data Processing:
Architect and implement robust solutions for Change Data Capture (CDC), large-scale batch processing, and low-latency real-time/streaming data processing.
· API Management: Work extensively with external APIs for data ingestion, as well as design, create, and manage internal REST APIs to serve data to downstream applications and users.
·
·
API Management:
API Management:
Work extensively with external APIs for data ingestion, as well as design, create, and manage internal REST APIs to serve data to downstream applications and users.
· AI-Augmented Deliverables: Actively leverage AI assistants to conceptualize, design, and accelerate the development of data pipelines and everyday engineering tasks.
·
·
AI-Augmented Deliverables:
AI-Augmented Deliverables:
Actively leverage AI assistants to conceptualize, design, and accelerate the development of data pipelines and everyday engineering tasks.
· DevOps & CI/CD: Own and evolve CI/CD pipelines (Git workflows, automated testing, release cycles, secrets management, documentation). Guide DevOps-oriented deployments utilizing Dockerized applications, Kubernetes orchestration, and monitoring/logging tools (Splunk, Datadog, Dynatrace).
·
·
DevOps & CI/CD:
DevOps & CI/CD:
Own and evolve CI/CD pipelines (Git workflows, automated testing, release cycles, secrets management, documentation). Guide DevOps-oriented deployments utilizing Dockerized applications, Kubernetes orchestration, and monitoring/logging tools (Splunk, Datadog, Dynatrace).
· MLOps Alignment: Collaborate with Data Scientists on data readiness for ML projects and ensure alignment with ML lifecycle stages (data prep, feature engineering, model deployment).
·
·