Sr Data Engineer -

Sr. Client Partner in San FranciscoEast Toronto (Studio District), Canadafull timeSenior
Active

Job description

Job SummaryWe are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems.Key ResponsibilitiesDesign and implement scalable ETL pipelines using Databricks & SparkBuild Lakehouse architecture using Delta LakeDevelop and deploy ML models using MLflowImplement MLOps pipelines for training, testing, and serving modelsOptimize cluster performance and reduce compute costBuild RAG and LLM-based solutions using Mosaic AIIntegrate analytics with BI tools (Power BI, Tableau)Implement data governance using Unity CatalogCollaborate with Data Scientists and Business teamsEnsure data quality, security, and complianceRequired SkillsMandatory5+ years of Databricks & Apache SparkStrong Python & PySparkExperience with Delta Lake & LakehouseMLflow & MLOps experienceCloud platform (AWS/Azure/GCP)Git & CI/CDPreferredExperience with LLMs & Generative AIRAG pipelines & Vector DatabasesDeep Learning frameworksDatabricks certificationsPower BI integration RequirementsJob SummaryWe are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems.Key ResponsibilitiesDesign and implement scalable ETL pipelines using Databricks & SparkBuild Lakehouse architecture using Delta LakeDevelop and deploy ML models using MLflowImplement MLOps pipelines for training, testing, and serving modelsOptimize cluster performance and reduce compute costBuild RAG and LLM-based solutions using Mosaic AIIntegrate analytics with BI tools (Power BI, Tableau)Implement data governance using Unity CatalogCollaborate with Data Scientists and Business teamsEnsure data quality, security, and complianceRequired SkillsMandatory5+ years of Databricks & Apache SparkStrong Python & PySparkExperience with Delta Lake & LakehouseMLflow & MLOps experienceCloud platform (AWS/Azure/GCP)Git & CI/CDPreferredExperience with LLMs & Generative AIRAG pipelines & Vector DatabasesDeep Learning frameworksDatabricks certificationsPower BI integrationJob SummaryWe are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems.Key ResponsibilitiesDesign and implement scalable ETL pipelines using Databricks & SparkBuild Lakehouse architecture using Delta LakeDevelop and deploy ML models using MLflowImplement MLOps pipelines for training, testing, and serving modelsOptimize cluster performance and reduce compute costBuild RAG and LLM-based solutions using Mosaic AIIntegrate analytics with BI tools (Power BI, Tableau)Implement data governance using Unity CatalogCollaborate with Data Scientists and Business teamsEnsure data quality, security, and complianceRequired SkillsMandatory5+ years of Databricks & Apache SparkStrong Python & PySparkExperience with Delta Lake & LakehouseMLflow & MLOps experienceCloud platform (AWS/Azure/GCP)Git & CI/CDPreferredExperience with LLMs & Generative AIRAG pipelines & Vector DatabasesDeep Learning frameworksDatabricks certificationsPower BI integration

Job Summary

Job Summary

We are seeking a highly skilled Databricks Engineer with AI/ML experience to design, build, and optimize scalable data and machine learning platforms on Databricks. The role involves end-to-end ownership of data pipelines, ML workflows, and production AI systems.

Databricks Engineer with AI/ML experience

Key Responsibilities

Key Responsibilities
  • Design and implement scalable ETL pipelines using Databricks & Spark
  • Build Lakehouse architecture using Delta Lake
  • Develop and deploy ML models using MLflow
  • Implement MLOps pipelines for training, testing, and serving models
  • Optimize cluster performance and reduce compute cost
  • Build RAG and LLM-based solutions using Mosaic AI
  • Integrate analytics with BI tools (Power BI, Tableau)
  • Implement data governance using Unity Catalog
  • Collaborate with Data Scientists and Business teams
  • Ensure data quality, security, and compliance

Required Skills

Required Skills

Mandatory

Mandatory
  • 5+ years of Databricks & Apache Spark
  • Strong Python & PySpark
  • Experience with Delta Lake & Lakehouse
  • MLflow & MLOps experience
  • Cloud platform (AWS/Azure/GCP)
  • Git & CI/CD

Preferred

Preferred
  • Experience with LLMs & Generative AI
  • RAG pipelines & Vector Databases
  • Deep Learning frameworks
  • Databricks certifications
  • Power BI integration

Similar jobs