Databricks Engineer

Sr. Client Partner in San FranciscoMilpitas, United Statesfull timeMid Level
Active

Job description

Role: Databricks EngineerExperience4–8 yearsLocationAs per business requirementEmployment TypeFull-timeJob SummaryWe are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Databricks.Build ETL/ELT workflows for batch and streaming data processing.Develop solutions using PySpark, Spark SQL, and Delta Lake.Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.Optimize Spark jobs for performance, scalability, and cost efficiency.Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.Implement CI/CD pipelines and deployment automation for Databricks workloads.Ensure data quality, security, governance, and compliance.Monitor, troubleshoot, and optimize production data pipelines.Document technical solutions and follow engineering best practices.Required SkillsCore TechnologiesDatabricks Lakehouse PlatformApache SparkPySparkSpark SQLDelta LakePythonSQLCloud Platforms (one or more)Microsoft Azure (preferred)AWSGoogle Cloud PlatformAzure Technologies (Preferred)Azure Data Factory (ADF)Azure Data Lake Storage (ADLS Gen2)Azure Synapse AnalyticsAzure Key VaultAzure DevOpsData EngineeringData WarehousingData ModelingETL/ELT DevelopmentBatch ProcessingStreaming (Kafka/Event Hubs)Data Lake ArchitectureDevOps & Version ControlGitAzure DevOps / GitHubCI/CD PipelinesPreferred QualificationsExperience with Unity Catalog.Knowledge of Databricks Workflows and Jobs.Hands-on experience with Delta Live Tables (DLT).Exposure to MLflow is an added advantage.Experience with data governance and security best practices.Familiarity with Infrastructure as Code (Terraform) is a plus.Educational QualificationBachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.Preferred CertificationsDatabricks Certified Data Engineer AssociateDatabricks Certified Data Engineer ProfessionalMicrosoft Certified: Azure Data Engineer Associate (DP-203)Azure Fundamentals (AZ-900)Good to HaveExperience with real-time analytics.Knowledge of Lakehouse architecture.Experience with Agile/Scrum methodologies.Strong analytical and problem-solving skills.Excellent communication and stakeholder management abilities.Mandatory SkillsDatabricksPySparkSpark SQLDelta LakePythonSQLAzure/AWS/GCP (at least one cloud platform)ETL/ELT DevelopmentData Lake ArchitectureNice to HaveUnity CatalogDelta Live Tables (DLT)MLflowKafka/Event HubsAzure Data FactoryTerraformAzure DevOps/GitHub ActionsRole: Databricks EngineerExperience4–8 yearsLocationAs per business requirementEmployment TypeFull-timeJob SummaryWe are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using Databricks.Build ETL/ELT workflows for batch and streaming data processing.Develop solutions using PySpark, Spark SQL, and Delta Lake.Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.Optimize Spark jobs for performance, scalability, and cost efficiency.Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.Implement CI/CD pipelines and deployment automation for Databricks workloads.Ensure data quality, security, governance, and compliance.Monitor, troubleshoot, and optimize production data pipelines.Document technical solutions and follow engineering best practices.Required SkillsCore TechnologiesDatabricks Lakehouse PlatformApache SparkPySparkSpark SQLDelta LakePythonSQLCloud Platforms (one or more)Microsoft Azure (preferred)AWSGoogle Cloud PlatformAzure Technologies (Preferred)Azure Data Factory (ADF)Azure Data Lake Storage (ADLS Gen2)Azure Synapse AnalyticsAzure Key VaultAzure DevOpsData EngineeringData WarehousingData ModelingETL/ELT DevelopmentBatch ProcessingStreaming (Kafka/Event Hubs)Data Lake ArchitectureDevOps & Version ControlGitAzure DevOps / GitHubCI/CD PipelinesPreferred QualificationsExperience with Unity Catalog.Knowledge of Databricks Workflows and Jobs.Hands-on experience with Delta Live Tables (DLT).Exposure to MLflow is an added advantage.Experience with data governance and security best practices.Familiarity with Infrastructure as Code (Terraform) is a plus.Educational QualificationBachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.Preferred CertificationsDatabricks Certified Data Engineer AssociateDatabricks Certified Data Engineer ProfessionalMicrosoft Certified: Azure Data Engineer Associate (DP-203)Azure Fundamentals (AZ-900)Good to HaveExperience with real-time analytics.Knowledge of Lakehouse architecture.Experience with Agile/Scrum methodologies.Strong analytical and problem-solving skills.Excellent communication and stakeholder management abilities.Mandatory SkillsDatabricksPySparkSpark SQLDelta LakePythonSQLAzure/AWS/GCP (at least one cloud platform)ETL/ELT DevelopmentData Lake ArchitectureNice to HaveUnity CatalogDelta Live Tables (DLT)MLflowKafka/Event HubsAzure Data FactoryTerraformAzure DevOps/GitHub Actions

Role: Databricks Engineer

Experience

4–8 years

4–8 years

Location

As per business requirement

Employment Type

Full-time

Job Summary

We are looking for a skilled Databricks Engineer with strong expertise in designing, developing, and optimizing modern data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have experience building scalable ETL/ELT pipelines, working with large-scale data, and leveraging Apache Spark to deliver high-performance data solutions.

Databricks Engineer

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Databricks.
  • Build ETL/ELT workflows for batch and streaming data processing.
  • Develop solutions using PySpark, Spark SQL, and Delta Lake.
  • Implement Medallion Architecture (Bronze, Silver, Gold) for data transformation.
  • Integrate data from various sources including relational databases, APIs, cloud storage, and streaming platforms.
  • Optimize Spark jobs for performance, scalability, and cost efficiency.
  • Collaborate with Data Architects, Data Scientists, BI developers, and business stakeholders.
  • Implement CI/CD pipelines and deployment automation for Databricks workloads.
  • Ensure data quality, security, governance, and compliance.
  • Monitor, troubleshoot, and optimize production data pipelines.
  • Document technical solutions and follow engineering best practices.

Required Skills

Core Technologies

  • Databricks Lakehouse Platform
  • Apache Spark
  • PySpark
  • Spark SQL
  • Delta Lake
  • Python
  • SQL

Cloud Platforms (one or more)

  • Microsoft Azure (preferred)
  • AWS
  • Google Cloud Platform

Azure Technologies (Preferred)

  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS Gen2)
  • Azure Synapse Analytics
  • Azure Key Vault
  • Azure DevOps

Data Engineering

  • Data Warehousing
  • Data Modeling
  • ETL/ELT Development
  • Batch Processing
  • Streaming (Kafka/Event Hubs)
  • Data Lake Architecture

DevOps & Version Control

  • Git
  • Azure DevOps / GitHub
  • CI/CD Pipelines

Preferred Qualifications

  • Experience with Unity Catalog.
  • Knowledge of Databricks Workflows and Jobs.
  • Hands-on experience with Delta Live Tables (DLT).
  • Exposure to MLflow is an added advantage.
  • Experience with data governance and security best practices.
  • Familiarity with Infrastructure as Code (Terraform) is a plus.

Educational Qualification

  • Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred Certifications

  • Databricks Certified Data Engineer Associate
  • Databricks Certified Data Engineer Professional
  • Microsoft Certified: Azure Data Engineer Associate (DP-203)
  • Azure Fundamentals (AZ-900)

Good to Have

  • Experience with real-time analytics.
  • Knowledge of Lakehouse architecture.
  • Experience with Agile/Scrum methodologies.
  • Strong analytical and problem-solving skills.
  • Excellent communication and stakeholder management abilities.

Mandatory Skills

  • Databricks
  • PySpark
  • Spark SQL
  • Delta Lake
  • Python
  • SQL
  • Azure/AWS/GCP (at least one cloud platform)
  • ETL/ELT Development
  • Data Lake Architecture

Nice to Have

  • Unity Catalog
  • Delta Live Tables (DLT)
  • MLflow
  • Kafka/Event Hubs
  • Azure Data Factory
  • Terraform
  • Azure DevOps/GitHub Actions
I'm interested{{criteriaIndex}} {{field}} {{captialize(condition)}} ( ) {{cxPropHelpText}} {{cxPropField.consent_details.description}} {{record.Posting_Title}} {{record.Posting_Title}} {{record.Posting_Title}} {{candidate.Email}} {{topMessage}} Job Details {{topMessage}} previous next {{title}} {{score}}% Job Description {{unescape(sanitizeHTML(descriptionHTML))}} {{message}} Step {{curStepInMandatorySecPrompt}}/{{totalNumOfStepsInMandatorySecPrompt}} For privacy and security purposes, please go through the following points and provide consent. Accept Decline {{getI18n("zr.careers.publicpage.meta.joblisting")}} {{getI18n("zr.careers.publicpage.meta.joblisting")}} {{cxPropAssessment.name}} {{getI18n("zr.questionnaire.question.information.heading")}} {{getI18n("zr.cw.autofill.head")}} {{cxPropBodyMessage}} {{getI18n("zr.careers.autopopulate.success")}} {{getI18n("zr.cw.apply.sign",cxPropCompanyInfo.name)}} {{if(cxPropDetails.applyWithSeekButton,cxPropDetails.applyWithSeekButton.buttonLabel,getI18n("zr.quickapply.apply.toggle","Seek"))}} {{getI18n('zr.eeo.questionnaire.portal.maintitle')}} {{botName}} {{getI18n("zr.zia.sb.consent.msg1")}}{{criteriaIndex}}{{criteriaIndex}}{{field}} {{captialize(condition)}}{{field}}{{captialize(condition)}}( )(){{cxPropHelpText}}{{cxPropHelpText}}{{cxPropField.consent_details.description}}

{{cxPropField.consent_details.description}}

{{record.Posting_Title}}{{record.Posting_Title}}{{record.Posting_Title}}

{{record.Posting_Title}}

{{record.Posting_Title}}{{record.Posting_Title}}

{{record.Posting_Title}}

{{record.Posting_Title}}
  • {{record.Posting_Title}}

{{record.Posting_Title}}

{{record.Posting_Title}}{{candidate.Email}}{{candidate.Email}}{{candidate.Email}}

{{candidate.Email}}

{{topMessage}}Job DetailsJob DetailsJob DetailsJob Details{{topMessage}}previousnext{{title}} {{score}}% Job Description {{unescape(sanitizeHTML(descriptionHTML))}}{{title}} {{score}}% Job Description {{unescape(sanitizeHTML(descriptionHTML))}}{{title}} {{score}}%

{{title}}

{{title}}{{score}}%Job Description {{unescape(sanitizeHTML(descriptionHTML))}}

Job Description

{{message}}

Step {{curStepInMandatorySecPrompt}}/{{totalNumOfStepsInMandatorySecPrompt}}

For privacy and security purposes, please go through the following points and provide consent.Accept DeclineAccept DeclineAcceptDecline{{getI18n("zr.careers.publicpage.meta.joblisting")}}{{getI18n("zr.careers.publicpage.meta.joblisting")}}
  • {{getI18n("zr.careers.publicpage.meta.joblisting")}}
{{getI18n("zr.careers.publicpage.meta.joblisting")}}{{getI18n("zr.careers.publicpage.meta.joblisting")}}{{getI18n("zr.careers.publicpage.meta.joblisting")}}
  • {{getI18n("zr.careers.publicpage.meta.joblisting")}}
{{getI18n("zr.careers.publicpage.meta.joblisting")}}{{cxPropAssessment.name}} {{getI18n("zr.questionnaire.question.information.heading")}}{{cxPropAssessment.name}} {{getI18n("zr.questionnaire.question.information.heading")}}{{cxPropAssessment.name}}

{{getI18n("zr.questionnaire.question.information.heading")}}

{{getI18n("zr.cw.autofill.head")}} {{cxPropBodyMessage}} {{getI18n("zr.careers.autopopulate.success")}}

{{getI18n("zr.cw.autofill.head")}}

{{cxPropBodyMessage}}

{{getI18n("zr.careers.autopopulate.success")}}

{{getI18n("zr.cw.apply.sign",cxPropCompanyInfo.name)}}

{{getI18n('zr.eeo.questionnaire.portal.maintitle')}}{{botName}}{{botName}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}{{getI18n("zr.zia.sb.consent.msg1")}}