Senior Data Engineer - Databricks

The HP Ecommerce CRM enablement team is seeking for its CDP program a proven and hands-on Databricks process architect and engineering leader to drive our Databricks transformation, pipelines architecture, governance and engineering patterns. Our CDP delivers a unified Customer data model across our global CRM operations. In this role, you will be responsible for driving our Databricks migration, designing and executing a comprehensive pipeline architecture strategy that aligns with CDP team's skillsets, business objectives and drives compliant, efficient and effective data management.

Responsibilities

• Drive our AWS Databricks transformation from current relational DB setup.

• Collaborate with our diverse teams, SQL developers/data engineers/CRM analysts, to develop, implement and maintain sustainable, high-performance, growth-ready data-processing and data integration systems

• Provide best practice on CI/CD techniques with Git and Databricks

• Provide best practice on Databricks cluster management.

• Reviews and evaluates designs and project activities for compliance with privacy, architecture, security and quality guidelines and standards; provides tangible feedback to improve product quality and mitigate failure risk.

• Design and maintain an appropriate metadata repository to enable the organization to understand its data assets.

• Builds and manages relationships throughout the organization.

• Drives innovation and integration of new technologies into projects and activities in the big data space.

• Collaborates and communicates with project team regarding project progress and issue resolution.

• Provides guidance and mentoring to less experienced staff members.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent.

• 10+ years of experience in data architecture, database design, and data management.

• Expert in AWS Databricks

• Expert in SQL & relational database systems (PostgreSQL preferred)

• Proficient knowledge in working with distributed data processing frameworks like Apache Spark.

• Hands-on experience in Python and Pandas

• Experience in overall architecture of big data systems within AWS

• Experience in Global Companies a great advantage.

Knowledge & Skills

• Extensive experience with designing data engineering solutions to manage data streams in complex, distributed and parallel systems.

• Excellent analytical and problem-solving skills. Ability to collaborate with and analyze source systems to drive data profiling/cataloging. Can present clear insights to colleagues to support the end use of the data.

• Extensive experience with data engineering pipelines, frameworks to mine, cleanse and explore data.

• Strong problem management. Initiating and monitoring actions to investigate patterns and trends to resolve problems. Determining preventative measures.

• Evaluating forms and processes for database architecture testing and methodology, including writing and execution of test plans, debugging, and testing scripts and tools.

• Excellent written and verbal communication skills; mastery in English and local language.

Complexity

• Applies advanced subject matter knowledge to solve complex business issues and is regarded as a subject matter expert.

• Frequently contributes to the development of new ideas and methods.

• Works on complex problems where analysis of situations or data requires an indepth evaluation of multiple factors.

• Acts as an expert providing direction and guidance to process improvements and establishing policies.

• Exercises significant independent judgment within broadly defined policies and practices to determine best method for accomplishing work and achieving objectives.

• Provide mentoring and guidance to lower level employees.