Job description
Data & AI EngineerProduction EngineeringGovernance & FederationJob DescriptionWe are looking for an experienced Databricks Solution Architect to join our team. You will design and lead end-to-end big data and analytics solutions on the Databricks Lakehouse Platform, translating complex business requirements into scalable, high-performance architectures. You'll work closely with clients and internal engineering teams to architect data pipelines, transformation frameworks, and governance models that ingest and process petabyte-scale data from diverse batch and streaming sources into cloud data lakes.Key Responsibilities›Next-Gen Architecture Delivery: Design and deploy modern Data Lakehouse solutions using Serverless Compute (for SQL, Jobs, and Notebooks) to eliminate infrastructure overhead and optimize TCO.to understand the business problem ›Production Engineering: Write, optimize, and deploy production-grade code. You are expected to fix what breaks—whether it’s a shuffling error in Spark, a CI/CD failure in Databricks Asset Bundles (DABs), or a complex Mosaic AI vector search pipeline.›Deep Performance Tuning: Diagnose and resolve bottlenecks (Skew, OOM, Spill). You must know when to apply legacy tuning (Z-Order/Partitioning) vs. modern Liquid Clustering to handle changing data patterns automatically›Governance & Federation: Implement strict data governance using Unity Catalog. Configure Lakehouse Federation to query external systems without data movement, and enforce Attribute-Based Access Control (ABAC).›Customer Obsession: Act as the 'Expert in the Room.' Explain complex concepts (like why Serverless reduces cold starts or how AI/BI Genie handles hallucination) to non-technical stakeholders.QualificationsLocation & Work TypeAll CT office / Onsite ( CT office )Full TimeAbout Celebal technologiesCelebal Technologies is a leading digital transformation company helping enterprises leverage AI, data engineering and cloud technologies to build scalable platforms.Apply for this positionSubmit ApplicationApplication takes about 5 minutesData & AI EngineerProduction EngineeringGovernance & FederationData & AI EngineerProduction EngineeringGovernance & FederationData & AI EngineerProduction EngineeringGovernance & Federation