Lead Data Engineer

Streaming data pipelinesMedallion ArchitectureDatabricks AutoloaderJob DescriptionCelebal Technologies is a leading Solution Service company that provide Services the field of Data Science, Big Data, Enterprise Cloud & Automation. We are at the forefront of leveraging cutting edge technologies to drive innovation and enhance our business processes. As part of our commitment to staying ahead in the industry, we are seeking a talented and experienced Data & AI Engineer with strong Azure cloud competencies to join our dynamic team. Key Responsibilities›Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming.›Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers.›Implement efficient ingestion using Databricks Autoloader for high-throughput data loads.›Work with large volumes of structured and unstructured data, ensuring high availability and performance.›Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization.›Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions.›Establish best practices for code versioning, deployment automation, and data governance.QualificationsLocation & Work TypeJaipur, Noida, Gurgaon, Bengaluru, Pune, Hyderabad Work from OfficeAbout Celebal technologiesCelebal Technologies is a leading digital transformation company helping enterprises leverage AI, data engineering and cloud technologies to build scalable platforms.Apply for this positionSubmit ApplicationApplication takes about 5 minutesStreaming data pipelinesMedallion ArchitectureDatabricks AutoloaderStreaming data pipelinesMedallion ArchitectureDatabricks AutoloaderStreaming data pipelinesMedallion ArchitectureDatabricks Autoloader