Sr Data Engineer

UkgBengaluru, KA,INfull timeSenior
ActiveVerified 4h ago

Job description

Partner with business and finance stakeholders to frame data problems and deliver end-to-end solutions. Design, develop, and maintain scalable data pipelines and ELT processes using GCP and BigQuery. Build and maintain layered data platforms using medallion architecture patterns, including Bronze, Silver, and Gold layers. Create curated data products, analytical datasets, and semantic models that support reporting, analytics, and decision-making. Integrate data from enterprise applications, finance systems, operational databases, APIs, and other source platforms. Conduct data profiling, source-to-target mapping, reconciliation, cataloging, and quality assessments. Translate stakeholder requirements into practical, scalable data solutions. Document designs, data flows, lineage, transformation logic, and operating procedures. Support, troubleshoot, and enhance data pipelines and interim legacy systems. Apply standards for security, performance, scalability, testing, monitoring, and governance. Contribute to sprint planning, backlog refinement, estimation, demos, and continuous improvement. Use automation, generative AI, and agentic tools to improve engineering productivity and delivery processes. Apply AI across the data engineering lifecycle, including discovery, mapping, SQL and pipeline generation, testing, quality rules, documentation, and lineage. Build AI-native data solutions such as natural-language data access, data agents, RAG/context layers, embeddings, and semantic search. Evaluate and operationalize AI tooling, including agent orchestration, tool/function calling, MCP servers, prompt and context engineering, evaluations, and guardrails. Curate metadata, definitions, and context that make data usable by people and AI agents. Own delivery from design through deployment, adoption, and ongoing enhancement, with accountability for business value and reliable operations. 5+ years of experience in data engineering, data warehousing, ETL, or ELT. Strong SQL skills and practical Python experience for data engineering use cases. Strong understanding of data warehouse architecture, dimensional modeling, data integration, and transformation techniques. Experience designing and supporting production-grade data pipelines with monitoring, error handling, testing, and performance optimization. Working knowledge of source control, CI/CD, automated testing, and deployment practices. Strong verbal and written communication skills, with the ability to collaborate across technical teams, product owners, analysts, and business stakeholders. Strong problem-solving skills, initiative, adaptability, attention to detail, and sound technical judgment. Experience implementing medallion architecture or similar layered data-platform patterns. Experience developing reports, dashboards, or semantic models in Microsoft Power BI. Experience with finance, accounting, general ledger, accounts payable, accounts receivable, financial reporting, or reconciliation data. Knowledge of financial data models, accounting concepts, performance metrics, and reconciliation processes. Experience with additional GCP services such as Cloud Storage, Dataflow, Dataproc, Cloud Composer, Pub/Sub, or Cloud Functions. Experience with data governance, metadata management, data cataloging, and data lineage. Hands-on use of generative, agentic coding, or AI tools such as Claude or Codex for coding, automation, analysis, documentation, or workflow orchestration. Experience with prompt and context engineering, LLM APIs, RAG, embeddings or vector search, agent orchestration, or AI-assisted automation. Experience contributing to analytics products from discovery and design through delivery, adoption, and enhancement. Experience owning solutions from business problem definition to adopted, quality-assured production data products. What Success Looks Like Within 90 days, key production data pipelines have documented monitoring, error handling, support procedures, and clear ownership. High-priority BigQuery datasets have clear definitions, lineage, quality checks, and stakeholder sign-off for reporting, analytics, and finance use cases. Business stakeholders, analysts, product owners, and engineering partners consistently rely on the role for solution design, delivery updates, issue resolution, and production support. Automation and responsible AI practices measurably reduce manual effort, improve documentation quality, and make recurring delivery tasks more repeatable. Within 6-12 months, at least one agentic or AI-enabled data solution is deployed to production and adopted for a defined business or engineering use case.