Job description
Who We Are
We seek a highly skilled Senior BigQuery Data Engineer with strong Python expertise to design and deliver high-performance data solutions within OneMIS. This pivotal role supports complex reporting functions and leads the migration of on-premise data platforms to Google Cloud Platform (GCP). You will manage large-scale financial datasets, ensuring secure, compliant, and low-latency data solutions for analytics and decision-making. This role demands technical mastery, problem-solving acumen, and a deep understanding of data best practices in the financial sector
What You’ll Be Doing
- Design, develop, and maintain scalable BigQuery data warehouse solutions (architecture, schema, data modeling)
- Lead cloud-native data warehouse design and implementation to replace on-premise infrastructure
- Optimize BigQuery datasets, tables, and views for performance, cost, and governance through partitioning, clustering, query tuning, and workload management
- Build robust data processing pipelines and workflows feeding BigQuery from diverse sources
- Develop and maintain efficient ETL/ELT processes for large-volume financial data ingestion, transformation, and loading
- Implement comprehensive data quality checks, validation, and reconciliation processes
- Utilize and optimize core GCP services: BigQuery, Cloud Composer (Apache Airflow) for workflow orchestration, Dataflow (Apache Beam) for batch/streaming, and Cloud Storage
- Develop, test, and deploy high-quality Python code for data processing, automation, API integrations, and custom solutions, adhering to best practices (Git, testing, documentation)
- Explore and integrate other relevant GCP technologies (e.g., GKE, Cloud Functions, Pub/Sub) and implement security, access control, and compliance
- Collaborate with product owners, analysts, and engineering teams to align solutions with business requirements
- Actively participate in agile ceremonies and foster team collaboration
- Support data needs for BI tools (Qlik Sense, Tableau, Looker).
- Apply strong understanding of banking/financial sector data, regulations, and reporting for compliance.
What We’re Looking For
- Bachelor’s degree in Computer Science, Engineering, or related field
- At least 5+ years of data engineering experience
- Strong SQL knowledge, preferably on Google Cloud Platform (GCP).
- Expert-level Python programming skills for data manipulation, scripting, and application development.
- Proven experience with Cloud Computing, demonstrating practical application of cloud services.
- Direct hands-on experience with core GCP services: BigQuery, Cloud Composer (Apache Airflow) for DAG development/orchestration, Dataflow (Apache Beam) for batch/streaming pipelines, and Cloud Storage
- Good understanding of data warehousing concepts, data flows, and data feeds
- Experience with Git-based source control, particularly Bitbucket
- Experience working in an agile development environment with familiar agile ceremonies
- Solid understanding of the banking and financial sector, including data types and regulatory landscapes
- Ability to work collaboratively in a dynamic environment
Nice to Have:
- Experience with other GCP services (GKE, Cloud Functions, Pub/Sub, Dataproc)
- Familiarity with CI/CD pipelines and DevOps practices
- Experience with Infrastructure as Code (Terraform)
- Knowledge of data visualization tools (Qlik Sense, Tableau, Looker)
- Understanding of data governance, data quality, and metadata management
- Experience with real-time data processing