Sr. Bigdata developer

Job Role: Sr. Bigdata developer Location: Toronto, ON (4 Days/Week Onsite) Job Type: Contract Note: Face-to-face (F2F) interview is required Job Description: We are looking for a Sr. Big Data Developer with experience in Big Data Engineering, API Integration, and AI-assisted development. The ideal candidate will design, build, and maintain scalable data pipelines and backend systems in an enterprise environment. Key Responsibilities Big Data & Spark Design and develop Spark-Scala applications for large-scale data processing on Hadoop/CDP clusters Build and optimize ETL/ELT pipelines using Spark Data Frames, Datasets and Spark SQL Tune Spark jobs for performance (partitioning, caching, broadcast joins, shuffle optimization) Migrate Spark 2 applications to Spark 3 on Cloudera CDP platforms Work with Parquet, ORC, Avro file formats on HDFS SQL & Data Engineering Write complex HiveQL / Spark SQL queries including window functions, CTEs, subqueries and aggregations Design and maintain Hive external/managed tables and partitioned datasets Optimize slow-running queries and resolve correlated subquery issues Work with HDFS encryption zones and data governance requirements Unix / Shell Scripting Develop and maintain bash shell scripts for job orchestration and automation Handle error management, return codes, logging and alerting in shell scripts Manage HDFS operations (hdfs dfs commands), file transfers, and data validation Manage Kerberos authentication (kinit, keytab handling) API Extraction & Integration Build scripts and pipelines to extract data from REST APIs using curl and Python Handle OAuth2 token generation, bearer token refresh and API health checks Parse and process JSON API responses and load into HDFS/Hive Manage pagination, error handling and retry logic for API calls Work with enterprise API gateways and URL parameter construction AI & Copilot Capabilities Leverage GitHub Copilot / AI coding assistants to accelerate development Use AI tools for code review, SQL generation, script debugging and documentation Contribute to AI-assisted data quality and anomaly detection pipelines Explore and implement LLM-based automation for repetitive data engineering tasks Scheduling & Orchestration Schedule and manage jobs using AAP (Ansible Automation Platform) / Control-M / cron Build and maintain Ansible playbooks for automated deployments Manage deployment pipelines including artifact versioning, Vault secret injection and environment-specific configuration Monitor job health, handle failures and implement alerting Nice to Have Experience with Cloudera CDP (7.x) and migration from HDP Knowledge of Kerberos, Vault, HDFS encryption zones Familiarity with CI/CD pipelines (Helios, GitHub Actions) Experience with MSSQL / JDBC connectivity from Spark Understanding of AML / Financial regulatory data domains

Design, build, and maintain scalable data pipelines and backend systems using Spark, Scala, and Hadoop. Manage API integrations, automate workflows with shell scripting and Ansible, and leverage AI tools to optimize development processes.

The candidate must have extensive experience in Big Data engineering, specifically with Spark, Hive, and Cloudera platforms. Proficiency in API extraction, shell scripting, and utilizing AI coding assistants is required.

Relevées dans la description du poste. Confirmez les exigences importantes ci-dessus.

• Big Data Engineering
• Spark
• Scala
• Hadoop
• ETL/ELT
• HiveQL
• Spark SQL
• Unix Shell Scripting
• API Integration
• Python
• REST APIs
• GitHub Copilot
• AI-assisted development
• Ansible
• Control-M
• Cloudera CDP

• Data & Analytics
• Engineering
• Software
• Technology
• Consulting

Expérience minimale

5+ ans

Postuler avant le

20 août 2026

Langue de l’offre

anglais

Heures de travail

40 heures par semaine

Présence au bureau

4 jours par semaine

Niveau d’expérience

Mid-Senior level

Mode de candidature

La candidature directe est offerte