Lead Data Engineer

Our Organization Optillium Inc. is a Calgary-based AI and Software Company, focused on building practical AI automation solutions for businesses, professional service firms, and operational teams. We design and implement systems that automate complex workflows, improve decision-making, reduce manual effort, and help organizations operate more efficiently. Our capabilities include AI automation, workflow orchestration, custom software development, enterprise integrations, data engineering, Microsoft cloud solutions, intelligent agents, and applied machine learning. Optillium has also developed and trained proprietary AI models for specialized business use cases, allowing us to build solutions that can be tailored to specific operational, financial, and industry requirements. We work closely with clients to understand how their processes operate today, identify opportunities for automation, and deliver secure, scalable solutions that can be adopted across their organization. Our team combines software engineering, data, AI, business analysis, and implementation experience to deliver end-to-end solutions from discovery and architecture through deployment, testing, training, and ongoing support. Optillium is based in Calgary, Alberta, and supports organizations across Canada. The Opportunity Reporting to Optillium's delivery leadership, the Lead Data Engineer will be responsible for designing, building, deploying, ad operating scalable data pipelines and modern data-platform capabilities for client analytic and business intelligence initiatives. The role is hands-on and combines technical leadership with delivery ownership across Microsoft Fabric, Azure data services, SQL, Python/PySpark, data modelling, governance, and deployment automation. The successful candidate will work closely with Business Analysts, Data Analysts, Power BI specialists, and client stakeholders to turn complex source systems and reporting requirements into dependable, governed, analytics-ready data products. The role will also contribute to architecture standards, data quality practices, documentation, technical estimation, and knowledge transfer. Key Responsibilities Include: Designing, building, and maintaining scalable batch and incremental data pipelines in Microsoft Fabric and related Azure services. Designing and implementing Lakehouse/Warehouse solutions using One Lake and medallion architecture principles (Bronze. Silver, Gold). Developing Fabric Data Factory pipelines, Dataflows Gen2, notebooks, and reusable ingestion frameworks. Building transformation logic using SQL, Python, PySpark, Spark, and Delta-based processing patterns. Integrating data from REST, APIs, SFTP, Sharepoint, OneDrive, Excel, Airtable, relational databases, and other enterprise systems. Designing source-to-target mappings, schemas, dimensional models, and data structures that support Power BI semantic models and self-service analytics. Implementing data-quality validation, reconciliation controls, lineage, monitoring, alerting, error handling, and other operational support patterns. Leading historical-data migration and validating migrated outputs against approved source and reporting baselines. Designing secure access patterns using Microsoft Entra ID, managed identities/service principals, RBAC, and appropriate platform controls. Implementing CI/CD, Git integration, deployment pipelines, environment promotion, and engineering standards across DEV, TEST, and PROD. Optimizing data processing, storage, query performance, and Fabric workloads for reliability, scalability, and cost efficiency. Collaboration with Power BI and analytics resources to ensure curated datasets and semantic models are accurate, reusable, and performant. Participating in technical discovery, architecture workshops, feasibility assessments, estimation, and delivery planning. Providing technical guidance, code review, engineering standards, and mentoring to other data and analytics team members. Preparing and maintaining architecture diagrams, data-flow documentation, pipeline specifications, runbooks, and as-built documentation. Supporting production deployment, cutover, troubleshooting stabilization, and knowledge transfer to client teams. Performing other data engineering and technical delivery duties as assigned. The Ideal Candidate will Possess: A Bachelor's Degree or Diploma in Computer Science, Data Engineering, Information Systems, Software Engineering or related field; equivalent practical experience will also be considered. Approximately 5+ years of progressive data engineering, analytics engineering, cloud data platform, or comparable experience. Hands-on experience with Microsoft Fabric or strong modern Azure data-platform experiences with the ability to transition quickly into Fabric. Strong SQL skills, including data transformation, optimization, complex joins, window functions, and performance troubleshooting. Demonstrate experience with Python and/or PySpark for production-grade data transformation and automation. Experience designing ETL/ELT pipelines, orchestration, incremental loads, retries, error handling, and data validation. Understanding Lakehouse, data warehouse, Delta Lake, dimensional modelling, star schemas, and medallion architecture. Experience with CI/CD, Git, Azure DevOps or GitHub, and controlled deployment practices for data-platform assets. Knowledge of data governance, data quality, lineage, security, and access-control principles. Experience collaborating with Power BI developers or analytics teams and supporting semantic models/reporting datasets. Strong troubleshooting, documentation, estimation, and technical communication skills. Ability to work with ambiguous requirements, multiple source systems, and changing priorities while maintaining delivery discipline. Strong interpersonal skills and the ability to communicate effectively with technical and non-technical stakeholders. Assets / Preferred Qualifications Microsoft Certified: Fabric Data Engineer Associate (DP-700) Microsoft Certified: Fabric Analytics Associate (DP-600) Experience with Microsoft Purview, Power BI, Direct Lake, Azure Key Vault, or related Microsoft Data Services Experience migrating legacy BI/data warehouse solutions to Microsoft Fabric or another modern cloud data platform. Experience with public-sector, tourism, finance, professional services, or other data-intensive operational environments. Compensation & Benefits Base Salary Range: $78,000 - $105,000 annually. This range represents Optillium's expected base-salary band for this position. Actual compensation will be determined based on the candidate's skill, experience, certifications, scope of responsibility, and overall fit for the role. Any additional bonus, benefits, or other compensation components applicable to the successful candidate will be communicated during the offer process. Conditions of Employment Must be legally eligible to work in Canada. Must be able to work from Calgary, Alberta and attend to client or team meetings in Alberta when reasonable required. Employment may be subject to satisfactory reference, background, and identity checks appropriate to the role and client environment. The successful candidate will be expected to follow Optillium and client confidentiality, security, privacy, and acceptable-use requirements. Equal Opportunity & Accommodation Optillium is committed to building an inclusive workplace. We welcome applications from qualified candidates of all backgrounds and make employment decisions based on role requirements, demonstrated skills, experience, and potential. If you require an accommodation during the recruitment process, please let us know when you apply. To Apply Please send your resume and a short Cover Letter to [email protected] with "Lead Data Engineer" in the Subject line. You may also apply through the job board where this position is posted. Only Candidates selected for the next stage will be contacted.

The Lead Data Engineer is responsible for designing, building, and operating scalable data pipelines and modern data-platform capabilities using Microsoft Fabric and Azure. The role involves turning complex source systems into governed, analytics-ready data products while providing technical leadership and mentoring to the team.

Candidates need a degree in Computer Science or a related field and approximately 5+ years of progressive experience in data engineering or cloud data platforms. Proficiency in SQL, Python/PySpark, and experience with Lakehouse/Warehouse architectures and CI/CD practices are required.

• Power BISouhaitée
• SQLSouhaitée
• CI/CDSouhaitée
• GitSouhaitée
• PythonSouhaitée

Relevées dans la description du poste. Confirmez les exigences importantes ci-dessus.

• Microsoft Fabric
• Azure Data Services
• SQL
• Python
• PySpark
• Data Modelling
• ETL/ELT Pipelines
• Medallion Architecture
• CI/CD
• Git
• Power BI
• Data Governance
• Azure DevOps
• Delta Lake
• Lakehouse
• Dimensional Modelling

• Data & Analytics
• Technology
• Software
• Engineering
• Consulting

Formation minimale

Baccalauréat

Expérience minimale

5+ ans

Postuler avant le

23 sept. 2026

Langue de l’offre

anglais

Heures de travail

40 heures par semaine

Exigences de lieu

Country, Calgary, Alberta, Canada

Niveau d’expérience

Mid-Senior level

Mode de candidature

La candidature directe est offerte