Principal Data Engineer – Databricks

Résumé du poste

Lead the transformation of legacy Cloudera platforms to a Databricks Lakehouse architecture, focusing on finance and risk data models. Act as a technical authority to design scalable data pipelines and mentor senior engineers while managing stakeholders.

Détails du poste

Inclusion without Exception Tata Consultancy Services (TCS) is an equal opportunity employer, and embraces diversity in race, nationality, ethnicity, gender, age, physical ability, neurodiversity, and sexual orientation, to create a workforce that reflects the societies we operate in. Our continued commitment to Culture and Diversity is reflected in our people stories across our workforce and implemented through equitable workplace policies and processes. TCS has been serving the Canadian marketplace for more than 35 years and today is the technology partner of choice for many of the country's leading organizations. As one of the largest IT services providers in Canada and globally, TCS is aiming to become the world's largest AI-led technology services company and is enabling its clients to transform themselves across the full AI stack, from infrastructure to intelligence. Rooted in the heritage and values of the Tata Group, TCS is focused on creating long-term value for its clients, investors, employees, and the communities it serves. With a highly skilled workforce supporting customers from coast to coast, TCS has consistently been recognized as a Top Employer by the Top Employers Institute, including a recent top ranking at the country level. TCS also sponsors 14 of the world's most prestigious marathons and endurance events, including the TCS Toronto Waterfront Marathon, reflecting its commitment to health, sustainability, and community empowerment. Principal Data Engineer – Databricks Key Requirements Years of experience in enterprise Data Warehouse and Data Lake platforms Hands-on experience with Databricks and Spark at scale Strong experience in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse Redesign ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake Refactor legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines Ensure data reconciliation, audit integrity, and consistency during migration Design and govern enterprise Data Warehouse and Data Lake/Lakehouse architectures Implement layered architecture including Raw/Landing, Curated/Conformed, and Semantic/Consumption layers Modernize traditional EDW platforms into scalable lakehouse architectures Strong experience in finance and risk data models including General Ledger, Sub-ledger, financial hierarchies, and risk exposure models (credit, liquidity, market risk) Enable reporting use cases including aggregation, drill-down, and drill-back capabilities Build and manage semantic/consumption layers for BI, reporting, and analytics Define business metrics, dimensions, hierarchies, and KPIs Experience with Databricks SQL, Delta tables, and dbt or similar frameworks Develop and optimize large-scale data pipelines using PySpark, Spark SQL, and Delta Lake Implement Medallion architecture (Bronze, Silver, Gold layers) Optimize workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations Implement data governance, data quality frameworks, reconciliation controls, and exception handling Establish data lineage and metadata management Ensure data security, access control, and compliance standards Experience with cloud platforms such as AWS or Azure Experience with CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps Familiarity with orchestration tools such as Airflow or Databricks Workflows Experience with dbt is a plus Act as a technical authority and lead architecture decisions Mentor and guide senior engineers and establish engineering standards Strong stakeholder management with finance, risk, analytics, and governance teams Ability to translate complex data structures into business-ready insights Nice to Have Experience in BFSI, Capital Markets, or regulatory reporting Exposure to SAP Finance, Oracle Financials, or S/4HANA Experience supporting AI/ML workloads Databricks or cloud certifications Impact Lead Cloudera to Databricks transformation initiatives Shape enterprise finance and risk data platforms Support regulatory, management, and analytical reporting systems Salary Range - CA$ 90,000 - CA$ 110,000 Per Year Tata Consultancy Services Canada Inc. is committed to meeting the accessibility needs of all individuals in accordance with the Accessibility for Ontarians with Disabilities Act (AODA) and the Ontario Human Rights Code (OHRC). Should you require accommodation during the recruitment and selection process, please inform Human Resources. Thank you for your interest in TCS. Candidates that meet the qualifications for this position will be contacted within a 2-week period. We invite you to continue to apply for other opportunities that match your profile.

Ce que vous ferez

Lead the transformation of legacy Cloudera platforms to a Databricks Lakehouse architecture, focusing on finance and risk data models. Act as a technical authority to design scalable data pipelines and mentor senior engineers while managing stakeholders.

Exigences

Requires extensive experience with Databricks, Spark, and cloud platforms (AWS/Azure) to implement Medallion architecture and data governance. Proficiency in PySpark, Spark SQL, and experience with BFSI data models is highly desired.

Compétences indiquées

• Microsoft Azure · Souhaitée
• CI/CD · Souhaitée
• Amazon Web Services · Souhaitée
• Stakeholder Management · Souhaitée

Autres compétences pertinentes

Relevées dans la description du poste. Confirmez les exigences importantes ci-dessus.

• Databricks
• Apache Spark
• PySpark
• Delta Lake
• Lakehouse Architecture
• ETL/ELT Pipelines
• Cloud Object Storage
• Data Governance
• Medallion Architecture
• AWS
• Azure
• CI/CD
• dbt
• Airflow
• Financial Data Modeling
• Stakeholder Management

Domaines d’emploi

• Data & Analytics
• Technology
• Consulting
• Software
• Finance & Accounting