About the Role

Duties \& Responsibilities:

Design and build a scalable, cloud-native data platform aligned with microservices.
Develop real-time and batch data pipelines to power data-driven products.
Implement SQL, NoSQL, and hybrid storage strategies for diverse business needs.
Enable self-serve data access with secure, well-documented data APIs.
Collaborate with Product \& Business teams to define and optimize data products.
Ensure data quality, lineage, and governance in all data pipelines and products.
Build event-driven architectures using Kafka, Azure Event Hub, or Service Bus.
Develop scalable ETL/ELT processes for ingestion, transformation, and distribution.
Optimize query performance, indexing, and caching for data-intensive apps.
Enforce data privacy, security, and access controls aligned with compliance standards.
Implement observability and monitoring for data infrastructure and pipelines.
Work with DevSecOps teams to integrate security into CI/CD workflows

Qualifications: Must Haves

5 years of experience in data engineering, with exposure to Data Mesh and Data as a Product preferred.
Expertise in modern data storage and processing, including SQL, NoSQL (Cosmos DB, PostgreSQL), Data Lakes (Azure Data Lake, Delta Lake, Apache Iceberg).
Proficiency in ETL frameworks (e.g. Apache Kafka, Airflow, Flink, Spark, Azure Data Factory, Databricks).
Experience with Event-driven architectures using Queues, Pub/Sub services (e.g. Azure Service Bus, Azure Event Grid, Amazon Event Bridge) and Containerized Environments (Container Apps, AWS ECS).
Experience with Apache and/or Azure data platforms or similar, e.g. Fabric, Databricks, Snowflake, and Apache Hudi.
Strong API development skills using GraphQL, REST, and/or gRPC for enabling data as a product.
Proficiency in Go, Java, and/or Python.
Deep understanding of data governance, security, lineage, and compliance using Microsoft Purview, OpenLineage, Apache Ranger, or Azure Key Vault.
Experience with Infrastructure as Code (IaC) using Bicep, Terraform, or CloudFormation for managing cloud-based data solutions.
Strong problem-solving and collaboration skills, working across data, engineering, and business teams.

Nice to Have

Knowledge of ML, AI, and LLMs, including data engineering for model training and inference with Azure Machine Learning, Databricks ML, and MLflow.
Hands-on experience in Notebooks (e.g. Jupyter, Databricks, Azure Synapse) for data workflows.
Experience in real-time data streaming architectures.
Exposure to data monetization strategies and analytics frameworks.
Familiarity with federated data governance and self-serve data platforms.
Building a diverse and inclusive team, supporting career growth and development.
Competitive base salaries.
Share Appreciation Rights program for salaried employees.
Paid vacation days and sick days.
An employee charitable donation program.
Hotel and travel discounts.
Comprehensive benefits package including extended health, vision, dental, Health Spending Account, TeleDoc, Employee Assistance Program, Life, Long-term Disability, AD\&D, and Critical Illness Insurance.
Calgary's Head Office: Located in beautiful Eau Claire in downtown Calgary, within a 5-minute walk to Prince's Island Park
Company-hosted events, and a game room.
Ability to join our Social Club and join colleagues for fun events such as golf, bowling, curling, stampede events and more.
Free access to the Aspen Properties Fitness Centers.

Black Diamond Group

Black Diamond Group

Latest Jobs