Data Engineer

Discovery – Insure

Senior Data Engineer  - Vitality Drive International

About Discovery

Discovery’s core purpose is to make people healthier and to enhance and protect their lives. We seek out and invest in exceptional individuals who understand and support our core purpose, and whose own values align with those of Discovery. Our fast-paced and dynamic environment enables smart, self-driven people to be their best. As global thought leaders, Discovery is passionate about innovating in order to not only achieve financial success, but to ignite positive and meaningful change within our society.

About Discovery Insure

Discovery Insure is committed to creating a nation of great drivers through our innovative Shared-value Insurance model. Discovery Insure is South Africa’s fastest growing short-term insurance company with comprehensive products that provide protection against current and emerging risks facing clients in the motor and home insurance sectors. Vitality Drive, an internationally-recognised and award-winning programme, is a key differentiator in the market that incentivises and rewards clients for driving well. The Vitality Drive programme has been scaled to local and international markets which now include Europe and the Middle East. The company employs over 1 000 people who are committed to putting our customers and financial advisers first by providing unique and innovative solutions and cover.

Key Purpose

The Senior Data Engineer is responsible for the design, development, and optimisation of scalable, enterprise-grade data platforms on Azure, with Azure Synapse Analytics at the core.This role enables the delivery of high-quality, trusted, and reusable data assets to support analytics, reporting, and AI use cases. The incumbent will play a key role in advancing a modern data lakehouse architecture, ensuring data is governed, performant, and aligned to business value outcomes.

Areas of responsibility may include but are not limited to:

Design

Translate business requirements into technical designs adhering to Discovery Insure processes, standards and guidelines, considering performance, security, scalability, and cost.

• Define and evolve medallion architecture patterns in Microsoft Fabric (Bronze/Silver/Gold), including ingestion, transformation, and curated consumption layers.
• Produce and maintain modern data models:

• OLTP-aligned models where needed
• Dimensional models (star/snowflake) for analytics
• Curated “Gold” layer data products (conformed dimensions, business-ready facts)

• Design ingestion and transformation strategies across Fabric components (Lakehouse/Warehouse, Data Pipelines, Dataflows Gen2, Notebooks/Spark).
• Establish data quality, lineage, and governance requirements (e.g., data contracts, validation rules, auditing, retention).

• Liaise with relevant parties where clarification of business requirements or resolution to technical issues is needed.
• Research and recommend effective solutions to technical issues that arise (e.g., performance bottlenecks, modelling trade-offs, orchestration patterns).

• Estimate development timelines based on business requirements and technical complexity; identify risks and dependencies early.

Development

• Construct robust, maintainable, scalable, optimally performing solution components in line with technical specifications, following prescribed process, standards and procedures.
• Develop and optimise:

• T-SQL for Fabric Warehouse/Azure SQL (as applicable)
• ELT/ETL transformations (SQL/Spark) aligned to the medallion pattern
• Reusable frameworks for ingestion, validation, and incremental processing

• Build and maintain orchestration using Fabric Data Pipelines (scheduling, dependencies, retries, parameterization).
• Implement patterns for incremental loads and change processing (e.g., CDC-like approaches, watermarking, SCD Type 1/2 where appropriate).
• Apply performance engineering:
• Partitioning strategies, incremental refresh patterns
• Query optimisation and model simplification
• Efficient file formats and table design for analytical workloads

Testing

• Conduct unit testing and fix any defects found, aiming for high-quality releases.
• Verify build stability and quality with development team before releasing to test team, aiming to release with zero defects.
• Build and maintain test data sets, validation rules, and reconciliation checks across Bronze/Silver/Gold.
• Assist the Business Analyst in ensuring the test pack includes relevant scenarios and test data.
• Consult and assist in reviewing risk / impact of defects found in testing and assist with fixing where necessary.

Knowledge and Skills:

• Modern Data Modelling & Analytics
• Modern data modelling (OLTP where needed, dimensional modelling, curated analytical models, semantic consistency)
• Strong understanding of medallion architecture concepts (Bronze/Silver/Gold, data quality gates, curated data products)
• Metric definition discipline (facts, dimensions, conformed entities, grain, business rules)
• Microsoft / Azure Data Platform
• Microsoft Fabric: Lakehouse/Warehouse concepts, Pipelines, Dataflows Gen2, Notebooks/Spark (as applicable)
• T-SQL expertise (complex querying, optimisation, robust patterns)
• Performance tuning and optimisation for analytical workloads (SQL/Spark and modelling considerations)
• Understanding of security patterns (RBAC, least privilege, workspace access, data access controls)
• Engineering Excellence
• Strong SDLC practices (requirements → design → build → test → release)
• Source control and peer review culture (Git; branching strategies; PR discipline)
• Data quality and validation frameworks (reconciliation, anomaly checks, rule-based validations)
• Documentation and maintainability: clear design artifacts, data dictionaries, lineage-ready documentation

Working knowledge of:

• Software development within SDLC.
• Unit Testing and quality practices.
• Data modelling and design of database structures.
• Practical understanding of cloud data solution concepts (security, orchestration, monitoring, cost-awareness).

Advantageous

• Power BI semantic modelling (star schemas, measures, performance considerations)
• Governance tooling awareness (cataloging, lineage, data classification)
• Exposure to CI/CD for data solutions (deployment pipelines, environment promotion concepts)

Education and Experience

Education:

• Matric (Essential)   
• BSc Computer Science or equivalent 3-year qualification - (advantageous)

Minimum Experience:             

• 5+ years consistent experience in Microsoft data development with demonstrable delivery of complex solutions (senior-level capability).
• Strong experience with SQL development and performance optimisation.
• Proven experience designing and implementing data models and data structures in support of operational and analytical requirements.
• Experience delivering data solutions aligned to layered architecture patterns (or demonstrable capability to adopt medallion practices quickly).

Advantageous

• • • Experience modernising or migrating data workloads from on-prem to cloud.
    • Experience building curated “Gold” layer models for enterprise reporting/analytics.

EMPLOYMENT EQUITY   
                             
The Company’s approved Employment Equity Plan and Targets will be considered as part of the recruitment process. As an Equal Opportunities employer, we actively encourage and welcome people with various disabilities to apply.