Data Quality Engineer (Data QE)

Key Responsibilities

Data Validation & Testing

• Perform end-to-end data validation across the data lifecycle, including ingestion, transformation, storage, and reporting.
• Validate data movement from source systems including:
• Customer Data Platforms (CDP)
• Google Analytics 4 (GA4)
• Google BigQuery
• Azure Blob Storage
• PostgreSQL
• MSSQL
• Snowflake
• Perform comprehensive Source-to-Target (S2T) data validation between source systems and Snowflake.
• Validate data accuracy, completeness, consistency, uniqueness, and timeliness across datasets.
• Execute data reconciliation and data profiling activities.
• Conduct source-to-target validation across databases, data lakes, and data warehouses.
• Validate schema changes, constraints, indexes, stored procedures, functions, triggers, and views.

Medallion Architecture Validation

• Validate data across Bronze, Silver, and Gold layers within the Medallion Architecture.
• Verify data transformations, cleansing, aggregations, and business rules between layers.
• Ensure data quality controls are enforced throughout the Medallion framework.

ETL/ELT Testing

• Test and validate ETL/ELT pipelines and data transformations.
• Validate business mappings, transformation logic, and data lineage.
• Verify incremental and full-load processes.
• Perform regression, integration, functional, and end-to-end testing of data pipelines.
• Identify and troubleshoot data discrepancies across source and target systems.

Snowflake Data Validation

• Validate data ingestion, transformations, and storage within Snowflake.
• Perform large-scale data validation using SQL and automation scripts.
• Validate Snowflake views, tables, materialized views, stored procedures, and data sharing mechanisms.

Power BI Testing

• Validate datasets, data models, measures, KPIs, and dashboards.
• Ensure Power BI reports accurately reflect Snowflake data.
• Verify report-level calculations, filters, row-level security, and business metrics.
• Perform end-to-end testing from source systems through Snowflake into Power BI.

Data Governance & Data Management

• Validate implementation of Data Governance policies and standards.
• Ensure compliance with enterprise data quality and governance requirements.
• Validate and maintain:
• Data Lineage
• Data Dictionary
• Metadata Management
• Business Glossary
• Data Catalogs
• Collaborate with Data Governance and Business teams to improve data quality processes.

Data Quality Frameworks

• Implement and execute data quality checks and monitoring solutions.
• Define and monitor Data Quality KPIs and metrics.
• Develop and automate validation rules for:
• Completeness
• Accuracy
• Consistency
• Uniqueness
• Validity
• Timeliness
• Support root cause analysis and remediation of data quality issues.

Test Automation

• Develop automated data validation frameworks and reusable test suites.
• Automate source-to-target reconciliation and regression testing.
• Integrate automated data quality tests into CI/CD pipelines.
• Build automated validation using SQL, Python, PySpark, and Data Quality tools.

Required Technical Skills

Databases & Data Warehousing

• PostgreSQL
• Microsoft SQL Server (MSSQL)
• Snowflake
• BigQuery
• Data Warehouse Testing
• Data Modeling Concepts
• Star Schema & Snowflake Schema
• Slowly Changing Dimensions (SCD)
• Fact & Dimension Validation
• CDC Validation
• Data Migration Testing

SQL Expertise

• Advanced SQL Query Writing
• Complex Joins
• Window Functions
• Aggregations
• Query Optimization
• Data Reconciliation Queries
• Data Profiling and Data Analysis

Modern Data Platform Validation

• Medallion Architecture Validation (Bronze, Silver, Gold)
• Source-to-Target Testing
• ETL/ELT Testing
• Data Lake & Data Warehouse Validation
• Power BI Report & Dashboard Validation