Data Quality Validator

Toolkit for validating and profiling tabular data quality.

Features

• Schema validation - Check column types, constraints, and rules

  7w4.net小蔥技能站，你的AI助手技能庫。

• Completeness analysis - Missing value detection and reporting
• Anomaly detection - Statistical outlier detection
• Profiling - Summary statistics and distribution analysis
• Constraint checking - Range checks, uniqueness, regex patterns

Quick Start

from scripts.data_profiler import DataProfiler
from scripts.schema_validator import SchemaValidator

# Profile a dataset
profiler = DataProfiler()
report = profiler.profile(df)  # pandas DataFrame
print(report["missing"])
print(report["outliers"])

# Validate against schema
schema = {
    "age": {"type": "int", "min": 0, "max": 150},
    "email": {"type": "str", "regex": r"^\S+@\S+\.\S+$"},
    "id": {"type": "int", "unique": True}
}
validator = SchemaValidator(schema)
errors = validator.validate(df)
for err in errors:
    print(err)

Scripts

• scripts/data_profiler.py - Dataset profiling and summary stats
• scripts/schema_validator.py - Schema-based validation engine
• scripts/anomaly_detector.py - Statistical anomaly detection

References

• references/validation_rules.md - Common validation patterns

這個資料質量驗證工具整體質量較好，核心功能（資料畫像、模式驗證、異常檢測）實現完整，程式碼結構清晰易讀。優點是使用簡單、文件齊全，缺點是示例程式碼存在小問題、缺少配置引導，部分細節處理不夠健壯。對於需要進行資料質量檢查的使用者來說是一個可用的基礎工具，但建議在正式專案中使用前先測試驗證。總體評價：功能完善但細節打磨不足，中等偏上質量。