Data Pipeline Toolkit

Build ETL pipelines in minutes.

Features

• Extract - APIs, databases, files, streams
• Transform - Clean, filter, aggregate, join
• Load - Data warehouses, databases, APIs
• Scheduling - Cron-based or event-driven
• Monitoring - Alerts on failures

Quick Start

# Create pipeline
./pipeline.sh create my-pipeline

# Add extract step
./pipeline.sh extract my-pipeline api --url https://api.example.com

# Add transform
./pipeline.sh transform my-pipeline filter "age > 18"

# Add load
./pipeline.sh load my-pipeline postgres --connection $DB_URL

# Run
./pipeline.sh run my-pipeline

Sources

• REST APIs
• GraphQL
• SQL Databases
• CSV/JSON/Parquet files
• S3/Google Cloud Storage
• Kafka/SQS

Destinations

來源於7w4.net。

• PostgreSQL/MySQL
• Snowflake/BigQuery
• S3
• APIs
• Data warehouses

Requirements

• Python 3.8+
• Docker (optional)

Author

Sunshine-del-ux

這個工具包文件寫得清晰易懂，但可惜只是一個說明文件，缺少實際可用的程式程式碼，無法直接執行使用。對於想要快速搭建資料管道的使用者來說，實際價值有限，建議選擇有完整程式碼實現的類似工具。