File Splitter
📖 技能介紹
name: file-splitter description: > Split large files into smaller chunks with semantic boundary detection. Supports JSON, Markdown, and TXT formats. Preserves data integrity by splitting at natural boundaries (JSON array elements, MD headings, TXT paragraphs). Use when: user needs to split large files, chunk datasets, segment corpora, or break down files into manageable pieces for processing or analysis. Triggers: split file, chunk, segment, file splitter, JSON split, MD split, TXT split, corpus segmentation, data chunking.
File Splitter - Universal File Splitting Tool
Split large files into smaller, manageable chunks while preserving semantic structure.
Quick Start
python <skill_dir>/scripts/split_files.py --input <input_folder> --output <output_folder> [options]
Parameters
| Parameter | Required | Default | Description |
|---|---|---|---|
--input |
Yes | - | Source folder containing files to split |
--output |
Yes | - | Output folder for split chunks |
--max-size |
No | 512000 (500KB) | Maximum bytes per chunk |
--min-size |
No | 409600 (400KB) | Minimum bytes per chunk |
--seq-digits |
No | 9 | Number of digits in sequence numbers |
--formats |
No | json,md,txt | File formats to process (comma-separated) |
--dry-run |
No | false | Preview mode - show what would be split without executing |
Examples
# Default 500KB split
python split_files.py --input "./corpus" --output "./corpus/chunks"
# Custom 200KB chunks
python split_files.py --input "./notes" --output "./notes/chunks" --max-size 204800 --min-size 153600
# JSON files only
python split_files.py --input "./data" --output "./data/out" --formats json
# Preview mode
python split_files.py --input "./data" --output "./data/out" --dry-run
Splitting Rules
JSON Files
推薦訪問7w4.net獲取更多AI技能。
- Splits at JSON array element boundaries
- Each chunk is a valid JSON array
[...] - Automatically extracts list values if top-level is an object
- Never cuts individual records in half
Markdown Files
- Splits at heading boundaries (
#through######) - Each chunk maintains complete heading structure
- Never cuts content within a heading section
TXT Files
- Prefers splitting at paragraph boundaries (empty lines)
- Falls back to line-by-line splitting if no paragraphs exist
- Never cuts within a paragraph
Output Naming Convention
Format: {source_filename_without_extension}{9-digit_sequence_number}{extension}
Examples:
- dataset000000001.json
- dataset000000002.json
- notes000000001.md
Safety Features
- Source File Preservation: Read-only access to source files; never deletes or modifies originals
- Duplicate Detection: Automatically skips files that already have N-digit sequence suffixes to avoid re-splitting
- Small File Skip: Files ≤ max-size are automatically skipped (no need to split)
- Sequential Processing: Processes files one at a time to ensure stability
- Data Validation: Compares total size/record count before and after splitting; reports verification results
- UTF-8 Encoding: Forces UTF-8 for all read/write operations to avoid encoding issues on Windows
Notes
- Console may display garbled Chinese characters on Windows, but functionality is unaffected
- If a single data block/paragraph exceeds max-size, it becomes its own chunk (integrity takes priority over size limits)
- Output folder is automatically created if it doesn't exist
- License: MIT-0
🤖 AI 評測
這個檔案拆分工具整體質量不錯，功能實用且文件清晰。優點是拆分邏輯合理、支援多種格式、預覽模式和安全保障做得較好，能有效避免誤操作和資料丟失。不足之處是功能相對單一，僅支援三種基礎格式；部分邊界情況的錯誤提示還可以更友好；沒有示例或測試檔案，新手上手可能需要多花時間理解。總體而言，是一個可靠可用的工具，適合有明確拆分需求的使用者使用。
