AutoML自動機器學習

Overview

借鑑Karpathy autoresearch理念的表格資料AutoML技能。AI代理自主進行ML實驗，通過迭代最佳化模型引數和結構，在CPU環境下自動尋找最佳配置。支援7種傳統ML模型、早停機制、交叉驗證和整合策略。

Workflow

使用者請求資料建模
    ↓
1. 收集引數（資料路徑/目標列/任務型別/指標/預算）
    ↓
2. 分析資料（讀取prepare.py元特徵，確定推薦模型和策略）
    ↓
3. 初始化工作區（複製指令碼/資料，git init，建立baseline）
    ↓
4. 實驗迴圈（修改train.py → 執行 → 評估 → keep/discard → 迭代）
    ↓
5. 輸出最佳模型和配置報告

Step 1: Collect parameters

Ask user for:

• data_path: CSV file path (required)
• target_col: target column name (required)
• task_type: classification / regression / multiclass (auto-infer if not specified)
• metric: evaluation metric (default by task type: accuracy / rmse / f1_macro)
• time_budget: seconds per experiment (default 120)
• max_experiments: max rounds (default 50)
• models: limit model list, e.g. "lgb,xgb" (default: auto-select based on data scale)

Step 2: Analyze data

Run data check:

python scripts/prepare.py --check-data

This outputs: sample count, feature count, target distribution, missing values, recommended models. For tuning strategy, read references/optimization_strategies.md when planning experiments.

Step 3: Initialize workspace

python scripts/auto_research.py --data <path> --target <col> --task <type> --setup-only

This creates:

./automl_exp/
├── prepare.py       # Fixed: data loading, preprocessing, metrics
├── train.py         # Modifiable: model, params, feature engineering
├── data.csv         # User data
├── results.tsv      # Experiment log
└── .automl_cache/   # Cache + config

Then run baseline to establish reference metric.

Step 4: Experiment loop (core)

For each round:

1. Analyze results.tsv — current best, recent trend, what worked
2. Design experiment — follow priority: hyperparameter tuning > model switch > feature engineering > ensemble
3. Modify train.py — update MODEL_NAME, MODEL_PARAMS, FEATURE_ENGINEERING
4. Run — python train.py > run.log 2>&1 with timeout
5. Evaluate — parse output for val_metric, memory, train_time
6. Decide — keep (improvement) or discard (no gain), git commit or revert
7. Loop — back to step 1

For experiment design rules and tuning strategies, read references/program.md and references/optimization_strategies.md.

Key tuning priority (GBT models):

• learning_rate + n_estimators (paired, coarse-to-fine)
• num_leaves / max_depth (complexity control)
• subsample + colsample_bytree (regularization via sampling)
• reg_alpha + reg_lambda (L1/L2)

Do NOT ask user "should I continue?" during loop. Run autonomously until max_experiments or user stops.

Step 5: Output results

Report:

• Total experiments / kept / discarded / crashed

  小蔥技能7w4.net有完整的技能分類。

• Best metric and improvement over baseline
• Best model configuration
• Feature importance (if available)
• Overfitting diagnostic (train-val gap)
• Next step recommendations

Model selection by data scale

Supported metrics

Classification: accuracy, f1, f1_macro, f1_weighted, precision, recall, roc_auc Regression: rmse, mae, r2, mape

Dependencies

scikit-learn, pandas, numpy, xgboost, lightgbm, catboost, psutil

Install if missing:

pip install scikit-learn pandas numpy xgboost lightgbm catboost psutil

Quick start example

請使用automl技能最佳化我的客戶流失預測模型：
- 資料：./customer_churn.csv
- 目標列：churn
- 任務：classification
- 指標：roc_auc
- 時間預算：180秒/輪
- 最多30輪

Resources

• scripts/prepare.py: Fixed data pipeline (do NOT modify during experiments)
• scripts/train.py: Training script (modify MODEL_NAME, MODEL_PARAMS, feature engineering)
• scripts/auto_research.py: Automated experiment runner with baseline and loop
• references/optimization_strategies.md: Hyperparameter tuning priority, model selection, ensemble strategies
• references/program.md: Experiment behavior rules, keep/discard criteria, phase strategies

這個AutoML技能質量中等偏上，勝在設計規範、文件齊全、模型支援豐富。但實際自動化能力有限，更像一份詳細的操作指南而非真正的自動化工具。優點是幫你把調參流程標準化了，有現成的實驗管理機制；不足是很多環節仍需你手動操作，自動化程度不夠高，對新手不太友好。