dskit · scaffold · harness

A production-grade ML scaffold
with the rules in code.

A copier template that generates complete ML projects, from environment and layout to API, monitoring, RAG and an AI harness. It doesn't just start a project: it ships with the quality pipeline and the governance already wired in.

github.com/cacelass/dskit ← back to projects
01
problem What it solves
Most ML failures are not model failures. They are reproducibility and consistency failures. Repetitive setup and inconsistent structure are the primary cause of broken projects. dskit eliminates environment drift and enforces a standard project layout from day one, so every generated project starts reproducible, testable and deployment-ready.
Type
Copier template · 4 ML profiles · 30 Python agents
Stack
uv · Copier · Sphinx · PyTorch · FastAPI · MLflow · Optuna
Version
1.15.1 · CI green · Apache-2.0
02
generation One command, a full project
copier copy --trust gh:cacelass/dskit mi_proyecto generates a project with the exact ML flavour you need. Four profiles set the defaults: minimo, estandar, completo and manual (one question at a time).
→ 4 ML types: supervisado, no_supervisado, redes_neuronales, hibrido; task clasificacion / regresion.
→ 11 model selectors: RandomForest, ExtraTrees, GradientBoosting, XGBoost, LightGBM, CatBoost, LogisticRegression, KNN, DecisionTree, SVM, or all in parallel.
→ 6 NN architectures: MLP, CNN1D, LSTM, GRU, Transformer, ResNet; 5 optimisers and 5 loss functions.
→ 6 clustering algorithms: KMeans, Agglomerative, DBSCAN, GaussianMixture, Spectral, Birch.
→ Python 3.10–3.13 and license (MIT, BSD-3, Apache-2.0 or none) chosen at generation time.
→ Live template: copier update brings improvements from later versions without restarting; answers are kept in .copier-answers.yml.
03
modules Optional modules, opt-in
→ API: FastAPI with /health, /info, /predict; Pydantic V2 schemas (make serve).
→ MLflow: experiment tracking, artifacts and Model Registry (make mlflow).
→ Optuna HPO: per-model objective functions; best params saved to artifacts/ (make tune).
→ Monitoring: KS/chi² drift + performance vs baseline with Evidently (make monitor).
→ RAG: local hybrid retrieval (ChromaDB + BM25), offline, no API key: indexes code, docs, harness memory and a knowledge corpus (make index-rag).
→ Spec-driven: Gherkin contract with a human gate before coding, mutation testing and CRAP metric (make mutation).
→ Conformal prediction: coverage-guaranteed, distribution-free sets and intervals.
→ Calibration: temperature scaling for neural networks.
→ SHAP: feature importance automatically computed on the best model.
→ DuckDB: SQL directly over CSV/Parquet/JSON (make query).
→ Docker + Gradio chat: containerised model interface (make docker-run).
→ MCP servers: filesystem, git, fetch and more for the coding assistant.
04
quality Quality pipeline
→ make check: lint + typecheck + test + harness in one battery.
→ make security: bandit + pip-audit; make audit: radon cc + team audit.
→ Tests for every layer: train, predict, features, dataset, API, tuning, monitoring, with a patch_paths fixture that isolates the filesystem.
→ TorchMetrics: consistent metrics between training and evaluation (Accuracy/F1/Precision/Recall, MAE/RMSE/R²).
→ Early stopping and configurable validation split for neural networks; TensorBoard wired in.
→ Profiling: cProfile → snakeviz (make profile); lockfile regeneration (make lock).
→ CI runs the harness gate and fails if agent prompts drift from the code.
→ Live PRD: docs/prd.md regenerates from the backlog every time a feature closes.
05
harness The AI harness

An AI model generates code much faster than a human reviews it. The harness puts the reins on: it lives inside the generated repo, travels with the project and is shared by the whole team. It is provider-agnostic: any coding agent that reads AGENTS.md and can run shell commands can use it.

→ AGENTS.md: the entry point. The first thing any agent reads. CLAUDE.md only points to it, no duplication.
→ init.sh: the gate: environment, harness files, backlog and test suite. Exit != 0 means stop.
→ harness/featureslist.json: a backlog with verifiable acceptance criteria.
→ harness/progress/: memory outside the context window: current task and append-only history.
→ Two agent layers: reasoning agents (lider, explorer, implementer, reviewer) and 30 deterministic Python agents (git, test, docker, data, ml, docs…).
→ One resource, one owner: contracts in agents/contracts.py declare what each agent may and may not do, validated by tests.
→ Prompt discipline: agent prompts are derived from code; make prompts-check fails CI if they desynchronise.
→ Multi-assistant: make assistants-sync mirrors the agents to .claude/agents/; a SessionEnd hook runs the gate when the session closes.
The rule No feature is marked done unless ./init.sh passes in green with real command evidence. That is not a prompt instruction; harness finish applies it in Python, so it cannot be talked around.
06
layout Generated structure
Every project ships with a production layout: package + data/{raw,interim,processed,external}, models/artifacts/, reports/, tests/, an optional Obsidian vault/ for project knowledge, and .copier-answers.yml so the template can evolve without restarting.
Ecosystem dskit generated the ClimaSafe project and shares its harness philosophy with MeshHarmes.