Skip to content
v0.1.3latest release121 skills4 categories6 sub-categories247 tested scripts397 reference docsContact us
Vector Agentic Skills
Submit a skill

ml-vs-rules-decision-advisor

Decide whether a problem should be solved with hand-written rules, machine learning, a hybrid of both, or not yet: score twelve factors (labelled data volume, label availability, signal complexity, change rate, rule maintenance pain, scale, error cost, explainability and regulation, latency, team ML maturity, personalisation, feedback speed) with hard gates, learn a shallow readable rule set from a CSV with pure-Python CART and compare its holdout AUC with the ML model, emit the rule as a deployable SQL CASE, and design hybrids (rules as filters, guardrails, model score inside rules, ML-suggested rules). Use when someone asks "do we even need ML?", "rules or a model?", wants to replace a sprawling rule engine, justify an ML project against existing business rules, or plan a migration from rules to ML.

AI & Machine Learning / Classical Machine Learningv1.0.0Updated OCT 25 20264 references2 scripts

Rules are cheaper, faster, auditable and need no labels; ML wins when the signal is spread over many weak, interacting features, patterns shift faster than people can edit rules, or the rule base has become unmaintainable. Most production systems end up as hybrids: hard rules for policy and catastrophic cases, a model for ranking the rest. scripts/rules_vs_ml.py scores twelve factors with hard gates and recommends RULES, HYBRID, ML or NOT YET; scripts/rule_learner.py learns the best small rule set from your data and shows how close it gets to a model, with a deployable SQL rule.

Trigger Conditions

Activate this skill when the user:

  • Asks "do we need ML for this?", "rules or a model?", "is a heuristic good enough?"
  • Maintains a large rule engine (fraud, routing, pricing, eligibility, alerting) and wants to know whether ML should replace or augment it
  • Must justify an ML project against the existing business rules, or a stakeholder insists rules are enough
  • Wants interpretable rules extracted from data (to deploy, to audit a model, or to seed a rule base)
  • Is planning a migration from rules to ML without breaking policy constraints

Framing comes first (ml-problem-framing-advisor); data sufficiency in ml-feasibility-assessor; the baseline table that includes the rule in ml-baseline-model-builder.

Workflow

text
1. DECISION   -> what is decided, how often, by whom today; current rules and their measured performance
2. SCORE      -> python scripts/rules_vs_ml.py --template > a.json; edit; python scripts/rules_vs_ml.py a.json
3. EVIDENCE   -> python scripts/rule_learner.py data.csv --target y --time-col ts --ml-auc <model AUC>
4. COMPARE    -> current rules vs learned rules vs ML on the same holdout and metric (precision@K, cost)
5. DECIDE     -> RULES / HYBRID / ML / NOT YET, with the gating reasons
6. DESIGN     -> for HYBRID: which rules stay hard, where the model sits, how overrides work
7. MIGRATE    -> shadow mode, champion/challenger, rollback; keep logging for labels

Decision Factors

FactorFavours rulesFavours ML
Labelled datanone, or a few dozen positivesthousands of labelled outcomes
Signalone or two obvious conditions ("overdue > 30 days")many weak, interacting signals
Change ratestable policyadversarial or trending patterns (fraud, spam, demand)
Rule basea handful of clear ruleshundreds of overlapping rules nobody understands
Scaledozens of decisions per daymillions; small accuracy gains add up
Error costone error is catastrophic (safety, legal)errors are cheap and reversible
Explainabilitymust cite the exact rule (regulation, adverse action)reasons are nice-to-have
Latency / inframicroseconds, no serving stackmodel serving exists
Teamnobody can retrain or monitorMLOps in place
Personalisationone policy for everyoneper-user decisions
Feedback speedlabels after monthslabels in minutes

Hard gates in rules_vs_ml.py: no data -> NOT YET (deploy rules and log outcomes); no labels and no path to labels -> RULES (supervised ML impossible); regulated decision requiring exact reasons -> RULES or interpretable scorecards; simple signal with a small rule set -> RULES.

Get Evidence: How Far Do Rules Go?

Opinions about "rules are enough" settle quickly with numbers on the same holdout:

bash
python scripts/rule_learner.py data.csv --target churned --time-col signup_date --max-depth 3 --min-leaf 20 \
    --drop customer_id zip_code refund_issued cancellation_reason --ml-auc 0.87
text
  FLAG IF support_calls >= 2 AND tenure_months < 16 (or missing)
         THEN rate 0.682  (n=22, lift 3.03x)
  ...
Holdout (120 rows): ROC-AUC 0.655  PR-AUC 0.222 ...
ML model AUC 0.870 vs rules 0.655 (gap +0.215): ML clearly ahead (> 0.08 AUC): the signal is beyond a few rules
Gap (ML AUC − learned-rule AUC)Reading
< 0.03rules capture the signal: prefer rules unless ML adds other value (automation of upkeep, personalisation)
0.03 - 0.08moderate lift: hybrid, or ML if the lift at the operating point pays its running cost
> 0.08signal lives in interactions and many weak features: ML earns its cost

A leaf with a positive rate of 1.000 on a single column (IF cancellation_reason != '') is a leak, not a rule: drop the column and refit. Always compare at the operating point too (precision@K, cost), not only AUC; rules produce few distinct scores, so their AUC understates what a single well-placed rule threshold achieves at one K.

Hybrid Patterns

PatternShapeUse when
Rules as hard guardrailsif blocklisted or amount > limit: decline before or after the modellegal/policy limits, catastrophic cases
Rules as eligibility filtermodel scores only rows rules allowout-of-scope populations, cold start
Model inside a ruleif score > 0.8 and amount > 500: reviewhumans want rules but need better signal
Rules for cold start, ML when data existsrule for new users, model after 30 dayssparse early data
ML proposes, humans approvemine rules from data/SHAP, analysts promote themregulated or adversarial domains
Rules override with logginganalysts can force decisions; overrides become labelstrust building, feedback
Champion/challengerrules = champion, model = challenger on a traffic slicemigration with a safety net

Code and diagrams: hybrid-patterns.md.

When Rules Are the Right Answer

  • The policy is the rule (eligibility thresholds, compliance limits, contractual terms).
  • No labels exist and the cost of errors is high: encode expert knowledge, log outcomes, revisit later.
  • Volume is small: a person or a rule handles it at lower total cost than an ML lifecycle.
  • A single interpretable threshold captures most of the value (check with rule_learner.py --max-depth 1).

Treat rules as software: owner, tests with examples, version control, a review date, metrics on hit rate, precision and conflicts. Details: rules-engineering.md.

When ML Is the Right Answer

  • Rule maintenance is the bottleneck (hundreds of rules, frequent conflicts, nobody can explain the order).
  • Fraud/spam/abuse patterns shift weekly and rules lag the adversary.
  • Ranking or personalisation over many items or users.
  • Large volume where 1-2 points of precision are worth real money.

Keep guardrail rules outside the model even then; a model should never be the only thing standing between a request and a legally prohibited outcome.

Migration From Rules to ML

  1. Measure the rules as if they were a model (precision, recall, volume) on a labelled period.
  2. Learn from rule-generated labels carefully: if labels only exist where rules fired, the model inherits the rules' blind spots (selection bias); add random exploration or review samples.
  3. Shadow mode: run the model alongside rules, compare decisions, review disagreements.
  4. Champion/challenger: give the model a small traffic share with guardrail rules still active.
  5. Retire rules gradually, keeping a rule-only fallback for model outages.

Checklist and pitfalls: migration-rules-to-ml.md.

Output Format

When advising, answer with:

  1. Recommendation - RULES, HYBRID, ML or NOT YET, with the gating factor or the strongest factors.
  2. Evidence - current rules' metrics, learned-rule metrics (rule_learner.py), the ML model's metrics, on the same holdout and operating point.
  3. Design - for RULES: the rule set (SQL/pseudocode) with owners and review dates; for HYBRID: which rules stay hard, where the model sits, override handling; for ML: the guardrail rules that remain.
  4. Costs - build, run and maintenance for each option, and the lift that would justify ML.
  5. Migration or revisit plan - shadow mode, champion/challenger, what data to log, when to reassess.

Scripts

bash
python scripts/rules_vs_ml.py --template > answers.json   # example answers to edit
python scripts/rules_vs_ml.py answers.json
python scripts/rules_vs_ml.py answers.json --json

Twelve factors scored 1 (favours rules) to 5 (favours ML), weighted and centred into a score from -1 to +1; hard gates for no data, no labels, regulated decisions needing exact reasons, and simple signal with few rules. Prints the recommendation, the gate or strongest factors with reasons, and next steps for that option. Exit codes: 0 printed, 2 usage error (missing or out-of-range factors).

bash
python scripts/rule_learner.py data.csv --target churned --drop customer_id --time-col signup_date
python scripts/rule_learner.py data.csv --target is_fraud --max-depth 2 --min-leaf 30 --ml-auc 0.91
python scripts/rule_learner.py data.csv --target churned --flag-rate 0.4 --json

Pure-Python CART (Gini) on a binary target: numeric thresholds on up to 32 quantile candidates, categorical equality splits on the top 20 levels, depth and minimum-leaf limits, missing numeric values routed to the "below" branch. Prints IF/THEN rules simplified into one interval per column, each leaf's positive rate, size and lift, holdout ROC-AUC, PR-AUC, precision and recall when flagging leaves above --flag-rate (default twice the base rate), a NULL-safe SQL CASE for the flagged leaves, and a verdict against --ml-auc. Exit codes: 0 fitted, 2 usage error or non-binary target.

Additional Resources

FileContent
decision-framework.mdThe factors in depth, cost comparison of rules vs ML over a year, worked decisions (invoice reminders, fraud, ticket routing, eligibility, alerting)
hybrid-patterns.mdGuardrails, filters, score-in-rule, cold-start rules, ML-suggested rules, overrides, champion/challenger, with code
rules-engineering.mdRules as code: representation, testing, versioning, conflict detection, metrics, decision tables, rule extraction from models
migration-rules-to-ml.mdStep-by-step migration, selection bias from rule-generated labels, shadow mode, rollback, retiring rules
More in Classical Machine Learning

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.