Feature selection is worth doing for four reasons: fewer features to compute and serve, less overfitting
when rows are scarce, easier explanations, and catching junk (IDs, leaks, duplicates). It is not a free
accuracy boost: modern regularised models and gradient boosting tolerate many weak features, and selection
done outside cross-validation silently inflates scores. scripts/filter_select.py runs the model-free
first pass (missing, constant, ID-like, univariate signal, redundancy groups, probable leaks);
scripts/importance_stability.py checks whether model-based selections agree across folds and seeds.
Trigger Conditions
Activate this skill when the user:
- Asks which features to keep or drop, or wants to reduce hundreds of columns to a manageable set
- Has more features than rows (genomics, text-derived, wide event tables) or few positives per feature
- Asks whether feature importance, SHAP ranking or RFE output can be trusted
- Sees selected features change between folds, seeds or retrains
- Wants to cut serving cost or latency by removing expensive features
- Suspects their CV score is optimistic because selection used the full data
Leak hunting goes deeper in target-leakage-detector; explaining the final model in feature-importance-reporter and shap-value-explainer; dimensionality reduction without a target in pca-dimensionality-reducer.
Workflow
1. JUNK -> python scripts/filter_select.py data.csv --target y --drop id time
(missing, constant, ID-like, probable leaks, redundant groups)
2. BASELINE -> CV score with all remaining features (same folds for everything after)
3. EMBEDDED -> L1 / elastic net path, or GBM with permutation importance on validation folds
4. WRAPPER -> RFECV or SequentialFeatureSelector only if the feature count must shrink a lot
5. INSIDE CV -> every selection step lives in the Pipeline; nested CV when selection is tuned
6. STABILITY -> python scripts/importance_stability.py importances.csv --top-k 20
7. DECIDE -> keep the smallest set within noise of the best score; document dropped featuresMethod Families
| Family | Methods | Pros | Cons |
|---|---|---|---|
| Filter | variance threshold, missingness, univariate AUC/F-test/chi²/mutual information, correlation redundancy | fast, model-free, good junk removal | ignores interactions; univariate ≠ useful in a model |
| Embedded | L1 (lasso, L1-logistic), elastic net, tree/GBM importance, SelectFromModel | uses the model; one fit | L1 picks one of a correlated group arbitrarily; impurity importance is biased |
| Wrapper | RFE / RFECV, sequential forward/backward, Boruta | optimises the actual metric | expensive; overfits the selection on small data |
| Permutation importance | drop in held-out score when a feature is shuffled | model-agnostic, honest | correlated features share (and hide) importance |
| Stability selection | selection frequency over subsamples/bootstraps | controls false selections | more compute; needs a base selector |
Code for each with scikit-learn: selection-methods.md.
The Rule: Select Inside Cross-Validation
Selecting features on the full dataset and then cross-validating the reduced model leaks the test folds' labels into the selection. With many features and few rows this can turn pure noise into an "AUC 0.8" model.
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectFromModel, SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold
pipe = Pipeline([
("prep", prep), # imputers, encoders, scalers
("select", SelectFromModel(LogisticRegression(penalty="l1", solver="liblinear", C=0.1))),
("model", LogisticRegression(max_iter=2000)),
])
cv = StratifiedKFold(5, shuffle=True, random_state=0)
print(cross_val_score(pipe, X, y, cv=cv, scoring="average_precision").mean())If the number of features or the selector's strength is tuned, wrap it in GridSearchCV and evaluate the
whole search with an outer CV (nested CV). Demonstration of the bias and nested-CV code:
selection-bias-and-nested-cv.md.
Correlated Features
Correlated features split importance and make selections unstable: with total_charges ≈ f(tenure, price),
L1 or a tree keeps one of them in some folds and the other in other folds.
| Goal | Approach |
|---|---|
| Smaller, stable set | cluster features by |
| Honest importance | permutation importance per cluster (shuffle the group together), or grouped SHAP |
| Prediction only | keep them; GBMs and ridge handle redundancy fine |
| Interpretation | keep the most interpretable member, or a composite (PCA of the group, a ratio) |
importance_stability.py makes this visible: correlated pairs show up as "unstable" features selected in
40-60% of runs while the overall ranking agreement stays high.
Choosing How Many Features
- Plot CV score vs number of features (RFECV
cv_results_, or a forward-selection curve). - Find the best mean score and its standard error across folds.
- Choose the smallest set whose score is within one standard error of the best (one-SE rule).
- Re-check on a time-based holdout; a set chosen on random folds can rely on features that drift.
Junk and Leak Signals
| Signal | Interpretation | Action |
|---|---|---|
| Single-feature AUC > 0.9 | leak until proven otherwise | target-leakage-detector |
| Missing for > 95% of rows | almost no information | drop (or keep a "has value" flag if missingness is informative) |
| One value in > 99.5% of rows | near-constant | drop |
| Near-unique text | ID | drop (aggregate by it instead) |
| Random noise column ranks above real features | importance method is biased (impurity) or the model overfits | use permutation importance; add a noise probe |
| Feature only useful on random folds, not on time holdout | time leakage or drift | drop or rebuild as-of |
Noise probe: add 1-3 columns of random numbers before computing importances; any real feature ranking below the probes is not distinguishable from noise (the idea behind Boruta's shadow features).
Output Format
When advising on feature selection, answer with:
- Junk removed - the
filter_select.pytable: dropped columns with reasons, probable leaks, redundancy groups and the member kept. - Method - filter/embedded/wrapper/permutation chosen for this data size and model, inside a Pipeline.
- Curve - CV score vs number of features, the one-SE choice.
- Stability - selection frequency across folds/seeds (
importance_stability.py), unstable groups. - Final set - kept features, dropped features with the score cost of dropping them, serving cost saved.
Scripts
python scripts/filter_select.py data.csv --target churned --drop customer_id signup_date
python scripts/filter_select.py data.csv --target price --corr 0.85 --min-signal 0.05 --drop-weak
python scripts/filter_select.py data.csv --target churned --keep-list > features.txt
python scripts/filter_select.py data.csv --target churned --jsonDrops columns that are mostly missing (--max-missing, default 0.95), near-constant (--max-top-share,
0.995) or near-unique text; scores univariate signal (AUC for binary targets, |Spearman| for numeric, smoothed
out-of-fold level means for categoricals) and flags single-feature AUC > 0.9 as a possible leak; groups
numeric features with |Spearman| >= --corr and keeps the strongest of each group; marks or drops features
below --min-signal. Exit codes: 0 done, 2 usage or read error.
python scripts/importance_stability.py importances.csv --top-k 10
python scripts/importance_stability.py imp.csv --run-col fold --feature-col name --importance-col gain --jsonInput: long CSV (run, feature, importance) from folds, bootstraps or seeds. Prints per feature the mean
importance, mean rank, rank standard deviation and top-K selection frequency; groups features into stable
(>= --stable, default 0.8), unstable and never selected; and the mean pairwise Spearman agreement between
runs. Exit codes: 0 agreement >= --min-agreement (0.6), 1 unstable, 2 usage error.
Additional Resources
| File | Content |
|---|---|
| selection-methods.md | scikit-learn code for VarianceThreshold, SelectKBest/mutual information, L1 and elastic net, SelectFromModel, RFECV, SequentialFeatureSelector, permutation importance, Boruta-style shadow features, stability selection |
| selection-bias-and-nested-cv.md | How selection outside CV inflates scores (with a noise-only demonstration), nested CV, time-based validation of feature sets |
| correlated-features.md | Redundancy clustering, grouped permutation importance, VIF, choosing representatives, effects on L1 and trees |
| selection-reporting.md | One-SE rule, score-vs-features curves, cost-aware selection, documenting dropped features |
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.