feature-store-schema-designer

A feature store exists to make two things true: training rows get feature values as they were at each row's timestamp (point-in-time correctness), and production gets the same definitions with bounded staleness (no train/serve skew). The schema decides whether that works: entities and join keys, sources with reliable timestamps, feature views split by entity and refresh cadence, TTLs that match the cadence, and feature services that pin what each model version reads. scripts/lint_feature_spec.py checks a design spec for 12 common errors; scripts/gen_feast_repo.py turns the spec into a Feast repository (verified with feast apply and a historical retrieval on Feast 0.67).

Trigger Conditions

Activate this skill when the user:

Designs a feature store or migrates feature tables into Feast, Tecton, Databricks, SageMaker, Vertex AI or Hopsworks
Asks how to structure entities, feature views, TTLs, online/offline stores or feature services
Has train/serve skew, stale online features, or missing values from historical retrieval
Needs naming conventions, ownership, documentation or PII governance for shared features
Wants a spec reviewed before features are built

Window logic for the features themselves: time-based-feature-builder; point-in-time pitfalls: target-leakage-detector.

Workflow

text

1. ENTITIES   -> business keys (customer_id, card_id, store_id+sku_id); one entity per key set
2. SOURCES    -> tables with event timestamp (+ created timestamp for corrections) and refresh cadence
3. VIEWS      -> group features by entity x source x cadence; dtypes; descriptions; owners
4. TTL/SLA    -> ttl >= source cadence (+ slack); freshness_sla for online views
5. SERVICES   -> one feature service per model version (pins the feature list)
6. LINT       -> python scripts/lint_feature_spec.py spec.json
7. GENERATE   -> python scripts/gen_feast_repo.py spec.json --out feature_repo/ ; feast apply
8. VERIFY     -> historical retrieval row count == spine rows; online vs offline parity check

Core Concepts

Concept	Meaning	Design rule
Entity	the thing features describe, with join keys	stable business keys; composite keys for pairs (store, sku)
Source	table/stream with an event timestamp	timestamp = when the value became true; created timestamp = when written (dedupes corrections)
Feature view	features from one source for one entity set	split by cadence: real-time, hourly, daily, static
TTL	how far back a value may be used for a lookup	>= source cadence plus delays; too short -> empty lookups; too long -> stale values
Offline store	full history for training (warehouse, files)	point-in-time join: latest value with timestamp <= row time and >= row time - ttl
Online store	latest value per entity for serving (Redis, DynamoDB, SQLite)	filled by materialisation or streaming push
Materialisation	copying latest offline values to the online store	schedule at least as often as the source cadence
Feature service	named, versioned list of features a model uses	one per model version; never edit, create v2
On-demand features	computed at request time from request data + stored features	for values only known at request time (amount vs card mean)

Feature View Design

Group by	Because
Entity (join keys)	a lookup joins on one key set
Source	one view reads one table/stream
Refresh cadence	a daily view and a streaming view need different TTLs and pipelines
Ownership	one owning team per view

Good split for churn:

View	Entity	Source	Cadence	TTL	Online
`customer_activity`	customer	nightly aggregates	daily	2d	yes
`customer_profile`	customer	CDC of the accounts table	daily	400d	yes
`customer_realtime`	customer	event stream (push)	streaming	1h	yes
`plan_catalog`	plan	price list	weekly	30d	yes

TTL and Freshness

TTL >= cadence + delay. A daily view with ttl=12h returns empty online lookups for half the day (FS006).
TTL bounds staleness: with ttl=2d, a training row never receives a value older than 2 days, the same rule as serving. Static attributes get long TTLs (hundreds of days).
Freshness SLA (FS007) is the operational promise ("updated by 06:00 daily"); monitor it and alert.
Historical retrieval returns NaN (or, in some offline stores, drops the row) when no value falls inside the TTL: assert len(result) == len(spine) after every get_historical_features. On Feast 0.67's file offline store, combining views where one view's TTL had expired for a spine row dropped that row entirely.

Point-in-Time Retrieval

python

from feast import FeatureStore
fs = FeatureStore(repo_path="feature_repo")
spine = labels[["customer_id", "score_ts"]].rename(columns={"score_ts": "event_timestamp"})
train = fs.get_historical_features(entity_df=spine, features=fs.get_feature_service("churn_model_v1")).to_df()
assert len(train) == len(spine), "rows dropped by the point-in-time join (check TTLs and entity coverage)"

For each spine row the store picks, per view, the latest value with timestamp <= event_timestamp and within the TTL. That is only correct if the source's timestamp is the time the value became available; a nightly job stamping yesterday's aggregates with yesterday's date while they land at 06:00 today needs the timestamp set to the load time (or a latency-aware spine).

Naming and Governance

Rule	Example
snake_case, entity implied by the view	`logins_30d`, not `CustomerLogins30`
window suffix on every aggregate	`spend_7d`, `n_txn_1h`, `orders_lifetime`
units in the name when ambiguous	`amount_usd`, `duration_s`
description says the as-of rule	"Spend in the 30 days before event_ts"
owner per view; PII flagged	`pii: true` for email, phone, address, IP (FS010)
no labels in feature views	`churned`, `is_fraud` belong to the training spine (FS012)

Full conventions and the review checklist: naming-and-governance.md.

Output Format

When designing a feature store schema, deliver:

Entity and source table - keys, timestamps, cadence, owners.
Feature views - grouping rationale, TTL, online flag, freshness SLA, features with dtypes and descriptions.
Feature services - one per model version.
Spec + lint - the JSON/YAML spec and lint_feature_spec.py output with errors fixed.
Code - generated features.py and feature_store.yaml (or the platform equivalent).
Verification plan - retrieval row-count assertion, online/offline parity check, freshness monitoring.

Scripts

bash

python scripts/lint_feature_spec.py spec.json
python scripts/lint_feature_spec.py spec.yaml --json          # YAML needs PyYAML

FS001-FS005 errors (entity without join keys or unknown entity/source references, source without timestamp_field, duplicate names, missing or unsupported dtype, feature service referencing unknown views/features); FS006-FS010 and FS012 warnings (TTL missing or shorter than source cadence, online view without a freshness SLA or with an SLA looser than its TTL, non-snake_case names or aggregates without a window suffix, missing descriptions/owners, likely PII without pii: true, label-like features); FS011 info (views with more than 50 features). Exit codes: 0 no errors, 1 errors, 2 read error.

bash

python scripts/gen_feast_repo.py --template > spec.json      # churn example: 1 entity, 2 sources, 2 views, 1 service
python scripts/gen_feast_repo.py spec.json --out feature_repo/
python scripts/gen_feast_repo.py spec.json --stdout

Writes features.py (Entity, FileSource / BigQuerySource / SnowflakeSource / RedshiftSource, FeatureView with typed Fields, ttl, online flag, tags for owner and SLA, FeatureService with view[[...]] projections) and a local feature_store.yaml (SQLite online store, file offline store). Imports lint_feature_spec.py from the same folder and refuses specs with lint errors unless --force. Exit codes: 0 generated, 1 lint errors, 2 usage or read error.

Additional Resources

File	Content
spec-format.md	Every field of the JSON/YAML spec, supported dtypes and source types, a YAML example
feature-store-concepts.md	Online/offline stores, materialisation, streaming push, on-demand features, point-in-time joins, TTL semantics, skew
naming-and-governance.md	Naming conventions, documentation, ownership, PII and access control, versioning, deprecation, review checklist
platform-mapping.md	The same design in Feast, Tecton, Databricks Feature Engineering, SageMaker Feature Store, Vertex AI Feature Store, Hopsworks

anomaly-detection-model-builder

Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.

catboost-model-builder

Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).

categorical-encoding-selector

Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).

classification-model-selector

Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.

clustering-algorithm-selector

Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.

collaborative-filtering-implementer

Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.

feature-store-schema-designer

#Trigger Conditions

#Workflow

#Core Concepts

#Feature View Design

#TTL and Freshness

#Point-in-Time Retrieval

#Naming and Governance

#Output Format

#Scripts

#Additional Resources