Offline vs online store
| Offline store | Online store | |
|---|---|---|
| Holds | full history (every timestamped value) | latest value per entity key |
| Backed by | warehouse / lake (BigQuery, Snowflake, Redshift, Parquet, Delta) | key-value store (Redis, DynamoDB, Bigtable, Cassandra, SQLite for dev) |
| Used for | training sets, batch scoring | real-time serving (< 10 ms lookups) |
| Read API | get_historical_features(entity_df, features) | get_online_features(features, entity_rows) |
Point-in-time join
For a spine row (entity key k, timestamp t) and a feature view with TTL T, the offline store returns the
value v with the greatest source timestamp ts such that ts <= t and ts >= t − T (ties broken by the
created timestamp). The same rule, applied at "now", defines what the online store should hold.
Correctness depends on the source timestamp meaning "became available". If a nightly job writes aggregates
for day D at 06:00 on D+1 but stamps them D 00:00, training rows at D 12:00 see values production had not yet
computed. Stamp with the availability time, or shift the spine by the latency.
Materialisation
feast materialize-incremental $(date -u +%Y-%m-%dT%H:%M:%S) copies the latest values in the window since the
last run into the online store. Schedule it at least as often as the source cadence; monitor lag (now − last
materialised timestamp) against the freshness SLA.
Streaming and push sources
For features updated by streams (velocity counts), a stream processor computes aggregates and pushes them:
from feast import PushSource
card_push = PushSource(name="card_velocity_push", batch_source=card_txn_batch_source)
# producer: fs.push("card_velocity_push", df_with_keys_and_ts, to=PushMode.ONLINE_AND_OFFLINE)Pushing to both stores keeps history for training consistent with what was served.
On-demand features
Values that depend on request data (the transaction being scored) cannot be precomputed:
from feast import RequestSource, Field
from feast.on_demand_feature_view import on_demand_feature_view
from feast.types import Float64
txn_request = RequestSource(name="txn_request", schema=[Field(name="amount", dtype=Float64)])
@on_demand_feature_view(sources=[card_velocity_fv, txn_request],
schema=[Field(name="amount_vs_mean_24h", dtype=Float64)])
def amount_ratio(inputs):
import pandas as pd
out = pd.DataFrame()
out["amount_vs_mean_24h"] = inputs["amount"] / (inputs["amount_sum_24h"] / inputs["n_txn_24h"].clip(lower=1))
return outThe same Python runs at training (on historical requests) and serving time.
Train/serve skew sources and fixes
| Source of skew | Fix |
|---|---|
| Different code computes features offline and online | one definition (feature view / on-demand view) |
| Online values staler than training assumed | TTL and freshness SLA; train with latency-aware timestamps |
| Missing online values replaced differently | same defaults in both paths; log served features |
| Type differences (int vs float, string categories) | declared dtypes; validation at materialisation |
| Late corrections in the offline store | created timestamps; snapshot what was served |
Parity check
Sample entities, read online features now and the offline point-in-time value at the same timestamp; the differences should be zero up to the materialisation lag. Run it after every pipeline change.
TTL semantics in practice
| TTL too short | TTL too long |
|---|---|
| online lookups return null between refreshes | stale values served silently after a pipeline outage |
| training rows get NaN although data existed | training uses values production would have considered expired |
Rule of thumb: ttl = cadence + max expected delay, e.g. daily job that can be 12 h late -> ttl = 36h; static
attributes -> long TTL; streaming -> minutes to hours.
anomaly-detection-model-builder
Build and evaluate anomaly detection for tabular, sensor and transaction data with few or no labels: define the anomaly type (point, contextual, collective), choose detectors (robust z-score/MAD and IQR rules, Mahalanobis with robust covariance, k-NN distance, LOF, Isolation Forest, one-class SVM, PCA reconstruction / autoencoders, forecasting residuals for time series), combine them by rank, explain each alert by its deviating features, set thresholds from an alert budget or the score distribution, and evaluate with precision@k, PR-AUC and recall on known incidents, or detector agreement when no labels exist. Includes pure-Python detectors (robust z, Mahalanobis, kNN, Isolation Forest, ensemble) and an evaluation script.
catboost-model-builder
Build CatBoost models that use its strengths correctly: raw categorical columns as cat_features (ordered target statistics, no manual encoding), text_features, Pool construction with NaN-safe categories, ordered boosting and has_time for temporal data, symmetric (oblivious) trees, overfitting detector with eval_set and use_best_model, losses for classification, regression (Quantile, Poisson, Tweedie) and ranking (YetiRank), class weights, monotone constraints, feature importance (PredictionValuesChange, LossFunctionChange) and SHAP values, GPU training and model export. Plans cat/text/numeric columns from a CSV and generates a runnable training script, and checks parameter sets for the conflicts CatBoost rejects (iterations vs n_estimators, Bayesian bootstrap with subsample, multiple class-weight options, depth > 16, use_best_model without eval_set).
categorical-encoding-selector
Choose and implement the right encoding for every categorical column given the model family and the data: one-hot (with infrequent-level grouping and handle_unknown), ordinal with an explicit order, cross-fitted target encoding (scikit-learn TargetEncoder, CatBoost ordered statistics, category_encoders inside CV), frequency/count encoding, feature hashing, native categorical support in LightGBM, HistGradientBoosting, XGBoost and CatBoost, prefix levels for codes (zip3, NAICS-2), and embeddings. Measures cardinality, rare-level share, unseen-level rate on a time-ordered holdout and target signal per column, emits the ColumnTransformer, and quantifies target-encoding leakage (in-sample vs out-of-fold vs holdout AUC).
classification-model-selector
Choose the right classification algorithm for tabular, text or mixed data and prove the choice with a fair bake-off: logistic regression (L2, L1, elastic net), linear SVM, naive Bayes, k-NN, RBF SVM, decision trees, random forest, extra trees, HistGradientBoosting, LightGBM, XGBoost, CatBoost, explainable boosting machines (EBM/GAM), points scorecards and MLPs, matched to data size, width, categorical and high-cardinality share, missingness, class imbalance, text, and constraints such as interpretability, monotonic constraints, calibrated probabilities, latency budgets and tiny deployable models. Ranks candidates with starting hyperparameters and cautions, then generates a bake-off script running every candidate on identical CV folds with per-fold metrics, fit time and prediction latency.
clustering-algorithm-selector
Choose the right clustering algorithm and check that clustering makes sense at all: cluster tendency (Hopkins statistic with a correlated-features caveat, nearest-neighbour isolation), then k-means / MiniBatchKMeans, Gaussian mixtures with BIC, agglomerative (Ward, average, complete, single) with dendrograms, DBSCAN, HDBSCAN, OPTICS, spectral clustering, BIRCH, mean shift, k-prototypes and k-modes for mixed or categorical data, and Gower distance, matched to data size, expected shapes, noise, varying density, known k, soft memberships, hierarchy and streaming. Covers preprocessing (scaling, encoding, dimensionality, distance metrics), validation (internal indices, bootstrap stability, external labels, business profiling) and assigning new points.
collaborative-filtering-implementer
Implement collaborative filtering recommenders that beat popularity: item-based kNN with cosine, Jaccard or asymmetric similarity, shrinkage, neighbourhood size, positive-only or mean-centred rating signals and 'because you liked X' explanations; implicit ALS with confidence weighting, BPR pairwise ranking and the closed-form EASE model in plain NumPy, tuned on a temporal validation split by recall@k and NDCG@k and refit on all data; plus user-kNN, explicit matrix factorisation, the implicit/LightFM libraries, hyperparameter ranges, cold-start fallbacks, scaling with sparse matrices and approximate nearest neighbours. Produces top-k lists in the format of the recommendation evaluator.