Directional price movement prediction in a non-stationary, low signal-to-noise domain, where the
stated goal is not profitability but facing overfitting and data leakage rigorously.
Walk-forward validation, out-of-sample backtest with costs, hybrid per-ticker routing and real news sentiment.
«Existen numerosas formas de arruinarte. De entre ellas, crear un algoritmo de machine learning
que trate de predecir el valor de las acciones en bolsa para luego invertir tu dinero,
es una de las más rápidas y efectivas.»
Exploración técnica de Machine Learning aplicado a series temporales financieras.
El objetivo no fue buscar rentabilidad, sino enfrentarse a un problema real con alta
incertidumbre y aprender de los errores — especialmente del overfitting y del data leakage.
Predict whether an asset rises more than 2% in 5 days using Random Forest over multi-ticker data
plus sentiment features. The honest framing is explicit: most published results in this domain fail
out-of-sample, and this project treats that as the problem to solve: not a model
to squeeze.
Task
Binary classification, 7 tickers · RF + VADER sentiment + NewsAPI.ai real news
→Walk-forward is the only honest evaluation: 3-fold expanding window. The single 80/20 split inflated accuracy from 54.1% to 57.7% — discovered and corrected.
→Scaler fitted on train only: the test set never touches the scaler.
→Target built with shift(-5): the future only labels, never leaks into features.
→RandomForest over HistGradientBoosting: RF wins AUC in 7/7 tickers (+0.028 avg). Calibrated probabilities rejected by predefined criterion.
→Hybrid routing: pool as fallback: NVDA routes to a global model (pool AUC 0.539 vs individual 0.501). Others stay individual — TSLA is too idiosyncratic for pooling.
→Real sentiment via NewsAPI.ai: 4,303 articles fetched, VADER-scored and aggregated daily. GDELT blocked from this network; free plan archives ~60 days.
→Leakage tests in CI: tests verify no feature correlates ≥0.95 with the target.
LightGBM breakthroughAfter 12 failed improvements, LightGBM + ensemble RF+LGB finally beats RF alone: +0.007 AUC average (6/7 tickers), +0.058 accuracy average. LightGBM wins AUC in 4/7, ensemble wins in 6/7. Different model biases (bagging vs boosting) complement each other.
Model comparisonRandomForest wins AUC in 7/7 tickers over HistGradientBoosting (+0.028 avg). Accuracy improved from 54.1% → 57.7% after tuning Optuna with walk-forward as the objective.
The value is the evaluation rigour, not the model performance: knowing when a model
does not work is precisely what protects real capital. The single-split evaluation inflated
results by +3.6pp — only walk-forward revealed the true picture. After 12 failed attempts,
the breakthrough came from combining models with different inductive biases (RF bagging + LGB
boosting), not from more features or more complexity. This is the foundation
for the portfolio manager system in development.
The codebase is structured into 16 modules under inversion/, each
with a single responsibility. Data flows through a deterministic DAG: raw OHLCV → features →
sentiment → targets → train → evaluate → serve.
VADER scorer, NewsAPI.ai article fetcher, daily aggregation, time-aligned with OHLCV
inversion/models
Per-ticker Random Forests, pooled global model, ONNX export, probability calibration
inversion/tuning
Optuna with walk-forward AUC objective, pruning, Bayesian search, 200 trials/ticker
inversion/backtest
Walk-forward backtest with costs (0.1% commission + 0.05% slippage), Sharpe, max drawdown
inversion/api
FastAPI endpoints, prediction routing, health check, Swagger docs
inversion/monitoring
PSI drift detection, model retrain triggers, prediction logging, alerting hooks
metrics
Detailed per-ticker metrics (RF+LGB Ensemble)
Ticker
Walk-Fwd Acc
Walk-Fwd AUC
RF AUC
Baseline Acc
Δ Acc
Δ AUC
Model
GOOGL
0.634
0.565
0.564
0.523
+0.111
+0.034
RF+LGB
META
0.633
0.580
0.574
0.567
+0.066
+0.044
RF+LGB
MSFT
0.651
0.560
0.547
0.448
+0.203
+0.112
RF+LGB
AMZN
0.646
0.547
0.538
0.608
+0.038
−0.027
RF+LGB
TSLA
0.561
0.542
0.530
0.586
−0.025
−0.028
RF+LGB
AAPL
0.632
0.514
0.503
0.660
−0.028
−0.025
RF+LGB
NVDA
0.543
0.496
0.496
0.482
+0.061
+0.014
RF+LGB
Notes: Walk-forward 3-fold, same protocol as baseline. Ensemble = 0.5×RF + 0.5×LGB.
Δ Acc = ensemble − persistence baseline. Δ AUC = ensemble − RF individual.
Ensemble improves AUC in 6/7 tickers (+0.007 avg). NVDA remains unpredictable (AUC < 0.50).
Accuracy is consistently higher than RF alone across all tickers.
Notes: Positive Δ means the model beats the baseline (persistence/naive forecast).
GOOGL shows the strongest improvement (+0.122 acc, +0.095 AUC). AMZN and TSLA actually degrade on
accuracy but improve AUC — the model learns better ranking but worse point predictions.
NVDA is routed to the global pooled model because its individual AUC dropped below 0.55.
backtest
Out-of-sample backtest with costs
A walk-forward backtest simulates daily trading: enter on BUY signals, exit on SELL,
with realistic frictions. This is the closest thing to "how would this actually perform?"
Model strategy return
53.59%
Buy & Hold return
281.70%
Costs applied
0.1% commission + 0.05% slippage per trade
Verdict
Model underperforms buy & hold. Demonstrates rigor, not profitability.
The strategy with model predictions + costs (0.1% commission + 0.05% slippage) yields 53.59%
vs 281.70% for simple buy & hold. This confirms the project's core thesis: honest
evaluation is the value. A model that underperforms buy & hold is better than a
model that overfits and hides behind inflated metrics.
limitations
Limitations & robustness
→Magnitude of improvement: The model beats persistence on AUC in all 7 tickers, but accuracy gains are inconsistent (positive for 4/7, negative for 3/7). Ranking is better than point prediction — typical for low signal-to-noise domains.
→Robustness across regimes: The model was trained on 2007–2026 data including the 2008 crisis, 2020 pandemic, and various bull/bear markets. Performance varies by regime — this is expected and honest.
→Economic utility: Better predictions do not automatically translate to profitable strategies. The backtest with realistic costs confirms this. The project is a technical exercise in rigorous evaluation, not a trading system.
→Monitoring & drift: PSI (Population Stability Index) monitoring is in place but the system has not been deployed in production for a full market cycle. Retrain triggers are defined but untested at scale.
failed improvements
12 failures → 1 breakthrough
After the baseline was established, 12 systematic improvement attempts were
implemented and tested. None surpassed the original Random Forest. This is the honest record —
documenting failures is what separates rigorous evaluation from cherry-picked results.
#
Improvement
Result
Δ AUC
1
HGB tuning (Optuna)
Worse than RF individual
−0.028
2
Ensemble (RF + HGB + RF-sent)
Worse than RF alone
−0.005
3
Macro features (VIX, sectors, treasury, DXY)
Noise overwhelms signal
−0.034
4
Advanced features (Hurst, entropy, Roll's spread)
No change from baseline
±0.000
5
Stacking meta-learner
Meta-learners worse than base
−0.012
6
SHAP top-5 feature selection
Removing context hurts generalization
−0.014
7
ElasticNet (strong regularization)
Marginal improvement (only one)
+0.009
8
Regime-specific models (bull/bear/neutral)
Per-regime models overfit
−0.003
9
Continuous target (regression)
R² negative (worse than mean)
n/a
10
Diverse ensemble (Linear + RF + HGB)
Different biases don't complement
+0.002
11
Purged CV + imbalance handling
Better methodology, same metrics
±0.000
12
Edgebet formulas (Poisson, BTTS, Over/Under)
Football formulas don't transfer
±0.000
★
LightGBM + RF ensemble
Breakthrough: 6/7 tickers improve
+0.007
Why none worked (until #13): financial time series have inherently low signal-to-noise
ratios (~1–5% predictable), markets are semi-formally efficient, and increased model
complexity amplifies overfitting rather than capturing real structure. The only marginal
gain came from ElasticNet's strong regularization (+0.009 AUC), which suggests the problem
benefits from simpler models, not more complex ones. The breakthrough (#13)
came from combining models with different inductive biases — RF's bagging + LGB's
boosting — not from more features. This is a well-known result in quantitative finance —
and the project's honest documentation of the 12 failures + 1 success is the actual contribution.