Produkcijski webhook i live-signal audit

Scope: requested analysis of all generators locally, plus a production (.207) check of whether signals are being dispatched over webhooks, whether they should have been, the per-signal statistics, and a counterfactual against deeper klines history.

TL;DR

• Production has dispatched 0 strategy webhooks — by mechanical failure, not by config. Every enabled generator runs every minute and logs produced 0 signals. No generator has emitted a single live signal since the 2026-05-20 backfill (~40 days). The only webhook target/routes that exist are armed for growth_threshold_quality_blacklist_cycle, which produces nothing live, so nothing is ever sent.
• Root cause: live-scan window vs feature-refresh cadence mismatch. Features are recomputed only by the daily ~05:00 UTC catchup, so the newest 4h feature is the candle closing ~03:59 (1h: ~04:59). The live scan window is just the last ~2 candles ending at utcnow() (GENERATOR_LIVE_LOOKBACK_CANDLES=2). For ~20 h/day that window starts after the newest available feature, so feature_store.get_features returns 0 rows → 0 candles processed → last_processed_at never advances past the May backfill boundary.
• Downloading the rest of history will NOT fix this. Prod klines (15m/1h/4h/1d) and the feature table are already fresh to today; the defect is in the live pipeline, independent of backtest depth. Counterfactual signal volume (from the local full-history DB) is large, but the webhook-wired generator is a net loser on 1h, so "fixing" delivery without changing the wiring would have streamed losing signals to 3Commas.

Production webhook configuration (.207)

• 1 webhook target: 3commas-paper → https://api.3commas.io/signal_bots/webhooks (enabled).
• 4 routes, all enabled, all keyed to generator growth_threshold_quality_blacklist_cycle: buy/1h, sell/1h, buy/4h, sell/4h.
• No route exists for any other generator, nor for the live market-scanner stream.

What is actually flowing live on prod

• signal_events__growth_threshold_quality_blacklist_cycle: 3,988 rows, all is_backfill=true, received 2026-05-20 00:52–16:26. 4h only (1995 buy / 1993 sell); 0 rows on 1h even though two routes target 1h. Zero live rows ever.
• All other enabled generators (bb_adx_combo_long, bollinger_band_pullback_long, best_symbols_cycle, best_symbols_cycle_v2, supertrend_long): 0 live signals; only best_symbols_cycle/v2 have backfill rows (2811 / 2293).

Evidence the pipeline is alive but emitting nothing

• generator_logs: 1.72M rows, current to 2026-06-29 18:07. Every cycle for all 6 enabled generators logs <gen> produced 0 signals.
• generator_runtime_state is being written today (best_symbols 4h, blacklist 1h: last write 18:15) but last_processed_at does not advance:
• blacklist_cycle 4h: max last_processed_at = 2026-05-19 (frozen).
• blacklist_cycle 1h: last_processed_at = None for all 54 symbols (never processed one candle).
• best_symbols_cycle 4h: frozen at 2026-05-20.
• Direct probe of the live read path inside the scheduler container:
• get_features(BTCUSDT, 4h, start=now-8h, end=now) → 0 rows.
• get_features(BTCUSDT, 4h, start=2026-05-01, end=now) → 355 rows, max close_time 2026-06-29 03:59 (feature table is fresh; only the live window is empty).

Klines / feature freshness (prod klines_cache.db, 28.6 GB, written today)

• klines fresh to today for the generator timeframes: 15m/1h/4h max open_time ≈ 2026-06-29; 1d ≈ 2026-06-27/28. Minor intervals (1m/3m/5m/30m/2h/6h/8h/12h) stale since the 2026-05-30 base-coverage build (not part of catchup).
• features fresh: 4h max close 2026-06-29 03:59 (139 syms), 1h 04:59 (137 syms), 15m 04:59 (68 syms), 1d 2026-06-28. regimes stale: 1h only, 2,377 rows, max 2026-06-01.

Local full-history generator analysis (counterfactual)

Re-ran tools/analysis/analyze_all_generators.py against local crypto_scanner.db on 2026-06-29 → reports/historical_generator_stats_2026-06-29.txt (full backfill = the "rest of history" scenario). Selected rows (after-fee = avg return − 0.2% round-trip):

Counterfactual vs actual:

• Actual live (prod, 40 days): 0 signals, 0 webhooks, for every generator.
• With a working live pipeline, blacklist_cycle would emit at roughly its historical cadence — dominated by 1h, which is negative expectancy (−0.23% after fee over 14.6k trades, 31.5% win). Its only positive slice is 4h, but that is a 13-trade sample.
• The genuinely profitable generators in the full-history view (dual_growth_trailing_stop, bb_adx_combo_long, supertrend_long 4h, best_symbols_cycle_v3) are either disabled or not webhook-wired.

Conclusions / recommended fixes (not yet applied)

1. Live pipeline (P0). Make features refresh to the latest closed candle intraday (not just the daily catchup), or widen the live scan / let the generator read fresh klines directly for the trailing window. Today the live window is empty of features ~20 h/day. Without this, no strategy webhook can ever fire.
2. Webhook wiring (P0, product). The only wired generator is a 1h net-loser. Decide: re-point routes to a profitable + enabled generator (e.g. bb_adx_combo_long, supertrend_long 4h), and/or drop the two 1h routes (the generator emits no 1h signals anyway). Confirm whether the live scanner_positive/negative stream should have a route.
3. Regimes (P1). Regime table is 1h-only and stale since 2026-06-01; regime-gated generators will misbehave once the live pipeline is restored.
4. History download (P2, optional). Deepens backtests for low-coverage/disabled generators; does not affect live generation. Launch only if deeper backtests are wanted.

Source data: prod PG crypto_scanner-db-1, prod klines_cache.db via scheduler-1, local crypto_scanner.db. Read-only; no production state changed.