Faza 3 — dnevni catchup i symbol churn

Status: deploy gotov, oba job-a verifikovana

Klines cache sad ima automatsko održavanje. Dva nova cron job-a u scheduler-u:

• 05:00 UTC daily — update_klines_cache_daily: per (symbol, interval) inkrementalni fetch od MAX(open_time)+1interval do now
• 06:00 UTC Monday weekly — weekly_symbol_churn: detekcija novih USDT+BTC perpetuala, 2y backfill za njih

Šta je deploy-ovano

1. src/klines_cache_maintenance.py (novi modul, ~250 lines)

Dva entry-point-a:

• daily_catchup(analyzer, intervals=DEFAULT_INTERVALS) → Dict — incremental update
• weekly_symbol_churn(analyzer, intervals, backfill_days=730) → Dict — new symbol detection + backfill

Karakteristike:

• Svaki entry-point kreira svoj AsyncClient (isolated event loop) — radi pravilno kad se zove iz scheduler thread-a
• Otvoreni candle filter na insert-u (close_time_ms > now_ms skip)
• INSERT OR REPLACE — idempotentno, stale rows mogu da budu refresh-ovani
• Per-symbol error catch — pad jednog ne ruši ceo job
• Eligible symbols (USDT+BTC base intersection) sad uključuje OBA USDT i BTC pair-a — generator koristi oba, churn ih ne sme markirati kao delisted

2. src/scheduler.py izmene

Dodato:

• last_klines_catchup_at, last_klines_catchup_status polja
• last_symbol_churn_at, last_symbol_churn_status polja
• _write_klines_maintenance_status() — piše JSON status u reports/klines_maintenance_status.json
• run_klines_catchup() i run_symbol_churn() metode
• setup_schedule(): dva nova schedule.every().day/monday.at(...).do(...) reda

Schedule setup: Scan every 5 minutes, Monitor every 10 minutes, Generator live every 10s,
                Klines catchup daily at 05:00 UTC, Symbol churn weekly Mon at 06:00 UTC

3. src/config.py — 2 nova ENV podržana

• KLINES_CATCHUP_TIME_UTC (default 05:00)
• SYMBOL_CHURN_TIME_UTC (default 06:00)

4. src/web_app.py — nova 2 admin endpoint-a

• POST /api/admin/klines_catchup/run — manual trigger
• POST /api/admin/symbol_churn/run — manual trigger
• /api/health proširena sa last_klines_catchup_* i last_symbol_churn_* poljima

5. docker-compose.yml — novi scheduler service

scheduler:
  build:
    context: .
    network: host
  restart: always
  command: python scheduler_service.py
  environment:
    - DATABASE_URL=postgresql://...
    - REDIS_URL=redis://redis:6379/0
    - ENABLE_SCHEDULER=true
  env_file:
    - .env
  volumes:
    - ./reports:/app/reports
    - ./database:/app/database
  depends_on:
    - db
    - redis

Razlog: web container koristi gunicorn app:app koji ne pokreće scheduler.start_scheduler(). Bez separate scheduler containera, nijedan cron job nije aktivan.

Backup originalnog compose-a: docker-compose.yml.bak.2026-05-20-v2

Bug otkriveni tokom validacije

Bug 1: BTC pairs lažno označeni kao delisted

Originalni _async_get_eligible vraćao samo USDT verzije. Pošto cache sadrži oba (generator računa btc_growth_pct), 71 BTC pair je lažno označen kao delisted. Fix: eligible sad vraća OBA {base}USDT i {base}BTC.

Posle fix-a: 139 cached, 136 eligible, 3 stvarno delisted (BTCUSDT [koji nema BTCBTC pair], ENAUSDT i ENABTC).

Bug 2: KlinesProvider._write_cache je pisao open candles

Otkriveno: clean_stale_open_candles.py --dry-run našao 275 stale rows posle Faze 2 equivalence test-a. Source: KlinesProvider._write_cache nije imao open-candle skip. Fix: dodat if close_ms > now_ms: continue. Posle deploy-a + restart-a web/celery containera (gunicorn worker re-import), 0 stale rows.

Validacija

Smoke test (manual trigger)

Daily catchup (1d only): elapsed=32.8s, symbols=139, 69 fetched (open candles, skipped on insert)
Symbol churn: cached=139, eligible=136, new=0, delisted=3 (BTCUSDT, ENABTC, ENAUSDT)
Stale rows after run: 0

Schedule registered

Iz scheduler container log-a:

2026-05-20 16:38:27 [INFO] src.scheduler: Schedule setup: ..., Klines catchup daily at 05:00 UTC, Symbol churn weekly Mon at 06:00 UTC
2026-05-20 16:38:27 [INFO] src.scheduler: Scheduler started
2026-05-20 16:38:27 [INFO] __main__: Scheduler service running

Operativni značaj

Pre Faze 3

• Cache je bio statičan posle bulk fetch-a (2026-05-20 14:55)
• KlinesProvider bi morao da uvek fetch-uje gap između cache_max i now na svaki generator backfill
• Posle 2 dana bez catchup-a, svaki backfill koji ide do now mora da napravi 100+ Binance poziva po simbolu

Posle Faze 3

• Cache se osvežava daily u 05:00 UTC — taman posle BTC daily close (UTC 00:00 + 5h cushion)
• KlinesProvider ima 99.99% cache hit za sve generator backfill-ove
• Symbol churn weekly hvata nove listed-ove (typical Binance ~2-5 novi USDT pairs/mesec)
• Delisted pairs ostaju u cache-u za istorijsku analizu — backtest može da uključi delisted simbole

Pre-flight provere (sve ✅)

• [x] Scheduler container running, log potvrđuje schedule setup
• [x] Daily catchup manual trigger uspeo
• [x] Symbol churn manual trigger uspeo
• [x] 0 stale open candles posle 2 catchup runs
• [x] Status file reports/klines_maintenance_status.json se piše posle svakog run-a
• [x] Web + celery containers restartovani sa novim kodom (gunicorn worker re-import)

Sledeći koraci (Faza 4 i 5)

• Faza 4 (opciono): klines_features tabela sa precomputed RSI/ADX/EMA/BB/Supertrend
• Generator code postaje munjevit: umesto df['rsi'] = compute_rsi(df, 14), koristi df['rsi'] = df['rsi_14']
• Trigger: posle svake catchup-batch-insert u klines → recompute features za novi range
• Faza 5 (kasnije): multi-exchange via ccxt (Kraken, Bybit, OKX) sa exchange u PRIMARY KEY

Provenance

• Skripte: src/klines_cache_maintenance.py, src/scheduler.py (izmena), src/config.py (2 nove ENV), src/web_app.py (2 endpoint + health), tools/history_cache/inspect_stale.py, tools/analysis/faza3_smoke_test.py
• Docker: docker-compose.yml + scheduler service; backup .bak.2026-05-20-v2
• Reports: ovaj report + reports/klines_maintenance_status.json (auto-updated)
• Linked: crypto-scanner-plan-klines-cache, crypto-scanner-session-2026-05-20-faza2