Algoenjin Documentation

Single source of truth for how this project is laid out, how to run it, and why things are the way they are. Rendered in the app at /docs, read straight from this file.

This file is part of every change. If you alter structure, commands, ports, endpoints, or conventions, update the relevant section here in the same pass — see CLAUDE.md.

1. What this is

Research platform for a parameterized Opening Range Breakout (ORB) strategy: a backtrader engine, a grid optimizer, walk-forward validation, and a Next.js front end to drive it all.

Scope is backtesting and optimization only. There is no live order execution. The MetaTrader 5 Python API is Windows-only and is deliberately not part of the research loop.

2. Layout

engine/
├── DOCS.md                 this file — the Algoenjin Documentation
├── CLAUDE.md               working conventions (incl. the rule that keeps this file current)
├── README.md               short project intro and quickstart
├── .gitignore              patterns are non-anchored, so they match at any depth
├── .claude/
│   ├── launch.json         both processes, ready to run
│   ├── settings.json       hooks (see §11)
│   └── hooks/check-docs.sh the doc-drift check
│
├── backend/                everything Python — this is the Python project root
│   ├── algo/               strategy, data, backtest engine, optimizer, validation, reporting
│   │   └── strategies/     orb.py (the engine's ORB) + the ported strategy types, see §10
│   ├── api/                FastAPI server (server.py)
│   ├── scripts/            CLI entry points
│   ├── tests/              pytest suite
│   ├── config/             instruments.yaml + experiments/*.yaml (sweep grids)
│   │                       + ou_universe/ (the OU stock universe, row order matters)
│   ├── strategies/         saved strategy definitions (*.json)
│   ├── data/               cached OHLCV parquet (gitignored)
│   ├── results/            optimization + run outputs (gitignored)
│   ├── reports/            generated HTML/PNG reports (gitignored)
│   ├── pyproject.toml
│   ├── .env                DATABENTO_API_KEY (gitignored)
│   ├── deploy/              systemd units for the nightly sweep (not installed by default)
│   └── .venv/
│
└── frontend/               Next.js app (port 3005)
    ├── src/app/            routes
    ├── src/components/     UI primitives + feature components
    ├── src/lib/            api client, formatters, strategy tiering
    └── next.config.ts      /api proxy + dev origins

Why backend/ holds the data folders too

backend/algo/config.py derives every path from the directory containing algo/:

ROOT = Path(__file__).resolve().parent.parent    # → backend/
CONFIG_DIR   = ROOT / "config"
DATA_RAW     = ROOT / "data" / "raw"
DATA_PROCESSED = ROOT / "data" / "processed"
RESULTS_DIR  = ROOT / "results"
REPORTS_DIR  = ROOT / "reports"

api/server.py and scripts/_bootstrap.py do the same. So config/, data/, results/, reports/, strategies/, and .env must stay siblings of algo/ inside backend/. Moving any of them up to engine/ silently breaks path resolution. This is also why the restructure needed zero Python source changes.

3. Running it

Two processes. The uvicorn command works from any directory.

# API — port 8500
backend/.venv/bin/uvicorn api.server:app --port 8500

# Frontend — port 3005
cd frontend && npm run dev

.claude/launch.json has both wired up.

Window discovery

Finding a strategy's trading window is a background job, not a request — see §9 for why.

# search around a saved strategy (slug from GET /api/strategies)
backend/.venv/bin/python backend/scripts/discover_window.py --slug my-strategy-v3 --budget standard

# overnight drift check across deployed strategies; reports, never applies
backend/.venv/bin/python backend/scripts/nightly_reoptimize.py --deployed --budget quick

The written rationale needs ANTHROPIC_API_KEY in backend/.env. Everything else works without it — the search, the gates and the plateau check are pure Python, and a missing key only costs you the plain-English write-up.

On the VPS, the API runs under systemd

algoenjin-api.service runs the same uvicorn command as ubuntu, Restart=always, so the API survives reboots and no longer depends on a tmux session:

sudo systemctl status algoenjin-api
sudo systemctl restart algoenjin-api
sudo journalctl -u algoenjin-api -f

Two things about that unit file are non-obvious, both caused by the spaces in the project path: WorkingDirectory= takes the rest of the line verbatim and rejects quotes, while ExecStart= is parsed into arguments and so requires the executable to be quoted. Quoting WorkingDirectory fails with path is not absolute.

Why no --app-dir and no cd

api is listed in pyproject.toml under [tool.setuptools.packages.find] include = ["algo*", "api*"], so api.server:app resolves from the installed package rather than from the working directory. Combined with the __file__-relative paths above, the server is fully cwd-independent.

First-time setup

cd backend
python3 -m venv .venv
.venv/bin/pip install -e ".[dev,report,api]"

cd ../frontend
npm install

Do not copy .venv/ or node_modules/ between machines. Both contain platform-native binaries — a macOS venv points at /opt/homebrew, and macOS node_modules ships swc-darwin-arm64 / sharp-darwin-arm64, which fail on Linux. Recreate them per machine.

4. Frontend ↔ backend wiring

The browser only ever talks to the Next origin. src/app/api/[...path]/route.ts forwards /api/* to the backend server-side, and src/lib/api.ts defaults its base URL to "" (same origin).

browser ──▶ :3005 (Next)  ──route handler──▶  127.0.0.1:8500 (FastAPI)

This used to be a rewrite in next.config.ts. It became real code because a Next rewrite cannot add request headers, and the deployed backend sits behind nginx demanding an X-API-Key. The handler reads the same API_ORIGIN variable, so local behaviour is unchanged; it also exports maxDuration = 300, because backtests run synchronously and have been observed at ~135s.

This matters because a client-side fetch("http://localhost:8500") resolves to the viewer's machine. When the app is served from a remote host, that request goes nowhere and every page spins forever. Proxying through Next fixes it and keeps 8500 bound to loopback.

• Override with NEXT_PUBLIC_API_BASE to point the browser at the API directly.
• Override with API_ORIGIN to change where Next proxies to.
• allowedDevOrigins in next.config.ts lists hosts allowed to load dev/HMR resources.

Port 8500 stays on 127.0.0.1 and is not exposed. Port 3005 binds all interfaces. The app has no authentication and /api/jobs can spawn processes, so prefer an SSH tunnel (ssh -L 3005:localhost:3005 …) or a firewall rule over leaving 3005 open.

Deployment: Vercel front end, VPS backend

browser ──▶ algoenjin.com (Vercel)  ──https──▶  api.algoenjin.com (nginx, VPS)
              password gate                       X-API-Key gate
                                                      └──▶ 127.0.0.1:8500

The FastAPI backend cannot run on Vercel — it spawns subprocesses and reads the local parquet cache — so it stays on the VPS and is published through nginx.

Vercel project settings. Root Directory must be frontend; the repository root holds backend/ and frontend/, and building from the root yields 404: NOT_FOUND. Environment variables, neither of them NEXT_PUBLIC_, so neither reaches the browser bundle:

Two independent gates, and both are load-bearing. ALGOENJIN_API_KEY stops anyone who finds api.algoenjin.com from reaching the API directly. It does not stop a stranger loading the public Vercel site and having its proxy attach the key for them — that is what SITE_PASSWORD is for. Leaving SITE_PASSWORD unset disables the gate, which is correct locally and wrong in production.

nginx config lives at /etc/nginx/sites-available/api.algoenjin.com: it returns 401 unless X-API-Key matches, and sets proxy_read_timeout 300s because the 60s default severs a long backtest mid-run. A pre-existing conf.d/00-default-deny.conf drops unknown Host headers and direct-IP access with 444.

5. Backend reference

CLI scripts (backend/scripts/, run with .venv/bin/python)

API endpoints (backend/api/server.py)

Long-running work (optimize, walk_forward, data generation) is spawned as a subprocess job with cwd=ROOT and in-memory log capture, then polled via /api/jobs/{id}.

Instruments

Defined in backend/config/instruments.yaml: EURUSD GBPUSD XAUUSD US30 US100 NQ ES GC, plus, for the ported strategies, XAGUSD XPTUSD USDJPY CHFJPY US500 and the two OU portfolio clocks OU_SP500 OU_NDX100 (benchmark index bars; the stocks themselves get their instrument from ou_modell.instrument_for). The yen pairs carry point_value: 0.0067 — a static ~150 JPY/USD conversion, so dollar risk and PnL are approximate while each trade's R is exact.

Swept parameters

Range length · session open/close · entry style (candle_close / candle_cross / retracement 25–75%) · stop (range-based) · take-profit (rr 1:1–4:1 or multi_tp scale-out) · risk sizing (percent of equity or fixed $).

Tests

cd backend && .venv/bin/pytest        # 119 tests (-m "not slow" skips the one slow run)

Covers entries, exits, sizing, sessions, splits, ORB logic, timeframes, time-based range, trade log, run store, strategy store, archive, and the strategy types (test_strategy_types.py: registry, params round-trip, the close-confirmed planner, replay fills at exact prices, partials, concurrency, multi-symbol books, margin capping, cap vs feed assignment).

6. Frontend reference

Design tokens (surfaces, accents, chart series, motion) are CSS custom properties in src/app/globals.css, exposed to Tailwind via @theme inline.

Dependencies: next react recharts (charts) lucide-react (icons) marked (renders this file at /docs). Markdown is styled by the hand-rolled .docs-prose block in globals.css rather than @tailwindcss/typography, whose defaults fight this palette.

Window discovery section

components/window-discovery.tsx renders on /strategies under the builder: strategy picker, objective, budget, a live cost estimate, the job log, then the result — window found vs. the one it started from, the four gates on out-of-sample, a baseline → in-sample → out-of-sample comparison, Claude's write-up, and the round-by-round trace. It reads the gate verdict from the backend rather than recomputing it with evaluateCriteria, so the two copies of the thresholds cannot disagree about a window the search has just proposed.

7. Responsive / mobile conventions

The layout breakpoint is lg (1024px): below it the sidebar is an off-canvas drawer, above it the permanent 240px rail.

.scroll-x also carries pure-CSS scroll shadows (background-attachment: local, scroll) that appear only when there is more content in that direction — no scroll listener.

Checking mobile after UI work

Zero horizontal page overflow at 390px and 360px is the bar. A quick throwaway harness:

mkdir -p /tmp/mcheck && cd /tmp/mcheck && npm init -y
npm i playwright && npx playwright install firefox
# then drive it: for each route, assert
#   document.documentElement.scrollWidth <= clientWidth + 1

Delete it afterwards — it pulls ~300 MB of browser binaries.

8. Conventions

• Python paths are always derived from __file__, never from cwd. Keep it that way.
• cn() in src/lib/utils.ts is a plain join, not tailwind-merge. Conflicting Tailwind classes passed via className will not reliably override the base. Change the component's own classes, or add a prop (see Table's minWidth).
• Secrets live in backend/.env, which is gitignored. Never commit keys.
• Generated data (data/, results/, reports/) is gitignored — never commit market data.

9. Window discovery

Answers "when during the day should this strategy trade?" by searching session_open × range_minutes × close_time instead of leaving it to a guess. backend/algo/optimize/discover.py is the engine; the builder page is the front door.

Three rounds, coarse to fine

Each round ranks by rank.add_robustness — a config scored together with its immediate grid neighbours — and recentres the next round on the winner.

Two axes per round is a correctness requirement, not a budget one. add_robustness can only find neighbours inside a complete cartesian product. Spreading a small budget over four axes leaves every config isolated, robust_<metric> silently collapses to the raw metric, and the curve-fit protection quietly stops working while still appearing to run. For the same reason the budget cap shrinks grid axes (_fit_grid) rather than sampling the expanded product.

Round 3 sweeps whichever take-profit knob is live — rr for tp_model: rr, target_mult for multi_tp. Params.signature() nulls the inactive one, so sweeping the wrong knob de-duplicates the whole round down to a single config.

Trade direction is deliberately never swept. Switching off whichever side lost money over the sample is the easiest way to manufacture an in-sample result that dies out of sample.

In-sample only, with one honest test at the end

The search never sees out-of-sample data. The single winner is run against it once, at the end, and graded by the four deployment gates (backend/algo/deployment.py, mirroring evaluateCriteria in frontend/src/lib/strategy.ts). Searching against OOS would turn the held-back window into another tuning target and destroy the only unbiased evidence there is.

The result carries plateau evidence alongside the score: plateau means every neighbouring value held up, spike means they collapsed and the number came from one lucky cell.

This is a background job, and why

One backtest over a multi-year 5-minute in-sample window costs on the order of a minute — cost is linear in bar count (~0.27 ms/bar measured on EURUSD 5min), and this host has 2 cores. So a sweep is priced in backtests, not seconds:

POST /api/discover-window/plan quotes that cost before anything runs, and discover_window re-measures the real per-backtest time on the actual data before round 1 so the logged ETA is grounded rather than extrapolated. Shortening the search window is not a shortcut: the same config scores Sharpe −0.30 over 18 months and +1.52 over 6, so a short window curve-fits to regime.

The written rationale

Install and configure (both optional):

backend/.venv/bin/pip install -e "backend[rationale]"   # or: pip install anthropic
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> backend/.env     # .env is gitignored

backend/algo/rationale.py asks Claude (claude-opus-5, structured output via messages.parse()) to explain the finished result. Claude does not choose parameters — the search does, deterministically — so selection stays reproducible, free, and immune to an API outage. The call degrades to a skip message when the SDK is missing, ANTHROPIC_API_KEY is unset, the request fails, or the model declines; nothing else in the feature depends on it.

Nightly drift

scripts/nightly_reoptimize.py re-runs the search and reports strategies whose best window has moved. It never applies a change — promoting a window has money attached, so it stays a human decision. Drift is only reported when the improvement clears the noise threshold, the window is genuinely different, and the winner still passes the gates out-of-sample; a better in-sample number that fails the gates is the search finding a nicer way to overfit.

Units live in backend/deploy/ and are not installed. To enable:

sudo cp backend/deploy/algoenjin-reoptimize.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now algoenjin-reoptimize.timer
systemctl list-timers algoenjin-reoptimize    # confirm the next elapse

The service is Nice=10 with reduced CPU/IO weight: on a 2-core box an un-niced sweep competes with the API for both cores and the UI goes sluggish for the hour it runs.

10. Ported strategy types

Besides its own ORB, the engine runs seven strategy types ported from the AndrowDeluxe/Forex-Backtesting repo — the ones its author traded live or on paper. A saved strategy carries strategy_type (absent = orb, so every older file is unchanged); its params are validated by that type's own dataclass. GET /api/strategy-types serves the catalogue, and the Strategy Builder generates each type's form from its defaults.

How a ported type runs, and why it is built this way

Each type's build_plans decides every trade in pandas — a line-by-line port of the source's own simulator — and returns TradePlans (entry bar and price, stop = 1R, exits with bar, price and size fraction). algo/strategies/replay.py then books them through backtrader: it sizes from live equity with the same risk.sizing rule as ORB, charges costs as commission, and records trades in ORB's shape, so the API, the four deployment gates and the frontend need no special case.

The split exists because the source strategies fill in two ways one Cerebro cannot mix: entries at the next bar's open and exits confirmed on, and filled at, a bar's close. With cheat_on_open and set_coc both on, coc wins and a "next open" entry silently fills at the close (verified).

runner.run_backtest dispatches on the params class, so the optimizer, walk-forward and every strategy endpoint work for all types. Plans are computed once over the full cached history (signals only look back, and indicators warm up before an OOS window starts) and cached per config; runner._window_plans cuts them to the IS/OOS window, closes trades still open at the window end at its last close (data_end), applies the type's concurrency cap, and gives a trade a fresh feed clone when the previous one on its feed exited less than a bar earlier (a close-based exit only executes on the next broker cycle — reusing the feed lost trades).

Faithfulness, and strict_causal

Every type has strict_causal: bool = True. With False, each port reproduces the source exactly on the same bars — same entry bar, exit bar, direction and R for every trade (verified per type: gold/silver 135, trend pullback 337 over 5 markets, gold ASB 189, CLS 208, CTNL 267, ORB 494/541/2225, OU 1202/320 in sizing="source_equity"). With True (the default) the source's lookahead and untradable fills are fixed, which matters a lot for some types:

• gold_asb: the liquidity filter read the entry day's own high/low; causal, it hurts OOS.
• ctnl_reversal: the ribbon gate read a 4h bar that had not closed; about a third of the edge goes.
• cls_practical: continuation entries could fill on the bar that confirmed the fractal.
• orb_ny: no lookahead, but on 5-minute bars ~70% of fill bars also touch the stop and the order inside the bar is unknowable — fill_resolution settles it with 1-minute bars.
• gold_silver / trend_pullback: the stop used ATR from the still-forming entry bar.

Costs default to what the source charged (spread_bps, or gold ASB's fixed $0.40/oz round trip); set spread_bps to override. Known gaps: CLS's rate multiplier needs TradingView 2-year yields the engine does not have (off by default; a BUND/USTBOND-only version is opt-in); OU swap is not charged; JPY conversion is static.

Data

All bars come from Dukascopy (BID, UTC) via fetch_dukascopy.py, except the OU stocks and their benchmarks (yfinance, auto_adjust=True). The source repo's author used the same feeds.

11. How this file stays current

Two mechanisms:

1. CLAUDE.md states that any change to structure, commands, ports, endpoints, routes, or conventions must update DOCS.md in the same pass.
2. A Stop hook (.claude/settings.json → .claude/hooks/check-docs.sh) compares the newest source-file mtime against DOCS.md. If source moved ahead, the hook blocks with a reminder naming the changed files, so the drift is caught before the turn ends.

The /docs page is force-dynamic and reads this file per request, so edits appear on refresh with no rebuild.

12. Changelog

Newest first. Add an entry for anything that changes how the project is structured, run, or used.

2026-10-28

• Backtest page: the per-strategy Backtest button is dark (#161c21, RUN_BTN) instead of the accent colour.
• Monitor hero shows dollars: the "$1 became" tile is now "Total made · on $100,000", the sum of trade P&L in dollars (net_usd, with cash, added to each strategy in /api/monitor).
• Always-on research: the research loop cron runs every 5 minutes (was 15) under the same flock, so a finished 50-minute run is followed within 5 minutes instead of sitting idle ~13 minutes an hour. Still pauses on results/research/PAUSE and 00:15-01:15 UTC.
• Nightly re-optimization installed: algoenjin-reoptimize.timer (02:00 UTC, system unit) runs nightly_reoptimize.py --all --budget quick --max 5 (was --deployed, which matched no strategy). It reports window drift to results/discovery/_drift.json and changes nothing.
• Favicon: frontend/src/app/favicon.ico, the same icon as www.algoenjin.com (/images/favicon.ico, 16/32/48 px). Next serves it as the site icon automatically.
• Profile menu (top-right avatar, components/layout/profile-menu.tsx): theme picker (moved from the old palette button; shared logic in lib/theme.ts) and links to Profile and the Settings sections. /settings: Appearance works; Data providers (Databento key — example format only), Trading platforms (MT5, IBKR, cTrader, Tradovate/Rithmic — nothing connects, live trading stays off), JEV, and Notifications (Slack webhook, email, WhatsApp, alert events) are placeholders — disabled fields, nothing saved or sent. /profile: placeholder. Secrets are meant to live on the server, never in the browser.
• Monitor: a paper desk (each paper strategy's paper trades / return, backtest OOS PF, last trade, live countdown to the 00:30 UTC run), a data-feed panel showing the age of each feed's newest bar (from the Parquet footer — a rewritten file is not a fresh one), and the research panel now splits ORB and ICT (count, rate, best fitness vs the gate) with the loop's pace (time since the last test, median time per test).
• Dev note: type-check the frontend on the VPS (cd frontend && npx tsc --noEmit -p .); the local scratch copy has no node_modules, so a local npx tsc checks nothing.
• Top bar status replaces the fixed "Backtest research — not live" label (read as an error): a live pill "Research running / paused / idle · N on paper" from GET /api/status, refreshed every minute, linking to the Archive's research tab. The platform still never trades real money; the tooltip says so.
• New Dashboard (/; GET /api/dashboard, algo/dashboard.py): five KPI blocks, each linking to its page — Datasets (bars, files / symbols, history span, freshness of the research feeds), Backtest (strategies, runs, pass / review / fail, best OOS PF), Optimization (configs, runs, tables, walk-forwards, best Sharpe, WF degradation), Fronttest (paper strategies, trades and return since PAPER_START, last nightly run) and Research (tests, last hour, ORB / ICT, best fitness vs the gate, holdout / promoted, running / paused). Below: pipeline activity, research fitness spread, IS-vs-OOS PF scatter per saved strategy, OOS-PF leaderboard with strategy-type counts, the paper-trading table, a feed of the latest events, a data-coverage timeline of every intraday market, the top optimization configs and the latest walk-forward. Refreshes every minute. /api/summary stays for scripts.
• Faster pages everywhere: load_bars with a start / end now reads only the Parquet row groups overlapping the window (one day of a 16-year 1min file: ~0.1 s, was ~2 s) — charts, candles and the ORB overlay; /api/datasets keeps per-file stats in results/datasets_cache.json keyed by mtime + size and warms it at API start (8.6 s -> ~5 ms, which also fixed /api/summary, 3.4 s -> 0.04 s); /api/price-series is cached until the symbol's files change; /api/recent-trades until the runs CSV changes (1.1 s -> 0.04 s). Chart animations on the dashboards are off: a background tab froze them half drawn.
• Faster Strategies page: run_store keeps the parsed strategy_runs.csv in memory, keyed by the file's path, mtime and size (any write refreshes it) and hands out copies. The strategies overview did three lookups per strategy and re-parsed the CSV on each — 3-7 s for 36 strategies; now ~10 ms server-side, same output. The monitor and Archive pages use it too.
• ICT strategy type (strategy_type: "ict", algo/strategies/ict.py, 5min, NY time): three ICT setups as strict rules — silver_bullet (FVG limit at the gap edge or midpoint inside 03-04 / 10-11 / 14-15 NY, optional minimum gap in ATR), judas (failed sweep of the Asian range in 02-05 NY or of the London range in 07-10 NY, confirmed by a close back inside or a market-structure shift) and smt (US500/US100/US30, EURUSD/GBPUSD: one market takes its London or prior-day level and the other does not). Optional premium/discount or daily-trend bias, stop buffer in ATR, R target, one trade a day, flat by the exit time. Signals on a close enter at the next bar's open; a bar touching stop and target is a stop.
• Research loop runs ICT too: half of each batch is ICT (family: "ict" in the config; ORB configs carry no family key, so existing ledger keys are unchanged). ICT mutations favour a higher target or an added bias, the counterpart of ORB's confluence. Same 60/20/20 guard and gates; promotions are saved as "Auto ICT ...".
• Archive page rebuilt (/archive, sidebar Backtest -> "Archive"; GET /api/archive, algo/archive_view.py, one read of the runs CSV, ~0.15 s): a stats dashboard — totals (research tests, ORB / ICT split, reached holdout / promoted, saved strategies, backtest runs, optimization configs), the fittest research setup, activity per day, the research fitness spread against the 1.2 gate, per-market tests / best fitness / saved strategies, OOS verdict split and an OOS-PF leaderboard — then four tabs (?tab= deep-links): Research tests (every config the loop tested, train / validation / holdout metrics; research.ledger_rows()), Strategies (every saved strategy with its latest IS / OOS / paper legs and verdict), Backtest runs (every recorded backtest and fronttest) and Optimizations (the old archive table, now components/archive/optimization-table.tsx). Sortable, filterable, CSV. Research tests are NOT saved as strategies (hundreds of losers a day would bury the Strategies list); only promotions are. /backtest/research redirects to ?tab=research; /backtest?slug=... opens that strategy's drawer. GET /api/research/ledger stays for scripts.
• Live monitor (/monitor, sidebar "Monitor"; GET /api/monitor, algo/monitor.py, cached 60 s): the paper strategies, topped up with the strongest saved OOS legs (up to six), from their latest two-leg backtest — growth of $1 (OOS compounding on the IS end), leg metrics, the last 60 trades in R, an estimated cost drag (spread + commission or the type's bps, plus swap) against gross earnings, and the OOS gate verdict — plus the research loop's ledger and the data feeds' file times. Animated: ticker tape, a pipeline graph (feeds -> bars -> engines / research loop -> strategies -> paper) with flowing edges, a replay playhead over the growth chart and the trade pulse, count-up hero numbers. Respects prefers-reduced-motion.
• Terminal theme (near-black / phosphor green, JetBrains Mono, Chakra Petch numbers, drifting dot grid and scanlines): third option on the top-bar palette button.
• algo/paper.py now holds the paper-trading list (PAPER_SLUGS, PAPER_START, the research loop's promotions); paper_forward.py and the monitor both read it.
• Dev note: after adding a new page, change globals.css content (not just its mtime) if its Tailwind classes are missing — the dev server caches the compiled CSS by content.
• Continuous research loop (algo/research.py, scripts/research_loop.py): cron starts it every 15 minutes under flock (one instance at a time), at nice 19, for up to 50 minutes; it stands still while results/research/PAUSE exists and sits out 00:15-01:15 UTC for the paper job. Each batch tests ORB configurations on XAUUSD, EURUSD, GBPUSD, US500, US100 and US30 (free nightly Dukascopy feeds): half random, half mutations of the ten fittest (biased to a larger take-profit or an added confluence). Search space: session opens per market, 5/15/30/60-min ranges, candle-close / retest / retracement entries, 1-4R targets, ATR range bands, VWAP-day / EMA 20/50/200 confluence, long / short / both, 1-2 trades a day. Overfitting guard: each market is cut 60 % train / 20 % validation / 20 % holdout by time; fitness = the worse PF of train and validation; only a config clearing PF >= 1.20, WR >= 40 % and >= 50 trades/yr on BOTH is run on its holdout, once; clearing PF >= 1.15, WR >= 38 %,

  = 40 trades/yr and a positive return there promotes it: saved as "Auto ORB ..." (IS = train + validation, OOS = holdout), risk scaled for <= 9 % drawdown, backtested, and paper-traded from the next day (results/research/paper.json, run by paper_forward.py, which now also refreshes the universe's 5min/1min bars). Ledger: results/research/ledger.jsonl. Promotion never means live trading.

• Fix: ORB positions carried past a session with no closing bar. When a day's data ended before close_time (gaps, early closes, US index CFDs in 2016-2017) the day reset cancelled the stop and target but left the position open, sometimes for months. A position still open at a new session is now closed at once (not counted against the new day's trades). GC v1 had no such trades.
• Fix: metrics of a blown account — CAGR of a negative final value raised (complex power).
• ORB: breakout-and-retest entry (entry_mode: retest): after a 5-min candle closes beyond a range level, a limit order waits at that level and fills when price trades back to it (touch fill — slightly optimistic). Stop and targets as for the other modes.
• ORB: confluence filter (confluence: vwap | ema, ema_length): longs only on bars closing above the line, shorts only below. VWAP is anchored at session_open each local day (zero tick volume counts as 1); EMA is over closes. Unset, the params signature is unchanged.
• Micro-futures commissions: NQ commission: 0.70 (MNQ), ES 0.28 (MES), alongside GC's MGC 0.18, in price units per point-value.
• Ember theme (black / orange, Work Sans): the palette button in the top bar switches between Classic and Ember; the choice is stored per browser and applied before first paint. Tokens in globals.css under :root[data-theme="ember"]; the purple accent slot becomes orange #fd5c38. Daily-bar trades show their bar date instead of a shifted NY evening time.
• Engine audit (four paper strategies): no lookahead found. Stop-vs-target on one bar resolves to the stop (ORB); close-confirmed exits filled at the close instead of the next open move R by ~0.005; GC v1's unprotected entry bar changes 4 of 760 trades. Missing costs were the real gap, fixed below.
• Broker commission and overnight swap: Instrument gained commission (round trip, price units per unit, like spread) and swap_long / swap_short (fraction of notional per year). Both engines charge commission; the replay charges swap as a cash debit per night held (UTC date changes), reported per trade as swap and included in pnl and r. GC: commission: 0.18 (micro gold MGC). OU stock CFDs: swap_long: 0.0624 (TTP). XAUUSD has no swap configured yet (needs the broker's rates).
• OU S&P 500 left paper trading: with swap its PF falls from 1.20 to about 1.06; its nightly yfinance refresh is skipped while no OU strategy paper-trades.
• ML filter in the engine: Params.ml_filter names a bundle in backend/data/models/. An ORB run with a filter first runs unfiltered, scores each trade on bars from before its fill (ml.filter.skip_days), then reruns with those sessions blocked (ml_skip feed line). Unset, it leaves the params signature unchanged.
• ORB-60min-NewYork v3 = v2 + ml_filter: orb-60min-newyork-eurusd (pooled model, see 2026-10-27). Its in-sample leg is scored by a model trained on those same trades, so ignore it; only its OOS (116 trades, PF 1.37) and fronttests count.
• EURUSD data switched to Dukascopy spot (1min + 5min, 2019 -> today). The old files were a different feed about 39 pips above spot (futures-like) and could not be extended; they are in ~/backups/eurusd-futures-feed/. All EURUSD strategies were re-run, and the IS/OOS boundary moved; earlier EURUSD runs are not comparable.
• v3 failed on spot data: rule set before running was "paper-trade v3 only if it beats v2 on the OOS leg". Spot OOS (2024-07 -> 2026-10): v2 PF 1.12 (194 trades), v3 PF 1.08 (108). The filter's futures-data gain did not carry over, so v3 is not paper-traded. v2 on spot: IS PF 1.12 with 12.9% DD (over the 10% gate).
• Strategy overlay on the trade charts: GET /api/strategy-overlay/{slug}?start&end returns, for ORB strategies, each session's range window, high/low (finest cached frame), close, first trade-through of each side, and whether the range filters let it trade. The chart draws the range box, dashed levels to the close and break dots, and the selected trade as a TradingView-style position tool (Target / Stop / Entry with R:R and result). Other strategy types return no overlay yet.
• All trade times are shown in New York time: the chart axis and crosshair (times are shifted to NY wall clock before drawing, lib/trades.toNy), the ORB labels, and the entry / exit times on the Trades page and in the strategy drawer (fmtNy). Stored trade times and the strategies' own session settings stay as they were (UTC / instrument timezone).
• Every dashboard timestamp is New York time too (stampNy, shown with an "NY" tag): strategy saves, run history, processes, the optimizer archive, window searches and the docs page. The server writes them in UTC (the VPS clock). The running-job banner now parses started_at as UTC; it used to read it as browser-local time, which skewed elapsed / ETA by the viewer's UTC offset. Date-only fields (IS/OOS windows, data ranges) are unchanged.

2026-10-27

• ML trade filter, research layer (algo/ml/, extra ml = scikit-learn): per-session features known when the opening range closes (features.py), meta-labeling helpers, year-by-year walk-forward out-of-fold scoring and a percent-risk trade re-booker (filter.py). scripts/ml_filter_gc.py tests it on ORB-15min-NewYork GC v1. Result: no usable signal (walk-forward AUC 0.52, OOS AUC 0.46; pooling spot-gold trades from 2010 gave 0.517), so no filtered strategy was saved and the engine does not apply filters yet. Rule for future filters: only score the OOS leg once the in-sample walk-forward AUC is at least 0.55. Entry-time features (features.ENTRY: breakout delay and push, momentum, VWAP distance, day range, time in session; from bars before the fill bar) on ORB-60min-NewYork v2 (EURUSD), scripts/ml_filter_entry.py: best walk-forward AUC 0.546 (gradient boosting, entry set only), just under the gate, so its OOS leg was not scored. Pooled (scripts/ml_filter_pooled.py: same rules on GBPUSD and XAUUSD before EURUSD's IS boundary, min_range at the same range percentile): EURUSD walk-forward AUC 0.562 (gradient boosting, entry set), gate passed; single OOS check AUC 0.551, PF 1.169 -> 1.371, DD 8.1% -> 3.8%, 195 -> 116 trades. Re-booked trade lists, not yet an engine run: skipping a signal is assumed to mean no trade that day.
• GC paper trading: orb-15min-newyork-gc-v1 joined the paper strategies; the nightly job appends GC 1min/5min bars from Databento (DATABENTO, about $0.005/day, $1 cap per pull). databento_source.fetch gained save=False so new bars merge into the cache instead of replacing it.
• XAUUSD history from 2010 (Dukascopy, all cached frames). This moved the 70% IS/OOS boundary of every XAUUSD strategy; runs recorded before it are not comparable.
• Paper forward-test scripts/paper_forward.py: refreshes the Dukascopy and OU data, then re-runs the paper strategies (gold/silver divergence, CTNL live cost, OU S&P 500) over the window since 2026-10-28 and records it as each one's fronttest. Meant to run daily from cron at 00:30 UTC; log in backend/logs/paper_forward.log. Do not edit those strategies' params while they paper-trade.
• Plan cache keyed on data writes (runner._data_stamp): the running API now re-plans after a data refresh instead of serving plans built on the old bars.
• Seven strategy types ported from the Forex-Backtesting repo (§10): gold_silver, trend_pullback, gold_asb, cls_practical, ctnl_reversal, orb_ny, ou_modell. Saved strategies gained strategy_type; files without it are ORB and unchanged.
• New replay layer (algo/strategies/replay.py) books precomputed trade plans, because the ported strategies need next-open entries and close fills in one run, which backtrader cannot mix.
• New data scripts fetch_dukascopy.py and fetch_yfinance.py; new instruments in instruments.yaml (§5).
• Run history (results/strategy_runs.csv) gained strategy_type and type_params (JSON) columns; ORB rows keep their per-param columns.
• Frontend: Strategy Builder has a strategy-type picker and generates the form of non-ORB types from /api/strategy-types; list views show setupBadges; the optimizer derives variables from a type's numeric/boolean params; the run-history diff reads type_params.

2026-10-14

• Window discovery (§9): a new section on the Strategy Builder searches when a strategy should trade — session_open × range_minutes × close_time — over three coarse→fine rounds, then checks the single winner once out-of-sample and grades it against the four deployment gates. New: algo/optimize/discover.py, algo/deployment.py, algo/rationale.py, scripts/discover_window.py, scripts/nightly_reoptimize.py, components/window-discovery.tsx.
• Each round sweeps exactly two axes, and the budget shrinks axes rather than sampling the expanded product. Both are correctness constraints: add_robustness only finds neighbours inside a complete cartesian product, so a decimated grid leaves every config isolated and the curve-fit protection stops working without failing — robust_<metric> just quietly equals the raw metric. Trade direction is never swept for the same reason: disabling the losing side is the cheapest way to fake an in-sample result.
• Searching is priced in backtests, not seconds, and the UI quotes the cost first. One backtest over the full EURUSD 5-minute in-sample window takes ~100s on this 2-core host, so standard is a ~70-minute job. POST /api/discover-window/plan estimates before committing; the run re-measures the real per-backtest cost before round 1 and logs a grounded ETA. Searching a shorter window is not a shortcut — the same config scores Sharpe −0.30 over 18 months and +1.52 over 6.
• Claude writes the rationale, never the parameters (claude-opus-5, structured output). The search stays deterministic and free; the write-up needs ANTHROPIC_API_KEY in backend/.env and degrades to a skip message without it.
• Nightly drift sweep reports strategies whose best window has moved and applies nothing. Units in backend/deploy/, not installed — see §9 for the enable commands.
• _split_windows moved out of api/server.py into algo/validation/splits.resolve_windows(), so the API, the saved-strategy runners and the discovery job cannot drift apart on what a strategy's IS/OOS split means. The .env reader moved from data/databento_source.py to config.load_dotenv() for the same reason.
• Fixed: a gate check with a missing profit_factor passed, because an absent key was read the same as an explicit null (which legitimately means "no losing trades, infinite PF"). Missing data now fails closed.

2026-08-06

• Published to github.com/Algoenjin/algoenjin and deployed: Vercel serves frontend/, the API stays on the VPS behind nginx at api.algoenjin.com. See §4.
• The /api/* rewrite is gone, replaced by the route handler at frontend/src/app/api/[...path]/route.ts. A rewrite cannot attach the X-API-Key header nginx requires, so the proxy had to become real code. API_ORIGIN is read exactly as before.
• Added a password gate (frontend/src/middleware.ts, HTTP Basic + signed cookie, keyed on SITE_PASSWORD). Without it a public deployment lets anyone spawn jobs on the VPS.
• The API runs under systemd as algoenjin-api.service instead of by hand in tmux.

2026-07-28

• api added to the installed package (include = ["algo*", "api*"]); --app-dir dropped from all docs and launch.json. uvicorn api.server:app now works from any directory. Previously the command only worked from engine/ and failed with ModuleNotFoundError: No module named 'api' anywhere else.
• Added this file, CLAUDE.md, the /docs route (new marked dependency), and the doc-drift Stop hook in .claude/settings.json + .claude/hooks/check-docs.sh.

2026-07-27

• Restructured engine/: web/ → frontend/, all Python assets → backend/. Zero Python source changes needed. Removed ~1.4 GB of stale artifacts.
• Rebuilt .venv and node_modules natively — both had been built on macOS and could not run on Linux.
• Fixed remote access: /api/* now proxied through Next (next.config.ts), api.ts base URL defaults to same-origin. Previously every page hung when opened from another machine.
• Mobile optimization: off-canvas nav below lg, safe-area insets, 16px inputs, ≥40px tap targets, responsive padding and card density, scroll-shadowed tables. Fixed two real bugs — a <select> forcing 728px page width, and the run-lock banner pinned at left-60.