Tech debt and version

What is missing, what was found and what was decided. The dashboard shows this file on its Tech debt page. Newest first.

Version

2026-10-15

• Merged from main (Camilo). It adds the Thesis Monitor page (Gold Thesis Monitor, docs/gold-bull-thesis.md), whose cards form the "Gold thesis" group on the Compass. It also moves Fiscal Dominance from Phase 2 to Phase 1.
• Checked after the merge: a full refresh of the 11 sections, and every page and /api/consensus answer without errors.
• Parity passes for both new sections: thesis 7/7, fiscal 12/12.
• Fixes after the merge (branch fix/integracion-merge):
  • With 12 links the top bar grows to two rows instead of overflowing (min-height).
  • A table wider than its card is cut on the right, where it scrolls, not on the left (safe center).
  • The refresh also watches data/thesis/*.csv (rglob).
• Consensus. The chart with the banks on top has one panel, without the probabilities underneath, and hovers along x. At the end of the series each zone shows its odds and a price from the last point:
• Offensive: the top of the cone.
• Neutral: the expected path.
• Defensive: the bottom of the cone.

A purple label gives the Consensus mean at 12 months. The optional mean comes with a shaded band from the 25% to the 75% of the banks. - Markets. - On the central bank chart the gold price is now drawn over the colored areas. Plotly always draws the overlaid axis on top, so the reserves moved to the base axis and gold to the overlaid one. - Every chart shows only its main axis, for cleaner screenshots. The price, the monthly flows and gold on the central bank chart are read from the hover. - The demand chart is titled "Gold demand" and the SHFE chart "SHFE positions updated: …"; neither is CFTC data. - Fixes from the agent review: - The box that adds Shadow Price bands now sits outside Plotly's fixed-height wrapper; inside it, the chart overflowed and showed scrollbars. - The browser builds the 5% and 10% bands from the 100% line. The page went back from 4.3 MB to 2.9 MB. - Band names use {b*100:g}%, so 2.5% is no longer labelled "2%". The comment warns that 25%, 40% and 100% already exist. - The skew note says the 25Δ put has a delta of −0.25. - Data health now lists Consensus and Options as readers of 2. XAU.csv, and Markets as a reader of 4. XAG.csv and 3. SI.csv (silver premiums and SI CFTC). A drift in those CSVs now warns on those pages too, including the Consensus block of the Forecast page. - Links to #consensus no longer land under the sticky top bar. - The selectors (past forecasts, charts, Consensus report) use autocomplete="off". On a reload, Firefox restored the old choice while the page drew the default chart. - Consensus counts months by calendar. With days // 30, the 3-month point of a January report came out as 2 (Jan 31 to Apr 30 is 89 days). That affected 11 reports and would have crashed the section once a January report became the latest (February 2027). Juan's cell that builds p still uses days // 30; p no longer feeds any chart. - Options. - "Smile today and a month ago" is gone. - Two 3D surfaces sit side by side. On the left, fixed: today's implied volatility by delta and tenor. On the right, picked from a selector: the 1M smile, the 2M smile or the contango by delta (2M − 1M), weekly over the last two years. - The contango surface is centered on zero, red when the curve inverts. - In the notebook the three surfaces are drawn one after the other. - Equity Crash. The page ends at "Gold drawdown scenarios". The crash-probability analysis is gone from the page and from the notebook (its section 3, cells 14-16, Juan 2026-10-15): - "Gold around the SPX trough", the base rate, the quintile table and "Where each condition is today". - The "Equity crash risk" lens on the Compass and the base-rate KPI. - The headline now gives the gold drawdown in past SPX crashes. - Old notebooks deleted. Data World and Options and Premiums and Discounts were replaced by Gold Markets and Gold Options; they remain in git history. - Forecasts by bank adds the InHouse expected path (12 months) and an InHouse row in the table. They appear only for the report that the InHouse forecast was compared with, because the forecast log only holds that release. - Forecasts by bank, past reports. The gold line no longer stops at the report date. It runs from 3 years before the report to the report's last horizon (today for recent reports), so a past report shows what the price did afterwards (Juan). - InHouse row of Forecasts by bank. It takes the expected price on each horizon date, while the banks give quarterly averages. Kept as is; point closed (Juan). - InHouse forecast with the banks on top has a selector with one forecast per month (the last 12). Each comes with the bank densities of its month's Consensus report, and past months show the price that followed. In the notebook, change archivo to pick another month. - The refresh writes the 12 monthly forecasts to cache/forecast_log/, each as of its date (cached cone, logit fitted with data up to that date). - The 12 panes take the Forecast page from about 1.5 MB to 3.1 MB. Each is drawn only when picked. - Which Consensus report goes with a forecast. The pipeline dates each report at the calendar month-end, but forecasts fall on the last trading day. Comparing exact dates gave 6 of the 12 forecasts the previous month's report (28 Aug 2026 got July). - A forecast now takes its own month's report, or the latest before it if that month's report is not out yet. The latest forecast (12 Oct) still gets September. - Forecasts by bank uses the same rule, so its InHouse row and path now appear for the last 12 reports. - Tickers. get_all_tickers() reads data/6. Tickers.csv instead of tickers.xlsx (Sheet2) in the Trading Desk library on SharePoint. The original file is untouched. - The CSV has one lowercase header row, ticker as the first column, Bloomberg #N/A errors left empty, ISO dates, and no exact duplicate rows (379 to 370). - The six ticker groups come out identical, in the same order. - It feeds Regime (IMF reserves), Markets (CFTC, SHFE) and the Phase 2 walk-forward notebook. - Fixes from the final agent review (notebook and page): - Markets: the price hover on the positioning charts lost its $ format when the right axis was hidden, because Plotly ignores tickformat on a hidden axis. It now uses hoverformat='$,.0f'. - Consensus: the labels to the right of the densities are ordered by the price they show. Before, Consensus at $4,677 sat above Neutral at $4,727. - Options: the two surfaces line up at the bottom, since the right one has the selector and its note above it. - Equity Crash: the VIX query (about 3 s per refresh) and CONDICIONES_EN only fed the removed section, so they are gone. The notebook intro marks section 3 as removed. - Gold today: the CSV drift warning no longer names the cause fixed on 2026-10-14; it says what to check instead. Data health lists data/5. Consensus.csv. - Shadow Price: the note names the bands from bandas_extra. - Docs: the README lists the Shadow Price band controls, and CLAUDE.md points to the files that exist (data/TradingView.ipynb, notebooks/Phase2/Resistencia del Oro.ipynb).

2026-10-14, evening (Juan's touch-ups)

• Forecast. Our model is called InHouse. In "InHouse vs the banks" each bank is shown as points. On the chart with the banks on top, the Consensus mean is off until turned on from the legend, and the densities are lighter. The histograms by horizon are gone. "Forecasts by bank" (long-term chart and table) can be filtered by report year and month across the 212 Consensus Economics reports since 1995 (quarterly until 2011, monthly since 2016). The coefficients, decomposition and classification tables sit in a collapsible panel under the InHouse chart, and Consensus comes after all the InHouse content.
• Markets. The central bank chart uses simple_white, with gold in orange, the world total in black and lighter colors for the groups. The monthly-flow bars have 0.3 opacity.
• Options. Each chart in the selector explains how its measure is computed.
• Shadow Price. Extra backing bands: 5% and 10% from the legend, or any other share from the box above the chart. The notebook has a bandas_extra list.
• Compass.
• The CFTC lens follows producers.
• The China premium lens shows USD and %.
• The consensus lens is called "Model vs Consensus Banks".
• The percentile bars are light green.

2026-10-14, afternoon

• Performance. The Regime backtest is drawn with WebGL: the page loads in 0.9 s instead of 5.6 s, and a zoom or range button takes 0.7 s instead of 7.8 s. The forecast selector draws each past forecast the first time it is chosen, so the landing page draws 2 charts instead of 13 (0.24 s instead of 0.52 s). The Regime slider no longer adds a grey margin of about 3 years on each side.
• Bull Cycle. The "First four years of each bull market" chart was removed from the page and from the notebook: it repeated the cumulative chart, zoomed in.
• Notebooks (waiting for Juan's review; the old notebooks stay until he approves deleting them):
• Gold Markets merges Premiums and Discounts and Data World and Options without the options part. It adds the central bank reserves chart, a smoothed premium line (EWM 63), silver premiums, positioning for silver, bitcoin, SHFE and gold demand by category, and COMEX, ETF and SHFE inventories.
• Gold Options is a mock-up of gold implied volatility: skew, smile, contango and surface.
• Gold Consensus adds forecasts by bank, histograms by horizon, the Consensus Economics forecasts drawn over the gold forecast chart, and the model's path against chosen banks.
• Gold Regime Strategy draws the backtest in two panels, as on the page.
• Pages matched to the notebooks:
• Markets now comes from Gold Markets. It has selectors for premiums, positioning and inventories, and new lenses for central bank reserves, ETF holdings, COMEX stocks and SHFE warrants.
• A new Options page comes from Gold Options.
• The Forecast page shows Gold Consensus under its main chart.
• A Tech debt page shows this file.
• Rule. Every change made on a page is also made in its notebook (see CLAUDE.md).

2026-10-14, morning

• Phase 1 dashboard: 7 sections in the Market Compass style, each one a literal port of its notebook; parity check 61/61 variables.
• TradingView CSVs rebuilt from the API: their dates had drifted 51 to 60 days. The updater now reads the raw CSV.

Open items

Data

• Consensus pipeline (data/Consensus Pipeline.ipynb). Kept as is (Juan, 2026-10-14).
• Consensus Economics asks for quarterly averages ("Nominal Quarterly Averages"), but the pipeline dates each one at the quarter end. The 3-month point of a September report is the Q4 average, centred in mid-November.
• The annual forecasts for 2028, 2029 and 2030 are dropped (25, 21 and 18 banks in September 2026). Their headers are numbers, and find_positions only keeps dates and text with a '-'.
• The long-term forecast is the 2031-2035 average, but it is placed 5 years after the report date.
• Month 0 is the XAU close at the month end (4,177 in September 2026), not the spot on the survey date (4,324 on 21 September).
• The months between survey points are pchip interpolation.
• GLD implied volatility.
• Only the 1M and 2M tenors exist in [XAU].[bronze] (deltas 10, 25, 50 and 75 since 2008), so there is no real term structure or surface. The options page stays a mock-up until 3M, 6M and 12M are loaded.
• Three days have one point printed at 171% to 513% (2012-07-19, 2026-03-13 and 2026-03-16). Gold Options drops any day where a point is more than 3 times that day's median.
• WGC demand by category (ds).
• Technology demand is not among the tickers. Gold Markets shows the gap to total demand (70 to 93 t a quarter) as "other".
• Gold Bar Fabrication and Gold Retail Investment are the same series.
• Total demand equals total supply by construction.
• SHFE warrants (SFGDTOTL Index) come in kilograms and are converted at 32.1507466 troy ounces per kg.
• Ticker catalog (data/6. Tickers.csv).
• get_all_tickers() still runs drop_duplicates() on the fields without the ticker. A ticker whose fields repeat another's is dropped: GT10 Govt, GTII10 Govt and GOLDLNPM Comdty today, none of them used.
• data/Consensus Pipeline.ipynb still reads the Consensus Economics reports from the Trading Desk folder on SharePoint.
• Security. The .env file with the database credentials is tracked in git and was pushed to origin. Rotate the credentials and remove the file from the history.

Notebook methodology (not changed; Juan decides)

• Gold Regime Strategy
• The signal is shifted twice, a 2-day lag. With one shift the maximum drawdown is -60% instead of -52%.
• avg_risk_free_rate is the T-bill on the first day of each 5-year bucket, not an average.
• The markdown says cb includes hidden Chinese purchases; the code does not add them.
• "Neutral" is both a central bank state and the default of np.select.
• The pchip interpolation of OPHNFB, IMF reserves and CPI uses later data points (look-ahead).
• Gold Forecast
• The ordered logit only runs for the last 12 report dates (fechas[-12:]). It can run for every date where there is enough history, which would give a longer track record and more past forecasts to compare with the banks. Pending; not now (Juan, 2026-10-14). The cell rebuilds the indicators for every date even though they only depend on the horizon; the dashboard builds them once per horizon, with identical results.
• forecast_corto compounds log returns as simple returns, which biases the cone about 1.5% low.
• The last price is left out of the fit.
• It uses a single bucket (d=365), and the anchor probabilities differ between cells.
• In the back-test (11 report dates, 216 trading days) the most likely zone was right 58.7% of the time, against 68.7% for always saying "Defensive".
• The expected +13.3% comes mostly from the GARCH drift; the cone median is +9.9%.
• Gold Shadow Price. The data stops 60 days after the last BOGMBASE observation. resample('D').ffill() ends at that observation, shift(60) carries the empty tail, and dropna() cuts gold there. Today the page ends on 2026-09-30, while the other pages reach 2026-10-12. Left as is for now: Juan will change the notebook, and the change then goes to the page (2026-10-15).
• Equity Crash
• Section 2 of the notebook ("Requerimiento de aguante") is not on the page since 2026-10-14, but the notebook still has it, with position placeholders far from the real position.
• The episode names for 1976 and 2011 never match.
• The data/gold/data.csv fallback does not exist.
• The crash-probability section, with its overlapping 12-month windows (about 53 independent years), was removed on 2026-10-15.
• Markets
• The US premium annualizes calendar days with /252.
• plot_cftc_positions assigns colours through list(set(...)), so they change from one kernel session to the next.
• The trading-session windows overlap.
• Fixed in Gold Markets: the y-range was the largest single series times 1.5. It clipped the stacked areas (demand by category by 28%, GC and SI slightly). It now uses the stacked total times 1.1, in the notebook and on the page. The hover now shows the full category name; Plotly used to cut it at 15 characters.
• With the plotly in the venv (plotly.py 7.1), the right axis of a chart with two y-axes aligns its ticks with the left axis and shows uneven values, such as $4,996 and $3,480. tickmode='auto' fixes it. In Markets it no longer shows, because the second axes are hidden.
• Bull Market Analysis
• dates_end is not used.
• The ffill(limit=3) trick leaves gaps where the price does not change for more than 3 days.
• The add_hline(y=1) on the drawdown chart is outside its y-range.
• Options (from Premiums and Discounts; Juan dropped this part on 2026-10-14). The "smile" and "inverse smile" were the risk reversal (correlation 1.000), option_parity compared a call and a put with different strikes, the signal could never turn green, and the rate went through the log twice. Gold Options now covers only implied volatility.

Thesis Monitor and Fiscal Dominance (Camilo; found by the merge review, 2026-10-15)

• The same metric differs between pages.
• IMF 12-month change: +358 t on Markets (sum of countries, to Sep) vs +513 t on Thesis (World 001.046, to Jul).
• Gold over 12 months: +5.0% on Gold today (calendar year) vs +1.9% on Thesis (252 trading days).
• The thesis state card overflows when the state is "Under review" or "Invalidated": the " (provisional)" scale makes it too long.
• Fiscal and the thesis disagree on C4.
• Fiscal says C4 reads a 3-point fall in its score as evidence against the thesis, but C4 is a deficit below 3.5% of GDP with interest/receipts falling.
• Interest/receipts is 19.6% on Fiscal (MTS, net) and 34% on Thesis (NIPA, gross).
• The thesis story has fixed wording. Central-bank buying is always "above the 900t line", and long Treasuries are always "falling faster than gold", whatever the data.
• An unreachable FiscalData API holds the refresh for about 12 minutes before Fiscal fails (the timeout and retries in fd()).
• Gold today > Data health leaves out the new inputs. It does not list WGC SGLDWD*Q, GTII10 or FiscalData, and "Used by" leaves out Thesis for the fix, SPX, DXY, IMF and OPHNFB.

Dashboard and infrastructure

• Tracked __pycache__/*.pyc. They change on every import and come in with every merge. Untrack them and add __pycache__/ to .gitignore.
• Refresh inside the web process. A refresh peaks at about 520 MB and leaves about 430 MB in use; memory grew 13 to 26 MB per refresh in a test of 3 refreshes. A separate process would return that memory after each refresh. Not needed yet (Juan, 2026-10-14).
• Page-only features with no notebook counterpart: curves that restart at 1 when zooming, range buttons and sliders (JavaScript), the lenses, the past-forecast selector, the collapsible panels, and the "today" line on the Bull Cycle drawdown histogram. Two features differ between page and notebook. The report selector of "Forecasts by bank" is fecha_reporte in the notebook. The box that adds any backing band on the Shadow Price page is the bandas_extra list in the notebook. The Bull Cycle notebook still draws that histogram with seaborn and its long charts without the 3,600-day limit.
• Plotly Cloud. The "Share chart" button is disabled. Check cloud.plotly.com for charts uploaded during 2026-10-14.
• Forecast log. cache/forecast_log/ holds the 12 monthly forecasts of the selector, recomputed as of each date on every refresh, plus each day's release since 2026-10-12. The record of forecasts actually published at the time starts on 2026-10-12.