Field Notes

meta

Todo

StatusStableUpdatedSupportsnotes

Open tasks and design questions for the wiki system itself. Subject-matter research belongs in question notes, which Index and just query "type=question" enumerate.

A 2026-07-25 design session worked through the architecture items and recorded conclusions inline below, so the reasoning survives without needing the conversation. Two findings reordered the list. The frontend assets live inside bin/wiki.py as string literals, which is the concrete cause of the recurring sense that agents work in two systems at once. And dist/wiki-data.js is 1.6MB against 1.86MB of source Markdown, because its search field stores each note’s visible prose, so what the build calls an index is very nearly a second copy of the vault.

Architecture

The static build’s ceiling is set by the eagerly loaded index, not by note count and not by how the client is built. Scale is a question about where rendering happens; whether navigation is client-side is a separate question about the state model, and it does not change how much data reaches the browser. At the measured 4.5KB per note, the current design stays comfortable to roughly 1,000–1,500 notes and becomes a problem past that. Replacing the stored-prose field with a real inverted index, and splitting metadata from bodies, moves the ceiling well past 10,000 without giving up anything. Past that, raising it further means chunking the index, which forfeits offline search, because a browser reading from file:// can inject a script but cannot fetch. Reading the vault from disk is therefore a property with a scale limit, not one the architecture preserves indefinitely.

  • Extract the CSS and JS from bin/wiki.py into real files under share/assets/, read at build time and cached. The first pass took the two page-wide assets and left the force-graph, Mermaid, and Vega scripts behind, so layout was still over a thousand lines; a second pass took those three as well, bringing the function to 168 lines and the module from 5,569 to 3,842. What remains inside layout is the HTML page template, an f-string interpolating about a dozen Python values, which is structure rather than a foreign language. Every rendered page is byte-identical to the previous build. The literals had been escaped for Python, so nine double backslashes in the search tokenizer became single ones on the way out; searching övergreppen and overgreppen both return four notes, which exercises the diacritic-folding expression that carried them. node --check now validates the script, which was not possible while it was a string.
  • Split bin/wiki.py into modules along its existing seams: note and frontmatter I/O, link graph, search, render, lint, commands. Do this after the asset extraction, which removes most of the bulk.
  • Fix masked_body_lines to mask inline code spans, not just fenced blocks, so a note can discuss the syntax it documents. Span contents are blanked rather than deleted to keep column offsets valid. Dataview’s inline query form is itself backtick-delimited, so that check and the fenced-block check read the unmasked line. wanted_links turned out to scan the body directly rather than the masked lines, so quoted and fenced wikilinks were entering the research queue; it now scans the masked lines too.
  • Make page_headings fence-aware. It line-scanned the raw body, so a commented line inside a fenced block became a phantom heading, and because decorate_headings pairs that list with the rendered tags by position, one phantom shifted every later heading’s copy link onto the wrong heading. No note in the vault triggered it, so the fix is latent rather than user-visible. The structural fix below is still worth doing.
  • Move heading decoration into WikiTreeprocessor. decorate_headings regex-matches heading tags out of rendered HTML and zips them against a separately parsed heading list by position, so two independent parses have to stay in lockstep or a heading silently gets another heading’s copy link. The treeprocessor already exists and receives the real elements with their real identifiers, in order.
  • Consider a fenced-code-block formatter hook for the Mermaid and Vega fences, which are currently rewritten by regex over rendered HTML. Lower priority than the heading defect, since the substitution is contained.
  • Reconsider the Markdown library only if Obsidian and the wiki are observed to render the same source differently. python-markdown is not CommonMark-compliant and Obsidian roughly is, so nested lists, emphasis boundaries, and raw HTML blocks are where any divergence would show up. markdown-it-py would be the replacement, for CommonMark compliance and a token stream that transforms more cleanly than ElementTree, but a rewrite of the render path is not worth doing on its own. The current extension use is otherwise sound: the abbreviation glossary, the preprocessor and treeprocessor registration, and the callout markdown="1" reprocessing all use the real library API rather than working around it.
  • Scope Playwright as an optional dependency so lint, build, and test stop resolving a browser automation library none of them can reach. It moved to a download dependency group, and the default environment went from fifteen packages to eleven.
  • Fix the failing round-trip test for Securing schools, protecting minds. The note was not at fault: the YAML writer never set a width, so ruamel’s 80-column default folded its PRISMA-ScR expansion onto a second line and just fix would have reformatted the note on any pass. Setting the width fixes every long frontmatter value at once, rather than shortening one expansion and leaving the next long one to hit the same edge.
  • Replace black with ruff and wire it into just lint and just fix. black was a dev dependency and a config block that no recipe ever invoked. Ruff formats near-identically, so adopting it cost no churn, and its linter immediately found a real defect black could not: bin/wiki.py used a parenthesis-free multiple-exception except, which is Python 3.14 syntax, while the project declares support from 3.11. The file failed to parse on 3.11, 3.12, and 3.13 and only worked because the tooling pins a managed 3.14. The black config had target-version = ["py314"], which is how the gap stayed invisible; the ruff target now tracks requires-python.
  • Move the structural data out of wiki-data.js into the pages that need it. Of the ten fields it publishes, only the prose field is about search; the rest are structure, and the browser recomputes the whole note graph and the tag co-occurrence graph from them on every page load, from data the build already held in memory and already derives for just links and just clusters. Backlinks, the per-note neighborhood graph, and tag co-occurrence are all build-time facts. Baking them into each page makes a note work without JavaScript, and leaves search as the only thing needing a global payload. The blob exists because one global file sidesteps build-time invalidation, which was a fair trade at 408 notes and stops being one at 1.6MB. The dependency set is small and computable in the pass that already builds the edge set: a note’s page depends on its own content, the titles it links to, and the notes linking back to it.
  • Give the whole-vault graph its own script tag instead of letting every page carry its data. The graph lives on one page, so this needs no on-demand loading and costs nothing offline.
  • Load the search payload on demand rather than on every page, by injecting a script element rather than fetching. A reader who never searches should not pay for the index, but fetch against file:// is CORS-blocked, so the deferred file has to stay a script that assigns to a global rather than a JSON document. That keeps offline search working at the cost of an awkward loader.
  • Replace the prose field in the search index with an inverted term index. Whether to chunk it is a fork, not a detail: chunking means fetching the postings for whatever terms were typed, which script injection cannot do responsively for an unbounded set of chunks, so a chunked index and offline search are mutually exclusive. Below the size where the index ships whole, keep it whole. Decide the fork when the measured index approaches that size, and record which property was chosen.
  • Emit a metadata tier with a short excerpt per note, separate from bodies, to serve the sidebar, link resolution, and hover previews without shipping full prose.
  • Decide between a single-page app and client-side navigation over prerendered pages. Either one fixes the problem worth fixing, which is that cross-note UI state such as sidebar scroll has to be rehydrated from localStorage on every load and flickers when it is. Two axes were conflated in the 2026-07-25 session and are worth keeping apart: where rendering happens decides scale, while how navigation works decides the state model, and only the first bears on how much data reaches the browser. The case for prerendered pages is that the build already does whole-vault work, so an app either duplicates it in JavaScript or consumes prerendered fragments and becomes the same thing, and that notes stay independently addressable documents. The case for the app is a cleaner state model. The deciding question is whether a note should remain shareable by link, indexable, and readable with JavaScript off. Note that this decision and the build-time structural data above pull in opposite directions: baking structure into pages makes each note more self-sufficient, while an app wants the global data resident to render anything, so doing the first well raises the cost of the second.
  • Decide whether to port bin/wiki.py to a Node build script. This is a question about the build tool’s language, not about where rendering happens: a Node build reads src/, resolves links across the whole vault, converts math ahead of time, and writes dist/, exactly as the Python one does. The case for it is one language instead of two and genuine code sharing, since the wikilink parser, the heading-slug rules, and the search scoring currently exist as a Python implementation and a JavaScript one that have to agree. The case against is mostly latex2mathml, which has no clean JavaScript equivalent: KaTeX needs its own stylesheet and does not emit standalone MathML, MathJax is heavy, and temml is less proven, so every equation in the vault would need re-verifying. Round-trip-safe frontmatter and the existing tests would also need reproducing. Try the asset extraction first. The complaint that motivates this is largely that the stylesheet and the browser script live inside the Python as string literals, and moving the structural data to build time shrinks the client further, which weakens the code-sharing argument on both ends.
  • Make the whole-vault graph neighborhood-scoped rather than complete. A force simulation stops being interactive well before 10,000 nodes, and the ontology proposals below increase link density deliberately.
  • Consider moving cluster graphs to index pages for each cluster.
  • System for building and publishing a subset of the wiki.
  • Add a git pre-commit hook that stamps updated for staged notes. Revisits the closed answer below. That decision settled on stamping during just fix, but the --fix path runs the mechanical fixes and stops short of frontmatter, so neither the closed decision nor any current command does what the decision claims, and the field has already drifted behind the actual edit date on notes touched without running just fix. A pre-commit hook fires on every commit, so the field is current at the moment the change goes in. Uncommitted edits still have to be stamped by hand, so decide whether just fix should also stamp as a fallback for in-progress work, and whether the hook belongs in .git/hooks/ or is checked in under share/ and wired by just setup-git.

Settled in the 2026-07-25 session

  • Should rendering move to the browser, with the build reduced to emitting data? No, but not because it cannot work. The browser already does whole-vault work: the note graph and the tag co-occurrence graph are both computed client-side over every note on each page load. What that demonstrates is the cost, not the impossibility. Wikilink resolution, heading maps, and graph markers would need the whole vault resident to render one page, which is the expense the build-time structural data task above exists to stop paying. Abbreviations and callouts are not whole-vault work at all: abbreviations come from the note’s own frontmatter and callouts are line-local. The argument that holds on its own is the math: converting to MathML at build time is why a note needs no math library at view time.
  • Would server-side rendering scale better? Yes, without limit, but nothing on the roadmap needs it. A tiered index reaches the projected scale while keeping the zero-operations deploy, and keeps offline reading for as long as the index ships whole. Chunking would raise the ceiling further but costs offline search either way, so past that point the question is which property to keep, not which architecture to run. Revisit only if a live global query becomes necessary, which is the use case the Dataview exclusion already declines.
  • Is updated in the frontmatter necessary when Git and mtime both exist? Keep the field, stop maintaining it by hand. Mtime does not survive clone, checkout, or worktree operations, and per-file Git calls are too slow for the renderer. Stamping it during just fix is the fix.

Renderer and reading experience

  • Redesign against mockups. Shipped 2026-08-01 and 2026-08-02 from the D prototype in Wiki reading interface prototypes; Reading interface redesign proposals records what was adopted. The sidebar listing every note is gone, replaced by a summoned finder with facets, a landmark ribbon, a proportional section spine that drives the running head, sidenotes in the margin, a facts row, and a tabbed tail whose connections group by kind.
  • Verify the redesign below 900 pixels. The ribbon becomes a bottom bar and the spine turns horizontal, and neither has been rendered at that width.
  • Verify the local graph refits when its tab is first shown. A graph built inside a hidden panel measures zero; the fix is in graph.js but has only been read, not seen.
  • Cover the scroll wiring with a test. Three regressions in the redesign — a dropped updateProgress, a stale h1 lookup, and grid placement that collapsed the article — all passed the suite, because nothing asserts that the handler is attached or that its selectors still resolve.
  • Decide the header’s remaining mockup differences: whether to keep tags in it, and whether to add the action row (copy, provenance, argument ledger) the prototype carries.
  • Change how section wikilinks render, both local same-page sections and page-plus-section links. [Graphs <span class="link-section">§</span> Child safety cluster](Graphs.html#child-safety-cluster) now renders as Graphs § Child safety cluster, and a same-page section link as § Section, which is the ordinary cross-reference shape. A # is a URL fragment marker rather than a typographic one, and it set two names solid in the middle of a sentence. An em dash and a parenthesis were both considered and rejected: each carries sentence-level meaning, so mid-clause the reader parses it as punctuation of the sentence rather than as structure inside the link. The mark sits in a .link-section span and is dimmed against the link colour, so the two names carry the label.
  • Preview pages and footnotes on hover. Cheap once the metadata tier carries excerpts. The prototype’s peek popovers are the model.
  • Add support for various custom checkboxes?
  • Add syntax highlighting for code blocks.
  • Enforce six spaces of indent for task lists? Make rendered wiki align text similarly?
  • Breadcrumbs tracing each note’s linked path from the Index. The type mark replaced the Library / breadcrumb, which named no real parent in a flat vault; a linked path would be a different thing.
  • Allow the rendered title to differ from the note name. The link resolver never read the H1: build_resolver keys on the filename stem and aliases, and Note.title returns the stem, so the identity rule was enforced by lint and consumed by nothing. The H1 was also redundant on both surfaces, since Obsidian shows the filename as a header and the wiki rendered an H1 from the body, which is why a note displayed its title twice. Titles now live in frontmatter as title, with an optional subtitle, and the body starts at ##. rumdl enforces the choice with no custom rule: it treats frontmatter title as a top-level heading, so MD041 level = 2 rejects a body H1 and MD025 rejects carrying both. Wikilinks still target the filename, as Obsidian and Markdown Oxide require, so Note.title stays the stem and no link text changed. title and subtitle are indexed for search so a real title stays findable from a shortened filename. All 413 notes migrated, and just new and just rename now take a free-form title and derive the stem, so the shortening that lost information is no longer done by hand.
  • Repair the note names the old rule deformed, and backfill title and subtitle where the real title was lost. Fourteen notes were renamed and the titles restored on 22: works truncated at their colon, comma splices standing in for a colon, em dashes flattened to a spaced hyphen, and citation identifiers stripped of the punctuation that made them citations, so SOU 2024 75 and C-136 17 are SOU 2024:75 and C-136/17 again in their titles and close to SOU 2024-75 and C-136-17 in their handles. Source notes were worst affected, because they name external works whose titles are not ours to choose. cmd_rename was also quoting updated, which it had done since long before this pass; 15 notes carried a quoted date and no longer do.
  • Split the source notes that bundle unrelated issuers. A bundle is legitimate when several artifacts document one subject at one moment, as Redact.dev official product and policy pages does for one vendor’s product, policy, terms, and pricing; splitting those would create four notes that co-occur in every citation using them. It stops being legitimate when the artifacts share only a topic. Swedish ecommerce compliance guidance holds ten pages from eight issuers spanning consumer law, VAT, packaging, radio jammers, knives, and chemicals, so a note citing it for a VAT claim points a reader at nine documents that do not support it. Its sources list also runs one entry ahead of its body links. Privacy gear market scan (14 artifacts, 14 hosts), Cryptocurrency protocol documentation (6 and 6), and Swedish privacy voucher compliance guidance (5 and 4) are the other candidates. Multi-host alone is not the signal: a paper plus its DOI landing page is two hosts and one work.
  • Check the eight source notes with no local artifact. At least one is legitimate: Securing schools, protecting minds records that PMC fronts the PDF with a proof-of-work challenge, which is the fallback AGENTS.md prescribes. No source note has a broken artifact link, so this is a gap in capture rather than drift. Titles and subtitles are now repaired on 22 notes, and all 17 question titles carry their question mark again, so what remains is the filenames. A vault-wide audit flags 20 stems: four comma splices, two flattened em dashes, six citation identifiers, and eight source handles that truncate a work’s title at its colon. Each rename rewrites wikilinks across the vault and leaves the old stem as an alias, so propose the list before executing it and drop the aliases that are only truncations.
  • Record in AGENTS.md that a title and its handle refer to the same subject but need not be the same string. A first pass assumed they should differ only in punctuation, which is right for concept notes and wrong everywhere else: a question note wants a nominal handle and an interrogative title, and a source note wants a short handle and the external work’s full title. The failure worth naming is not divergence but drift, where the title names a different subject than the handle, so just search and the rendered page disagree about the note. That is not mechanically checkable, so state the convention and add no lint rule: a stem-substring check would reject the good cases.

Ontology redesign

Ontology redesign proposals records the 2026-07-23 design session. Nothing there is adopted, and its open decisions stay in that note. The tasks here are the ones with scheduling consequences.

  • Run the extraction pilot by hand on the evidence-law cluster before writing any skill, and encode the decisions the pilot actually produces.
  • Decide the type-to-kind swap early if it is happening at all. The sweep touches frontmatter, lint, queries, templates, tests, the skills, and agent memory files, and its cost is linear in note count and instruction surfaces. Stale type= references in skills fail silently rather than under lint, so the sweep must cover them explicitly.
  • Cost the ratchet pass against re-reading rate, not note count. Current syntheses carry note-level provenance with little claim-level mapping, so extraction is a claim-attribution pass first and a moving-citations pass second.
  • Write a distill-syntheses skill, after the pilot.
  • Consider adding a description, abstract, or summary field to frontmatter.

Agent workflow

  • Add a workflow for searching for news from e.g. the past week, relevant to topics covered by the wiki, and integrate interesting sources. This is the only item that grows the vault rather than maintaining it.
  • Research claims against F-Droid’s security model more deeply. The claim is that F-Droid signs nearly every app on the main repo, with outdated build infra and poor moderation, creating serious security issues. Done 2026-07-31. All nine listed sources were read; the blocked ones were recovered through the Wayback Machine (the GrapheneOS tweet) and a crawler view (the forum thread). archive.ph/j7qql turned out to be fdroid-website MR 834, the July 2022 permissions-blog discussion where the critique’s misleading-permissions point was conceded in substance and the F-Droid–GrapheneOS feud escalated; it is imported as fdroid-website mr 834. Findings: the certificate-pinning claim is real and worse than described — six demonstrated AllowedAPKSigningKeys bypasses over 2023–2025, fixed in fdroidserver 2.3.5, root cause unreworked through 2.4.4 — but never observed exploited, with new installs of Binaries: apps as the blast radius. “Signs nearly every app” is down to about two thirds and falling. “Outdated build infra” is a recurring-lag pattern (AGP 8.12 blocked dozens of apps in mid-2025) whose named instance was retired in 2025. “Poor moderation” resolves to automated scanning rather than review, in both directions: a false-positive uninstall recommendation (Shattered Pixel) and a policy-violating updater shipped for six months (WireGuard). The one realized supply-chain incident (Nextcloud News) was dependency poisoning that left source correspondence intact. New source notes: fdroidserver pinning bypass disclosures, wireguard inclusion policy violation, Malware in F-Droid build of Nextcloud News App; synthesis landed in F-Droid, Android app distribution trust models, and PrivSec.dev on F-Droid security issues. Sources: https://discuss.grapheneos.org/d/18731-f-droid-vulnerability-allows-bypassing-certificate-pinning/21, https://xcancel.com/GrapheneOS/status/1883895255142932816#m, https://archive.ph/j7qql#note_1025646994, https://privsec.dev/posts/android/f-droid-security-issues/, https://codeberg.org/ironfox-oss/bugs/issues/7, https://www.openwall.com/lists/oss-security/2024/04/08/8, https://github.com/CatimaLoyalty/Android/issues/2608, https://gitlab.com/fdroid/admin/-/issues/593, https://github.com/00-Evan/shattered-pixel-dungeon/issues/1394.
  • Research Rust and secure Android app development.
  • Research setting up secure sandboxes for AI agents.
  • Decide whether to reserve question notes for unresolved questions that arise during research, with ideas for new research topics instead placed in the todo, or if the latter should also get question notes.
  • Create a skill for assisting with writing essay notes, where the point is understanding the user’s idea rather than doing any deep research or ensuring factuality. Agents should ask follow-up questions for anything the user didn’t sufficiently specify, or for gaps in the general idea, that the agent needs for a complete write-up, or to explicitly note the gap. They should reference existing notes where relevant, and create redlinks for anything that would make for a useful note to add, and they should create question notes for research that would enable a grounded synthesis to evaluate the essay’s idea.
  • Make agent-written notes use richer markup and more redlinks. The prose is lean: most pages are paragraphs with the occasional list, and the structural forms the wiki supports (tables, Mermaid diagrams, callouts, block quotes, Vega-Lite charts) appear in a small minority of notes. The gap is execution, not instruction. The renderer supports them, the agent prompt advertises them, and AGENTS.md devotes a section to “Choose structural forms by meaning.” Treat that list as active choices on every pass. Redlinks are the same kind of underuse. AGENTS.md and the agent prompt say to link independently nameable subjects liberally, and just wanted reports a queue of unresolved links that surrounding prose could have named without any new research. Both are habits, not gaps in the rule, and both are observable in notes written this year.
  • Consider creating subagents or skills for post-edit review.
  • Make a skill for producing reports based on the knowledge base?
  • Define what makes a source worth preserving under src/sources/. AGENTS.md says “worth keeping” without saying what that means, which is the actual gap behind the download, link, and footnote question: those three are not alternatives. Download is durability, an external link is reachability, and a footnote is claim-level attribution, and one source often warrants all three.
  • Tighten the file-naming rules in AGENTS.md. The filename is a handle, not the title: short, typeable, Windows-safe, and unambiguous, and not required to reproduce the work’s title now that title carries that. The rules name the substitutions to stop making: no comma standing in for a colon, no em dash flattened to a spaced hyphen, no diacritics dropped, since ä, ö, and å are legal and already appear in 27 names, and no citation identifier with its punctuation punched out. stem_for_title now implements this, so the rule is executable rather than advisory.
  • Decide where per-note provenance lives, and whether that needs a frontmatter field. The gap appeared on 2026-07-28. Desktop operating system security comparison rests on project self-documentation because no independent evaluation exists, and establishing that took a scholarly sweep across several indexes and query framings. The note carried the method for an hour before it was cut as process narration, which was right for the reader and lost the thing that makes the absence checkable: which engines were searched, and what was excluded. That detail now sits in Log.md, which is chronological and not indexed by note. A future agent doubting the absence has no route from the claim to the evidence for it. Before adding a field, settle three things. Whether Log.md backlinks already close the gap: its entries wikilink the notes they touch, but it links most of the vault, so the backlink may be present and useless. Whether the field would render, since anything appearing in the page recreates the problem that motivated the cut. And what the field is actually for, because notes is the least specific name available in a wiki made of notes, and a junk drawer is the predictable outcome: the candidates are a dated record of what was checked and not found, which is narrow enough to name checked or verified, versus general editorial commentary, which is not. The bar is the one AGENTS.md already sets after the subtype episode: propose a field only with evidence that some class of note needs different handling. One instance is not that evidence. Collect the cases first — negative findings are the likeliest recurring class, since What is the production false-positive rate of automated CSAM detection is the same shape and solved it by being a question note.
  • Add a convention for an inbox type dir?
  • Reuse the Smart Connections embeddings? Scope this before committing: reading a plugin’s private vector cache carries the same portability problem as Dataview.

Obsidian feature scope

The 2026-07-25 session replaced the blanket exclusion with a narrower and more defensible rule: a note must stay readable as Markdown and renderable without the plugin that produced it. Client-side JS in the static renderer makes more of these viable than the original framing assumed.

  • Record the scope rule in AGENTS.md so the question stops recurring.
  • Decide on Canvas and Kanban. Both pass the rule. Canvas is a documented JSON sidecar, and the Kanban board format is standard Markdown lists with a small metadata comment.
  • Decide on task due dates and recurring tasks. These are plain text inside a standard task line, so they need no plugin to read and break no invariant.
  • Should Dataview be supported? No. Its queries evaluate against a live index, so a static build would have to reimplement the query language, and notes carrying DQL stop being readable as prose. This is the case that earns the constraint.
  • Should Excalidraw be supported? Not now. The scene is embedded JSON and could render in principle, but the library needed to do it is heavy, and Mermaid already covers diagrams and mind maps.

Done

  • What happens to a question note once it is answered?
  • Scan vault for further potential question notes to create
  • Did the commit “Queue the pornography cluster’s missing counter-evidence as questions” miss the point?
  • run /maintain-tags
  • run /audit-structure
  • Per-note link graphs?
  • Use the Git trailers to attribute each AI author in the rendered wiki, perhaps by percentage of contribution? Might be useful for discovering which AI’s writing style one likes best.
  • Document the wiki architecture and design choices as notes in the wiki itself

This note cites no sources of its own.

Working out connections…