[{"data":1,"prerenderedAt":197},["ShallowReactive",2],{"post-\u002Fblog\u002Fmeasuring-my-wiki":3},{"id":4,"title":5,"body":6,"cover":182,"date":183,"description":184,"draft":185,"extension":186,"meta":187,"navigation":188,"path":189,"seo":190,"stem":191,"tags":192,"__hash__":196},"blog\u002Fblog\u002Fmeasuring-my-wiki.md","How measuring my own wiki kept proving me wrong",{"type":7,"value":8,"toc":172},"minimark",[9,14,18,29,37,40,44,47,50,54,57,60,75,85,88,92,95,98,105,111,115,118,121,124,128,137,140,146,149,162,166,169],[10,11,13],"h2",{"id":12},"it-started-with-laziness","It started with laziness",[15,16,17],"p",{},"I wanted to automate my interview prep. The idea was to build a knowledge base from my notes and use it to simulate interviews, surfacing the gaps I needed to close rather than redoing the same manual review before every application.",[15,19,20,21,28],{},"So I built a wiki. The shape came from ",[22,23,27],"a",{"href":24,"rel":25},"https:\u002F\u002Fgist.github.com\u002Fkarpathy\u002F442a6bf555914893e9891c11519de94f",[26],"nofollow","Karpathy's LLM Wiki gist",". Rather than re-derive knowledge from raw sources on every query, you let the LLM incrementally build and maintain a persistent, cross-linked wiki. Raw sources stay immutable. The LLM owns the wiki layer. A schema file tells it how to behave.",[15,30,31,32,36],{},"I extended that in two directions. Every agent needed the same orientation, so the shared instructions went into a top-level ",[33,34,35],"code",{},"AGENTS.md"," that any agent reads on arrival. And because one agent doing everything was a mess, I split the work into seven single-purpose agents, each writing only where its mandate allows. Of those seven, LIBRARIAN is the one that matters for this story. It restructures notes, merges stubs, expands thin sections, and maintains cross-links. Every other agent reads what it maintains.",[15,38,39],{},"The wiki is the single source of truth. Agents compound on each other through it, never through private state.",[10,41,43],{"id":42},"_250-pages-later-the-tokens-added-up","250 pages later, the tokens added up",[15,45,46],{},"At around 250 pages, \"give the agent what it needs to know\" had quietly become \"give the agent a big slice of the wiki,\" and the token bill showed it. Time for retrieval: store the wiki as vectors, pull back only the few chunks each query needs.",[15,48,49],{},"So I built a RAG, and immediately hit the question that turned out to be the whole story: what granularity should the chunks be?",[10,51,53],{"id":52},"first-wrong-turn-thinking-the-files-were-already-my-chunks","First wrong turn: thinking the files were already my chunks",[15,55,56],{},"My notes have headers and sections, so I figured I had semantic chunks for free. One file, one chunk. Done.",[15,58,59],{},"Then I measured. My files average eight sections each, so one vector per file smears eight topics together, and a query about one of them drags back the whole file. Splitting on headers fixed that. Section-level it is.",[15,61,62,63,66,67,70,71,74],{},"The same measurement caught a different mistake before I made it. I had planned to find my biggest file and buy an embedder whose context window could swallow it. My biggest files were ",[33,64,65],{},"log.md"," at 14,501 tokens, ",[33,68,69],{},"backlog.md"," at 11,417, and ",[33,72,73],{},"index.md"," at 7,976. None of them are knowledge. They are plumbing. I nearly sized my whole embedder choice around a file that should not be in the corpus at all.",[76,77,82],"pre",{"className":78,"code":80,"language":81},[79],"language-text","ROOT: ~\u002Fwiki\nincluded files: 251   excluded: 0\ntotal tokens (content only): 310633\n\nFILE-level:    n=251  max=14501  p95=2484  median=972  mean=1237\nSECTION-level: n=1969  max=4685   p95=459   median=104  mean=157\n  oversized  (> 512): 72 (3%)   -> split candidates\n  undersized (\u003C 64):  483 (24%) -> merge candidates\n  sections\u002Ffile: median=8 max=52\n\nbiggest sections:\n  4685  wiki\u002Fbacklog.md\n  4479  wiki\u002Fbacklog.md\n  2933  wiki\u002Fdecisions\u002Fledger.md\n  2146  wiki\u002Fdebriefs\u002F2026-05-27-[company]-[person].md\n  1902  wiki\u002Findex.md\n","text",[33,83,80],{"__ignoreMap":84},"",[15,86,87],{},"Lesson: scope the corpus before you measure it, not after.",[10,89,91],{"id":90},"the-corpus-not-the-chunk-size","The corpus, not the chunk size",[15,93,94],{},"So I dropped the plumbing and measured content only. The corpus itself drives every later decision.",[15,96,97],{},"Even without plumbing, my biggest sections were not the material I cared about. They were interview transcripts and debriefs. My wiki keeps very different kinds of writing side by side: interview prep, debriefs, AWS notes. All useful to me, all shaped differently, and one chunking rule across the lot is a compromise everywhere.",[15,99,100,101,104],{},"The tail surprised me too. I had braced for chunks too big for the embedder. The real problem was the reverse. 27% of my sections are under 64 tokens, too thin to retrieve well on their own. (The 64\u002F512 boundaries are rules of thumb for the class of embedders I'm evaluating, small models in the ",[33,102,103],{},"text-embedding-3-small"," range, where the sweet spot sits between roughly 50 and 512 tokens. I haven't locked a final model yet; the point of measuring first is to let the data inform that choice rather than the reverse.)",[76,106,109],{"className":107,"code":108,"language":81},[79],"ROOT: ~\u002Fwiki\nincluded files: 227   excluded: 24 (index.md x20, backlog.md, ledger.md,\n                                     2026-05-27-[company]-[person].md, log.md)\ntotal tokens (content only): 229689\n\nFILE-level:    n=227  max=4647  p95=2028  median=889  mean=1011\nSECTION-level: n=1686 max=1369  p95=417   median=96   mean=136\n  oversized  (> 512): 48 (2%)   -> split candidates\n  undersized (\u003C 64):  458 (27%) -> merge candidates\n  sections\u002Ffile: median=7 max=15\n\nbiggest sections:\n  1369  wiki\u002Finterviews\u002F2026-05-26-[company].md\n  1156  wiki\u002Finterviews\u002F2026-05-26-[company].md\n  1155  wiki\u002Finterviews\u002F2026-05-28-dry-run.md\n  1006  wiki\u002Fsources\u002F[company]\u002Fmotivation_raw.md\n   988  wiki\u002Finterviews\u002F2026-05-26-[company]-[role].md\n",[33,110,108],{"__ignoreMap":84},[10,112,114],{"id":113},"what-this-taught-me","What this taught me",[15,116,117],{},"Some questions you look up, some you measure. \"Does this chunk fit the embedder?\" is a lookup, the same answer for everyone. \"What granularity, and which embedder, work on my notes?\" depends entirely on my data. My first plan treated a measure-question as a lookup.",[15,119,120],{},"Scope beats mechanics. Twice, \"what is my biggest chunk?\" resolved to a file that should not be indexed at all. What goes in the corpus moved my results far more than how I sliced it, and that is a judgment about what the system is for, not a number.",[15,122,123],{},"The leverage is in the source. Retrieval chunks are derived from how I write notes, and LIBRARIAN edits that same structure directly. Get the notes right, small atomic sections with clean headers, and both jobs get easier at once. The clever chunker I thought I needed is mostly downstream of how I write a single note.",[10,125,127],{"id":126},"scoping-down-to-what-i-actually-retrieve","Scoping down to what I actually retrieve",[15,129,130,131,136],{},"Mid-puzzle, I found a well-kept external repo, ",[22,132,135],{"href":133,"rel":134},"https:\u002F\u002Fgithub.com\u002Fartreimus\u002Fnotes-aws-machine-learning",[26],"artreimus\u002Fnotes-aws-machine-learning",", that already maps the full MLA-C01 syllabus and is actively maintained against current AWS docs. It has no licence, which rules out folding it into my wiki. That turned out to clarify things rather than block them.",[15,138,139],{},"Narrowing to gaps means most of what I might have indexed was never mine to index. I measured the concepts-and-entities slice with everything else excluded. 95 files in, 156 out.",[76,141,144],{"className":142,"code":143,"language":81},[79],"ROOT: ~\u002Fwiki\nincluded files: 95   excluded: 156\ntotal tokens (content only): 92769\n\nFILE-level:    n=95   max=3205  p95=2028  median=879  mean=976\nSECTION-level: n=801  max=832   p95=298   median=91   mean=115\n  oversized  (> 512): 7  (0%)  -> split candidates\n  undersized (\u003C 64):  190 (23%) -> merge candidates\n  sections\u002Ffile: median=8  max=15\n\nbiggest sections:\n   832  wiki\u002Fconcepts\u002FAI Coding Agents in Enterprise.md\n   631  wiki\u002Fconcepts\u002FTokenization Strategies.md\n   619  wiki\u002Fconcepts\u002FLLMOps.md\n   593  wiki\u002Fconcepts\u002FVector Databases and RAG Architecture.md\n   590  wiki\u002Fconcepts\u002FTokenization Strategies.md\n",[33,145,143],{"__ignoreMap":84},[15,147,148],{},"The scoping paid off cleanly:",[150,151,152,156,159],"ol",{},[153,154,155],"li",{},"Tokens fell from 229k to 93k, not from harder chunking but because 156 files never belonged here. More were excluded than included, most of them operational artefacts.",[153,157,158],{},"The oversized problem nearly vanished: 48 sections down to 7, the largest a comfortable 832 tokens. The files driving my chunk-size anxiety were transcripts and application notes, not the concept entries I actually retrieve.",[153,160,161],{},"The undersized tail held at 23%. That one doesn't bow to scoping. It lives in the concept notes, short stubs that need merging. Mechanical, bounded, and mine to do.",[10,163,165],{"id":164},"whats-next","What's next",[15,167,168],{},"The merge step. 190 sections under 64 tokens need a look. Some merge cleanly, some are stubs to expand, a few are fine as they are.",[15,170,171],{},"After that, the corpus is finally homogeneous. All 95 files are concept entries written to the same schema. The question is whether a clean single-domain index over these entries retrieves well on its own, without routing or clustering. My prediction: it will. If the next round of measurements proves me wrong, that will be interesting too.",{"title":84,"searchDepth":173,"depth":173,"links":174},2,[175,176,177,178,179,180,181],{"id":12,"depth":173,"text":13},{"id":42,"depth":173,"text":43},{"id":52,"depth":173,"text":53},{"id":90,"depth":173,"text":91},{"id":113,"depth":173,"text":114},{"id":126,"depth":173,"text":127},{"id":164,"depth":173,"text":165},null,"2026-06-05","Building a RAG over 250 pages of interview notes, and watching my assumptions die one measurement at a time.",false,"md",{},true,"\u002Fblog\u002Fmeasuring-my-wiki",{"title":5,"description":184},"blog\u002Fmeasuring-my-wiki",[193,194,195],"rag","embeddings","chunking","TnDG6-358fFN-V4RwhsaMl10jgKgiy2EnRdQ-KWVxG8",1781020791608]