Context#

rkb asked Laya "Does this lesson help with this problem?" for each candidate. On the harder eval set (tests/fixtures/eval, 60 lessons with near misses, 2026-09-27) Laya ranked worse than BM25: recall@5 0.86 and MRR 0.62 against 0.93 and 0.82. It rated general lessons, such as "never commit directly on main", as helpful for many questions in their topic and put them above the lesson that answers.

Decision#

The question is "Is this lesson about the same problem as the user describes?". The state is unchanged: problem: <query>, then lesson: with the title, lead section, Cause and Fix or Steps. tests/fixtures/laya/ref.json is made with spikes/laya/ref.py using the same question.

Why#

On the first 52 queries it gave recall@5 0.95 and MRR 0.80, and on a second batch of 72 queries written afterwards 0.92 and 0.69 (the old question 0.90 and 0.65, BM25 0.87 and 0.68). Recall at 0.85 / 0.05 added 3 wrong lessons on the second batch instead of 9. On 15 questions against the real lessons it put the right lesson first 8 of 10 times (old 7), and the top score for questions no lesson answers fell from 0.86 to 0.52.

Rejected options#

• "Does this lesson solve this exact problem?": worse than the old question (recall@5 0.83, 6 wrong adds on the first batch).
• Keeping the old question with stricter recall thresholds: ranking itself stayed worse than BM25.