I Tried My Own Context Engineering Advice. It Went Badly.

The part where I eat my words

Last week I published a post about context engineering. Five rules, clean code snippets, a checklist at the end. It read like I had this figured out.

I did not have this figured out.

What I had was a set of ideas that sounded right, and a small study-helper agent sitting in a folder that I hadn't touched in a month. So this weekend I did the obvious thing and pointed the rules at the agent. I expected a tidy "before and after" post.

Instead I spent Saturday breaking things that used to work, and Sunday figuring out why. This is that post.

What the agent does

Nothing fancy. A student types something like "I don't get chain rule," the agent picks a tool (explain, quiz, or worked example), calls it, and turns the result into a reply. There's a chat history, a small pile of notes it can search, and about eight tools. Basically the tool calling loop from a few months ago with more stuff bolted on.

It worked. Slowly, expensively, but it worked. Which is exactly the kind of thing my own post said to fix.

Break #1: The summary forgot the one thing that mattered

Rule 3 was "compact the history." Keep the last few turns, summarize the rest. I had the code already. I plugged it in and felt productive for about ten minutes.

Then I ran a longer test conversation. Early on, the student said they were in class 11 and preparing for a specific exam. About twenty turns later, after compaction kicked in, the agent started giving university-level explanations. Politely. Confidently. Completely wrong for the person asking.

I opened the summary the model had produced. It said things like "user is learning calculus" and "user found the first explanation helpful." Nice bullet points. Not one of them mentioned the class or the exam.

My summarization prompt said to preserve "constraints the user stated." Turns out the model and I disagree about what counts as a constraint. To me, "I'm in class 11" is the most important sentence in the whole conversation. To the summarizer, it was small talk.

What actually fixed it was embarrassingly simple. I stopped trusting the summary with the stuff that can't be lost, and pulled it out into its own thing:

// profile.ts
type Profile = {
  level?: string;      // "class 11", "first year undergrad"
  goal?: string;       // "JEE", "just curious"
  language?: string;   // "hindi", "english"
};
 
// this never gets summarized. it gets re-injected every turn.
const profile: Profile = {};
 
// after each user turn, ask the model one narrow question
const update = await extractProfile(userMessage);
Object.assign(profile, update);

The profile goes at the top of every request, verbatim, forever. The history can be squashed as hard as I like because the facts I can't afford to lose don't live in the history anymore.

I should have known this. My own post said "preserve identifiers and constraints." I just assumed saying it in a prompt was the same as guaranteeing it. It isn't.

Break #2: The threshold was too smart

Rule 2 was "retrieve, don't stuff." Filter search results by score, and if nothing clears the bar, tell the model nothing was found instead of feeding it junk.

Still believe that. But I picked 0.75 as the threshold because it was in my blog post, and it was in my blog post because it looked like a reasonable number.

For my notes it was way too high. A question about "integration by parts" scored around 0.68 against the note that literally explains integration by parts. So the agent, now proudly refusing to use irrelevant context, told the student it had no material on the topic. The note was right there. My filter was just confidently throwing it away.

I don't have a clever fix for this one. I logged the scores for fifty real-ish questions, looked at where the useful matches actually landed, and set the number to 0.6. That's it. The number depends on your embedding model, your chunk sizes, and how your notes are written, and there is no way to pick it from a blog post. Including mine.

The rule is still right. "Pick a threshold" was doing a lot of work in that sentence though.

Break #3: I trimmed the tool result and lost the error

Rule 4: tool results are huge, return only the fields the model needs. I went through every tool and trimmed them down. Very satisfying. Token count dropped a lot.

Then the quiz tool started failing silently. Not crashing. Just returning an empty quiz, and the agent would cheerfully say "here's a quick quiz for you!" followed by nothing.

The quiz generator had been hitting a rate limit. The full response had an error field explaining that. My trimmed version kept questions and topic and threw away everything else, including the error. So the model saw an empty list, had no idea anything went wrong, and did its best with what it had.

// before: kept only the "useful" fields
return { topic: data.topic, questions: data.questions };
 
// after: the error is a useful field
if (data.error) {
  return { topic: data.topic, error: data.error };
}
return { topic: data.topic, questions: data.questions };

Obvious in hindsight. When I was trimming, I was thinking about the happy path, and the happy path never has an error field in it.

The thing that actually helped most

Here's what I didn't expect. The single biggest improvement didn't come from any of the five rules.

It came from the logging I added in Rule 1 to count tokens, which I had put in mostly so I'd have numbers for the blog post. Once I could see the breakdown per request, everything else got easier. I could see the summary was too short. I could see retrieval was returning nothing. I could see the quiz tool result was three tokens long, which is not what a quiz looks like.

None of the actual fixes were hard. Finding them was hard, and finding them was only possible because I could see what the model was seeing.

So if I had to rewrite the original post with one rule instead of five, it would be: look at the context. Not the prompt. The whole assembled thing, per request, in a log file you actually open. Everything else follows from that.

Where this leaves me

The agent is better now. Faster, cheaper, and it remembers what class you're in. It took two days instead of the afternoon I'd planned for, and it's still not something I'd put in front of real students without more testing.

I'm not taking the original post down. The rules are fine. I just wrote them as if applying them was the easy part, and it wasn't. If you read that post and thought "this sounds simple," it does. Build it anyway. You'll find your own three breaks, and they'll teach you more than my five rules did.