I learned a lot during the Claude training about using Claude Code in development and how to make production-grade AI products which make use of the API. Distilling what I had learned down to first principles in the days before the exam, I realised that many of these principles applied to any interaction with an LLM and once I had got the exam out of the way, I decided to investigate this.

I set myself the challenge of applying what I had learnt in the exam to more everyday uses of AI, interacting with a chatbot, whether that’s Claude or ChatGPT. Here I’m going to describe a few things I found that worked well, that I didn’t know or do before.

Managing context

When interacting with an LLM, I had long considered the initial context I was giving it, ensuring that it had enough to understand what I was asking, but, until studying for the exam, I really didn’t consider that the context over the whole conversation was the user’s responsibility or that it was even something that could be managed by the user.

The exam makes this responsibility very clear. You, as the developer using Claude Code, need to know the context the model has, how and when to use persistent instructions, understanding that these take up space in the model’s context window, and how and when to use on-demand instructions using skills and rules, so you don’t ‘use up’ the context window unnecessarily. You need to recognise when the context has become stale, when you’re at risk of losing important detail, when the conversation has wandered, and how to reestablish the context you need to continue working. In the exam, they expect you to know the techniques to manage context in these ways.

‍

Externalising the working state

I’ve found one such technique that transfers very well to general chatbot interactions and that is externalising important context. By this, I mean creating a concise, external record of the important details from the conversation. Where I have found this most useful is when I’m brainstorming or exploring an idea with ChatGPT. These conversations can get very long and it’s easy for the LLM to get mixed up between ideas you are still working on, and ideas you’ve decided to keep or ones you have rejected. I’ve often felt in these types of conversations that, from one turn to the next, we are aligned, but as the conversation continues, we end up revisiting old ground, or the LLM goes off on a tangent and seems to get stuck there. It can end up being a bit of a mess.

This is where the working state document comes in, creating a division between where you explore and iterate over ideas (the conversation) and where those ideas are stored (the separate document). The working state document is a persistent and visible record of the key points from the conversation, aiding continuity. It helps keep the conversation on track and, at the end, you’ve got a portable record you can use to seed other conversations, if necessary.

So how do I create and use this document? It varies slightly by application and by how automated you want the workflow to be. I’m going to describe things I’ve tried with ChatGPT. The simplest route is asking the LLM to produce a working state document (.md) at the beginning of the chat, and then throughout the conversation, when you come to something you want recording, you ask it to add it to the document. You can then ask to download the file at the end. At the other end of the spectrum, you can create instructions for what you want the LLM to include in your document and how you want it to be organised, save these instructions in Project Sources. Then in the conversation, you can ask it to create the working state document and to periodically update it. The LLM will decide, using your instructions, what to record.

Reviewing AI-generated output with AI

Other key concepts from the Claude exam relate to how to use the LLM effectively to review its own output. Two techniques in particular transfer well to general AI use cases, such as reviewing a proposal document you’ve created with Claude.

A separate instance

The first technique is as simple as using a separate instance of the model to review the output.  Just as we struggle to proof-read our own work, LLMs find it a challenge too. The context they have from the generation stage, their reasoning and original framing of a solution, can ‘anchor’ them, meaning they can’t see what they’ve produced with ‘fresh eyes’. In Claude Code, the solution is starting a new session, isolated from chat that generated the code. When working with applications like Claude or ChatGPT, we have the extra challenge of project- and application-wide memory that we might want to isolate ourselves from. This clean separation can be achieved in a few ways. If you created the document with Claude, you could review with ChatGPT, or any other LLM, or in an incognito chat inside Claude.  

Multi-pass reviews

The second technique aims to solve the issue of attention dilution and, in particular, the ‘lost-in-the-middle’ effect. If the input to review is large - say a large codebase in Claude Code or a long document in ChatGPT - and would use a significant amount of the context window, asking the LLM to critique this input at any real level of depth means it won’t be able to ‘attend’ to the whole input equally. The LLM may provide good feedback on the start and end of the input, but it might get ‘lost-in-the-middle’ and give you shallow feedback on material in the middle.  

So, what’s the technique taught in the course to address this problem? The best practice they recommend here is to ‘do multi-pass reviews plus a final integration review’, that is to break a large review task into smaller focused ‘passes’, where, for each turn, it focuses on one small aspect of feedback required, and then a final integration ‘pass’ that collates and evaluates the findings of these separate reviews, resolving conflicts or interactions between them. Applied to reviewing a long proposal document, this might be:

1. Does it answer the client brief?

2. Does it make sense to the reader?

3. Is the case persuasive?

4. Is it technically correct?

5. Is it clear and easy to understand?

Plus, the integration review that evaluates all these findings, with a prompt such as:

"Which findings actually require changes to the proposal, which are duplicates or symptoms of the same underlying issue, and which should not be acted on? Produce a prioritised set of changes that improves the proposal without creating new inconsistencies or altering the intended meaning."

Iterating well

This final technique I’ve adapted from the exam is another simple one, but again, something that I hadn’t considered before, and it’s regarding iterating on the AI-generated output. Making changes to a document with AI sounds easy - you just tell it what you want to change, surely? - but in my experience it’s never been as simple as that.  

It seems to refuse to change the one thing you want to change, while changing other things you didn’t want it to touch, or it takes your feedback and applies it incredibly literally, all the while you’re passing the document back and forth 100 times.  

The exam technique doesn’t claim to solve all these issues but it does make the iterations easier to control. Here, the key idea from the Claude exam is that we need to identify whether the changes we want to make are coupled to each other, or independent. If they are dependent on each other, we should give the LLM instructions to make all the changes at the same time. If they are independent, each change can be implemented separately. The same thing applies to iterating over a document.

Conclusion: go for a structured workflow

Personally, I’ve found the results of these experiments interesting, and they have definitely changed how I interact with ChatGPT or Claude. The biggest lesson is perhaps that a structured workflow for these interactions makes an enormous difference. And it makes sense: if we are using an LLM as part of our work, we need to structure the interaction much as we would structure work with a human colleague. The techniques are different, but the fundamental idea is the same.