The Brick: Before accepting a response, make sure you know whether the task is one Claude can perform reliably or one that requires context.
When you're working with Claude it’s important to be aware that there can be two kinds of responses that look almost identical. These are competent and plausible responses, and knowing the make-up of both is an important skill to have in your AI toolkit.
The underlying issue I want you to be aware of here is the slippery-slope between these responses. Both will be presented confidently, professionally, and clearly written. In many instances, the plausible response may even appear to be the stronger of the two, which is the concern.
Without a glaring mistake, reading the response alone will make them practically indistinguishable. The slippery-slope is you may develop a bad habit of trusting all outputs equally, and this can have real implications if someone else notices it and you don’t.
The Competent Response
Claude is genuinely good at transforming information you’ve provided. Here are some example tasks:
- Summarizing a document
- Analyzing a spreadsheet
- Pulling action items out of a meeting transcript
- Drafting a first pass from a clear brief
- Turning a messy email thread into a structured client brief
These each provide raw material that is directly related to the task, so Claude is operating with context.
The Plausible Response
Claude may appear good when generally asked to produce outputs that require specific facts, numbers, or details without being given the proper context. Consider questions like:
- What’s the average time to onboard a new enterprise client?
- How often do project managers run retrospectives?
- What’s a healthy retention rate for B2B SaaS?
- How many accounts should a Client Success Manager have?
- How long should a typical discovery call run?
These questions leave a lot of room for creative thinking. You’ll get responses. Some of it may be correct. Some may be wrong, out of date, or just fabricated. But at first glance, the response will be plausible.
This is the failure mode people mean when you hear the word “hallucinate”. It’s not random either. When you ask an AI questions that require specific details, but don’t supply them, the result is hallucinations. Even though some of the information is fabricated, it will still be presented confidently and professionally.
A Practical Example
Imagine a project manager is building a business case for their company and needs to consider more buffer time for their software rollout planning. They need a benchmark to back up their request.
Scenario 1: You provide Claude data from previous projects. Timelines, project plans, scope documents, retrospectives, and ask to draft a recommendation for appropriate buffer time based on historical performance. Claude shapes your own material into a clean recommendation tied to the facts of the business and its performance in similar projects. This will be a competent response.
Scenario 2: You ask Claude “What percentage of enterprise software implementations fail to meet their original timeline?” Claude returns a response with some percentage, a source, and an explanation. The number is fabricated, not in the sense that it just made it up, but in the sense that it grabs whatever data point it could that may be relevant, presenting it with confidence and professionally. But it’s plausible.
The Takeaway
Before you accept an output, take the 10 seconds to ask yourself: did I give Claude the material to answer this? Or am I asking it to produce the material? If you have, you can assure a competent answer. If you haven’t, then you’ll need to recognize that Claude is producing a plausible response, at best.
A fluent operator is the one who makes this check without thinking about it. That comes with practice and time.