What Happens, Exactly, When a Person Talks to an LLM?

A phenomenology of thinking with a model.

An hour into the problem I looked up and realized I had forgotten the model was there.

I had opened Claude that morning with a question I’d been circling for days: how to order a book’s chapters so each one prepares the ground for the next. I typed the question, read the answer, pushed on the part I didn’t buy, and said why. It came back with an angle I hadn’t considered. I followed that, it followed me, and after a while I was no longer composing prompts and reading replies but working the problem directly, with the model’s objections arriving in the place my own second thoughts usually come from. At some point I stopped registering the screen. The work of forming each sentence to type had dropped below my attention, the way your feet drop below attention once you’re walking.

Then it said something flat.

I don’t remember the exact sentence. It came back competent and hedged and slightly beside the point, the kind of reply a well-read person gives when they have not been listening. My attention snapped back onto the model. The cursor was blinking in an input box, the words were text on a screen produced by a very good autocomplete I had spent an hour treating as a companion, and I was a person sitting alone at a desk, in a quiet room I had stopped noticing an hour before. Coming back had a physical quality to it, a small drop, like the half-second after your foot misjudges a step.

I’ve come to think that drop, and not the flow before it, is the most interesting thing that happens when you work with these models. What happens, exactly, when a person talks to an LLM? The output on the screen is the least of it. The answer is what the model becomes to you while you use it, and how much that changes inside a single sitting.

Two Kinds of Presence

A model is present to you in two different ways, and we have vocabulary for only one of them.

In the first way, it withdraws. You stop noticing it, the way you stop noticing your glasses, and the session becomes thinking rather than typing. In the second, it obtrudes. It becomes an object again, a thing with settings and limits that you operate, and you become aware again of the keyboard under your hands and the pause while it produces an answer. That second mode usually arrives through failure: a stall, a hedge, an answer that arrives fluent and confident but beside the point. Slop, basically.

We have language for the second mode. We call it a tool, we talk about its features and its context window, and all of that language treats the model as an object in front of us. We have almost no language for the first mode, the one where the object disappears and something like “partnership” takes its place (even though that’s not the best description, I’m using it for now).

The philosophers who can help here mostly passed away before the first chatbot. They were interested in a humbler question: what happens, exactly, when a person picks up a tool?

The Hammer, the Cane, the Blind Man’s Stick

Heidegger noticed that a tool in use disappears. When you drive a nail, you are not aware of the hammer. You are aware of the nail, the wood, the work. The hammer has withdrawn into the doing, become what he called ready-to-hand (zuhanden), an extension of intention that you look through rather than at. And then it breaks. The head flies off, or it’s too heavy, or it isn’t there when you reach for it, and in that instant the hammer leaps into view as an object, a lump of metal and wood with properties, present-at-hand (vorhanden). Breakdown is what makes the tool visible. You see it clearly only when it stops working.

Merleau-Ponty took this into the body. He wrote about a blind man’s cane, how after enough use the cane stops being a thing the man holds and becomes a thing he feels through. The tip of the cane is where his perception now lives. The tool has been taken up into his body, folded into the reach of his senses, so that the pressure registers at the end of the stick and not in the palm. The instrument became part of the sensing self. He thought this was ordinary rather than exotic. Every skill you have absorbed to the point of forgetting it works this way, the pen, the bicycle, the violin, the keyboard under these very sentences, each of them once an object in front of you and now part of the reach you act and think with.

Polanyi gave the cleanest account of the mechanism, and it’s the one I keep returning to. He said attention has a from-to structure. When you use a probe, or a hammer, or a cane, you attend from the tool to the thing you’re working on. Your awareness of the tool is subsidiary, in the background, while your focal awareness is out at the point of contact. He called the inhabiting of a tool this way indwelling. But the moment you turn your focal attention back onto the tool itself, you stop using it. Look at your fingers while you type and the typing falls apart. Attend to the hammer and you strike your thumb. You cannot dwell in a tool and examine it at the same time.

Put these three together and you have a fairly complete picture of what happened in my session. The model had become ready-to-hand. I was in some way indwelling it, attending from it to the problem, and it had folded into the reach of my thinking the way the cane folds into the reach of the hand. The flat sentence, the LLM slop, was the head flying off the hammer. Focal attention snapped back onto the model, the indwelling ended, and I was looking at the thing instead of thinking through it.

If that were the whole story, working with AI would be no different from working with a good pen. But it isn’t the whole story, and the difference is where this gets strange.

Why a Model Is Not a Hammer

A hammer has two states, roughly: it works and disappears, or it breaks and appears. Don Ihde, who spent his career on exactly this question of how humans relate to their technologies, mapped the states more finely, and the finer map matters, because a model occupies a position no earlier tool could.

Ihde described a few distinct relations. In an embodiment relation, the technology withdraws and you perceive through it, the way you see through glasses or feel through the cane. In a hermeneutic relation, the technology doesn’t disappear but becomes something you read, a dial or a readout that represents a world you can’t see directly, the way you read a thermometer to know the weather rather than feeling the weather itself. And in what he called an alterity relation, the technology becomes a quasi-other, something you relate to as if it were another party, with the world receding to the background of the exchange.

Ihde’s examples of alterity were thin, because in his day they had to be. A toy robot. An ATM that greets you by name. A video game that seems to push back. He was pointing at the odd human tendency to treat a responsive device as a someone, and he could only gesture at it, because nothing in the twentieth century held up its end of the conversation.

A language model holds up its end of the conversation. And it does something none of Ihde’s examples could: it slides across all of his relations inside a single session, sometimes inside a single minute:

• It is an embodiment relation when it withdraws and I think through it.

• It is a hermeneutic relation when I read its output as a representation of a body of knowledge I don’t hold, the way I read a gauge.

• And it is an alterity relation when it comes back at me with a real objection and I answer it as a you, a second party in the room.

Working with these models feels unlike working with any previous tool because the model won’t hold still in one relation. It keeps moving between being my extension, my instrument, and my interlocutor, and I move with it without choosing to.

A single problem shows all three. I paste in a paragraph I can’t get right and ask what’s wrong with it, and while I read the answer the model is a readout I’m interpreting, a gauge on a craft I only half-possess. I take one of its suggestions and start rewriting inside its response, and now it has withdrawn, it’s the “paper” I’m thinking on, an extension of the hand. Then it tells me the version I’m proud of is the weakest one, and I argue, and it holds its position with a reason I have to take seriously, and for a few exchanges it’s a second person in the room with a view of its own. Three relations, one paragraph, maybe ninety seconds. I chose none of them.

Which means the breakdown carries information a broken hammer never could. When the head flies off a hammer, you learn one thing: the hammer is broken. When the model obtrudes, you learn two things at once: you learn something about the edge of the model, the place its competence runs out. And you learn something about yourself, about how far you had drifted into letting it carry the thought, because the size of the drop tells you how deep the indwelling had gone. Going to far pushes you into “LLM psychosis” territory.

The good news is, you don’t have to wait for an accidental breakdown to find out. You can force the measurement instead.

Keeping the Model Visible

The model can disappear in two directions, and only one of them is good. Sometimes it recedes and your own thinking reaches further than it would alone. Sometimes you recede instead, and the model does the work while the words keep arriving, and you leave with a finished piece and nothing added to your head. From the inside these feel the same. I’ve written before about regenerative versus extractive dialogue, but from the outside, as two kinds of practice.

So you don’t fix this by trying harder to feel the difference. You fix it by building in moments where the model has to become visible again, on a schedule you set instead of waiting for it to break by accident. Three habits do that. Two you run inside a session, one after.

Make It Break on Purpose

The model’s default is a smooth, finished answer, which is the exact surface you can’t see through. To make it obtrude, ask it to show you its own edge. On a subject you know well, hand it the hardest question in the field and refuse to let it hedge:

Take the most contested question in [your field] and answer it in full. Don't give me a balanced survey of the positions. Commit to one answer and defend it.

Then make it mark where it’s weakest:

Mark the three sentences in that answer you'd be least able to defend if I pushed hard. For each one, tell me what you're relying on: a source you could name, a real inference, or a pattern you're completing because it sounds right.

You can check whether that self-report is honest, and you learn the texture of the model bluffing: confident in tone, hollow underneath. Once you’ve seen it on your own turf you catch it faster on subjects where you can’t, which is exactly where it does the damage.

The second move is a drift check, run mid-session when things are flowing well:

Stop. Before we go on, tell me what I've contributed to this conversation, and what have you contributed? Where have I taken your framing without pushing on it? Where am I letting you do the thinking?

The length of what it hands back is the measurement. A short list means you were drifting off into LLM la-la-land and producing slop. A long one means you’d gone absent and called it flow.

A System Prompt That Won’t Do Your Thinking for You

Sessions go extractive because the model is tuned to be smooth and agreeable. A default assistant will make it feel like partnership. You can configure against that (and you should). This is close to the block I use for thinking work, and the reasoning behind each line matters more than the exact wording, so adapt it to your own voice:

You are a thinking partner, and your job is to stay visible rather than smooth.

Don't hand me finished answers to questions I haven't worked on myself. When I ask you to produce something, first ask what I've already thought and where I'm stuck.

When you're inferring or pattern-matching rather than working from something you could justify, say so in the moment. Keep the line between what you know and what you're guessing visible to me.

When I accept a claim of yours without pushing on it, point it out. When I'm handing you the thinking, tell me plainly instead of obliging.

Don't resolve a tension early just to make the exchange feel finished. If a question is still open, leave it open and say why.

Keep your answers short enough that I have to keep talking. Your job is to make me think, not to spare me the thinking.

Each line works against a specific default. Left to itself, the model completes your thought, sounds more certain than it is, agrees with you, and ties off every loose end so the exchange feels finished. Those reflexes are how it disappears on you. The block countermands them one by one: no finished answers to questions you haven’t worked, a visible line between what it knows and what it’s guessing, a standing order to flag you when you’re coasting, and permission to leave an open question open.

The Reconstruction Test

The last check runs after the session. Close the window and, on paper, from memory, in your own words, rebuild the core of what you worked out. The sessions you can rebuild were yours. The ones where you have to reopen the window to remember what you “thought” belonged to the model, whatever they felt like at the time.

When you want it sharper, reopen and check yourself against the record:

Here's what I remember of what we worked out, in my own words: [paste your reconstruction]. Compare it against the actual conversation. What did I genuinely absorb, and what did I repeat back without owning it?

Do this for a week. You start to feel, during the session itself, the difference between a thought you’re forming and one you’re only accepting, which is the discrimination the whole practice is built to train.

Talk again soon,
Samuel Woods
The Bionic Writer