You Were Never There

When Jean Piaget was a boy, he had a memory he could see. He was about two years old, strapped into his pram, being wheeled along a busy avenue near the Champs-Élysées in Paris. A man lunged out of the crowd and tried to snatch him. His nurse threw herself in the way and fought the man off. Piaget remembered the scuffle. He remembered the scratches the man’s nails left on the nurse’s face. He remembered a policeman in a short cloak arriving with a white baton, and the man running off. He could place himself in the scene, looking out from the pram at the whole thing.

He held onto that memory for years. His family believed it too. The nurse had been given a watch as a reward for her bravery.

Then, when Piaget was around fifteen, the nurse wrote his parents a letter. She had joined the Salvation Army, and she wanted to clear her conscience. There had been no kidnapper. She had made the whole story up. She sent back the watch.

So here is the strange part. Piaget’s memory did not feel like a story someone told him. It felt like something he had seen with his own eyes. He could picture the scratches. Later, as one of the most careful students of the child’s mind who ever lived, he worked out what must have happened. He had heard the adults tell the tale when he was small, and his own mind had built it into a picture and filed it under “things that happened to me.” Here is how he put it: “I must therefore have heard, as a child, the account of this story which my parents believed, and projected it into the past in the form of a visual memory.”

A false story walked into his head and came out the other side as a memory, and from the inside he could not tell the difference. Hold that in mind. We are going to ask why he couldn’t — and the answer is going to cost you something you probably feel very sure about.

What you already believe

Most of us carry a simple picture of what memory is, and it goes like this.

Memory is a recording. Somewhere in your head there is a store, a kind of filing cabinet or a library or a hard drive, and when something happens to you, a copy of it gets written down and put away. Later, when you remember, you go and pull the file. The copy might be faded. Some files get lost. Some get dusty and hard to find. But the file itself is a record — its whole job is to be faithful to what really happened. That is what makes a memory a memory. It points back at a real event, like a photograph points back at the day it was taken.

And imagination is the opposite faculty. Imagination is the part of you that makes things up. It builds castles that were never built, dragons that never flew, conversations you never had. Its job is not to be faithful to anything. Its job is to invent. Where memory looks backward at what was, imagination looks off to the side at what could be.

Two different powers. Two different jobs. One retrieves the truth; the other spins fiction. You can feel the line between them right now. Picture your kitchen — that’s memory, a real room you can walk through. Now picture your kitchen with a full-grown giraffe standing in it — that’s imagination, and you know it’s imagination, because you know the giraffe was never there.

This picture feels obviously correct. It matches how remembering feels from the inside: like opening a drawer and finding the thing you put there. And for most of daily life it works fine. But it is wrong in a way that, once you see it, changes how much you can trust the story you tell about your own life. Piaget’s pram is the first crack. Let’s widen it.

The same faint stuff

Start with the material. What is a memory actually made of?

When you remember the beach, you get a picture of the beach — dimmer than the real beach, smaller, missing details, but a picture all the same. When you imagine a beach you’ve never visited, you also get a picture — dimmer, smaller, missing details. The two are made of the same thing. Thinkers have known this for a very long time and given the stuff a plain name: it is sense gone faint. An image is what a sensation leaves behind when the thing itself is gone.

Thomas Hobbes said it flatly. After you see something and look away, “we still retain an image of the thing seen, though more obscure than when we see it. And this is it the Latins call imagination, from the image made in seeing.” Imagination, he wrote, “is nothing but decaying sense.” A sight, fading.

Now watch what he does next, because it is the whole essay in one line. Hobbes says memory is made of the very same fading sight. The only difference is which part of it you are pointing at:

This decaying sense, when we would express the thing itself (I mean fancy itself), we call imagination… But when we would express the decay, and signify that the sense is fading, old, and past, it is called memory. So that imagination and memory are but one thing, which for diverse considerations hath diverse names.

Read that again slowly. One thing. Two names. When you attend to the picture as a picture, you call it imagination. When you attend to the fact that it’s old, faded, from the past, you call it memory. Hobbes is not saying they are similar. He is saying they are the same faculty, and the words “memory” and “imagination” just pick out two things you can notice about one faint image.

Hobbes is at the far end of this. He’s a hard materialist, and he collapses the two words all the way down to one thing. Not everyone goes that far, and the disagreement is worth seeing, because it turns out to sharpen the point rather than blunt it. Aristotle, two thousand years earlier, kept a real distinction — but not the one you’d expect. In his little book On Memory he seats memory in the very same image-holding part of us that does the imagining, “the primary faculty of sense-perception.” What sets memory apart, for him, is not a different organ or a different kind of picture. It’s that memory is always of the past: “only those animals which perceive time remember.” Remembering is imagining plus a reference backward in time. So even the thinker who insists memory and imagination are distinct locates the whole difference in something added on top of a shared picture — not in the picture itself.

So the first neat wall comes down. You thought memory was a recording device and imagination was an invention device — two different machines. But at the level of raw material they run on the same fuel: the faint image, sense in decay. Piaget’s kidnapper and your real childhood home are cut from identical cloth.

That alone might not worry you. You could say: fine, same material, but surely there’s a tag on the real one. Surely a genuine memory comes stamped somehow — a little mark that says TRUE, this really happened. Let’s go looking for the stamp.

Where the difference isn’t

If memory and imagination share their material, then the difference between them has to live somewhere else. The obvious place to look is in the image itself. Maybe a memory-image just looks different — brighter, sharper, wearing a badge that says “on file, verified.”

It doesn’t. And the people who looked hardest all came back with the same report: the difference is not a second thing added to the picture. It is a way of holding the same picture.

Aristotle noticed something odd about any image in your mind. It can be two things at once, depending on how you look at it. He uses the example of a painting. A painting on a wooden panel is, at the very same time, a real object — paint and wood you can look at for its own sake — and also a likeness of somebody, a picture of someone who isn’t in the room. Same panel, two ways to regard it:

A picture painted on a panel is at once a picture and a likeness: that is, while one and the same, it is both of these… one may contemplate it either as a picture, or as a likeness.

Now put a mental image where the panel was. When you look at the image just for itself — this shape, this color, this scene — you are imagining. When you look at the very same image as a likeness of something that actually happened to you, as a token standing in for a real past moment, you are remembering. Aristotle’s word for a memory is exactly that: a presentation “related as a likeness to that of which it is a presentation.” Nothing has been added to the image. You have added a relation — you’ve aimed it at the past.

Plotinus, working in the same tradition, said it in one clean sentence that sounds shocking if you still hold the filing-cabinet view: “Remembrance, thus, is vested in the imaging faculty; and memory deals with images.” Memory doesn’t get its own private vault. It borrows the imagination’s picture-making room and works there.

This is the second wall, and it’s a stranger one. You thought that even if memory and imagination shared material, a real memory would at least carry some inner mark of being real.

There is a mark — but it’s weaker than you’d hope, and it can be counterfeited. Psychologists who study this closely, above all Marcia Johnson, describe how we actually tell a memory from a daydream: we lean on the qualities of the image. Memories of things that really happened tend, on average, to come with more sensory detail and more of a setting — where you were, what time it was, what things looked and sounded like — while things you merely imagined tend to come thinner, with fewer of those trimmings and more trace of the mental effort you spent building them. That’s a real cue, and most of the time it works. But notice the words tend and on average. It is a rule of thumb, not a seal. And a rule of thumb can be fooled. An image that gets imagined vividly enough, and often enough, acquires the sensory detail and the setting — and then it passes the test. A story told at the family table year after year comes, in the end, wearing all the trimmings of something seen. That is exactly what happened in the pram.

So the mark inside the image is only a tendency, and a vivid enough imagining forges it. Which raises the obvious, uncomfortable question. If the cue can be counterfeited, what actually makes you treat one image as your true history and another as make-believe — and go on trusting it?

The answer is the part that costs you something.

It’s a feeling

Here is what actually flags an image as a memory. Not evidence. Not a tag. A feeling.

William James looked straight at this and refused to blink. Suppose, he said, you have a perfect mental image of some past event. Is that image, all by itself, a memory? No. An image on its own “is simply a duplicate, a second event, having absolutely no connection with the first event except that it happens to resemble it.” A clock strikes today the same way it struck yesterday, but today’s strike does not remember yesterday’s. Resemblance is not memory. So what has to be added?

James lists the pieces. A real memory needs the image, plus a sense that the thing lies back along the direction of time, plus one more ingredient he can only describe as a feeling — the “warmth and intimacy” of something owned, the sense that this happened to me, I was there. And then he says the line that should stop you cold. Strip that feeling away, and “a merely imagined past event differs from a recollected one only in the absence of this peculiar-feeling relation.”

Let that sink in. The difference between a memory and a daydream about the past is, at bottom, a feeling of ownership that rides on top of the image. Not the image. Not any proof in the image. A warmth. A hum of “mine, real, then.” James called memory “an object imagined in the past… to which the emotion of belief adheres.” Belief is an emotion here, a thing you feel, not a thing you check.

David Hume had already dug to the same floor from a different direction. He was chasing the difference between believing something and merely imagining it. And he found that the difference is not in the idea at all. You can picture a true thing and a made-up thing with equal clarity; picturing is not believing. “We can,” he wrote, “in our conception, join the head of a man to the body of a horse; but it is not in our power to believe that such an animal has ever really existed.” Imagining the centaur is easy. Believing in it is not something you can just decide to do. So what is belief, if it isn’t in the picture? Hume’s answer:

Belief is nothing but a more vivid, lively, forcible, firm, steady conception of an object, than what the imagination alone is ever able to attain.

Belief is a feeling attached to a picture — a certain forcefulness, a weight, a there-ness — and crucially, you can’t summon it at will. It arrives on its own. When it arrives, the picture feels real. When it doesn’t, the same picture feels like fancy.

Now go back to Piaget in his pram. His mind held a vivid image of a kidnapper and a brave nurse. That image carried the warmth James described and the force Hume described. It felt owned. It felt real. And it was pure invention. The feeling of “this happened to me” had simply attached itself to a picture his mind built from an overheard story. Nothing was broken in Piaget. His memory machine was working exactly as designed. The design is the problem: the feeling that marks a memory as true is a separate thing from the truth, and it can settle onto a fantasy without asking permission.

This is the turn the whole essay was walking toward. You believed memory and imagination were two faculties, one that reports the truth and one that invents. They are one faculty. It builds pictures. Some of those pictures come wrapped in a feeling of belief, and those you call memories. But the feeling is not proof. It is not even attached to proof. It is a warmth that usually — usually — lands on real events, and sometimes lands on nothing at all, and from the inside the two feel identical.

If you want to watch this happen to ordinary people on purpose, the psychologist Elizabeth Loftus has spent a career doing it. In one well-known study her team gave adults short write-ups of childhood events supplied by their families — real ones, with a single invented story mixed in: the time, around age five, they got lost in a shopping mall and were rescued by a kind stranger. It never happened. But after being asked to think back on it, a good share of the subjects “remembered” it. They added details the researchers never gave them — what the stranger wore, how scared they felt, what the mall looked like. They were not lying. The feeling of belief had arrived and attached itself to a supplied picture, exactly as Hume and James said it could. Handed a story, the mind did what Piaget’s did: it imagined the scene and then owned it.

Now, the feeling is not literally the only thing you have to go on, and it would be cheating to pretend it is. You also check a memory against the rest of your life. A real memory usually locks into the others — it has a before and an after, it fits with where you lived and who you knew. René Descartes leaned on exactly this to tell waking from dreaming: our memory, he wrote, “can never connect our dreams with one another, or with the whole course of our lives, as it unites events which happen to us while we are awake.” Coherence is a genuine check, and a good one. It is, in fact, the check that finally caught Piaget’s kidnapper — not an inner alarm, but an outside fact, the nurse’s letter, that refused to fit. But coherence only postpones the problem. The web you test a memory against is itself made of memories, built the same reconstructive way, wearing the same feeling of belief. A false memory that fits — a plausible scene, in the right year, with the right people — sails through, because nothing in the web contradicts it. Coherence raises the odds. It does not hand you the truth, and it never touches the memories that quietly agree with each other.

Filled in by habit

You might still hope the damage is small. Maybe the feeling of belief slips onto a fantasy now and then — a Piaget here, a mall there — but surely, most of the time, the picture is at least a real leftover of a real event. Surely memory only fails at the edges.

The trouble is that the picture itself is not pulled up whole. It is rebuilt, on the spot, out of scattered pieces — and it is rebuilt along the grooves of your own habits.

Baruch Spinoza gave memory a definition that sounds dry until you see what it means. Memory, he said, “is nothing else than a certain concatenation of ideas” — a chaining-together, one idea tugging the next into view. And what decides which idea tugs which? Not the truth of the past. Your personal history of associations. He gives a sharp little example. A soldier and a farmer both see the same thing — the print of a horse’s hoof in the sand — and their minds go to opposite places:

The soldier… if he sees the footsteps of a horse in the sand, will immediately turn from the thought of a horse to the thought of a horseman, and so to the thought of war. The countryman, on the other hand, from the thought of a horse will turn to the thought of his plough, his field.

Same track in the sand. Two different chains, because two different lives laid down two different sets of grooves. Now notice what this does to remembering. When you recall an event, you don’t retrieve a sealed file. You start from a fragment — a face, a room, a smell — and your mind runs the chains outward, pulling in whatever is associated, whatever fits, whatever is plausible to a person like you. The gaps get filled not with what happened but with what usually happens, what would make sense, what you’d expect. A soldier’s memory fills a blank one way; a farmer’s fills it another. The reconstruction leans toward the rememberer.

This is why false memories are not random noise. When Loftus’s subjects invented the mall, they didn’t invent nonsense — they invented a believable rescue, with a plausible stranger and plausible fear, because their minds filled the empty scene with the kind of thing that could have happened to them. The imagination doing the filling is competent. It builds scenes that fit. That’s exactly what makes its work so hard to catch: a reconstructed memory looks like a real one because it was assembled by the same skilled hand that would have assembled the real one.

So memory does not just occasionally let a fantasy through. It is reconstructive by nature. Every act of remembering is partly an act of building, and the builder is your own associative habit, quietly reaching for what fits. The record you trust is edited in the very act of reading it.

The self you keep rewriting

There’s a last step, and it’s the one that reaches past the page into your own life.

Memory is not just a party trick that sometimes misfires. Memory is the thread you hang your whole self on. The reason you are the same person who woke up yesterday, the same person who was a child, is that you can remember being them. Your memory is your evidence that there is a you stretched out across time. It is the record you trust more than any other, because it is yours and it is from the inside.

And we have just seen what that record is made of: images, indistinguishable in kind from imagination, marked as “real” by a feeling that can be wrong.

It gets one turn stranger, and here modern brain science finishes a thought the old writers started — though it cuts both ways, and honesty means telling both. John Locke, back in the 1600s, already saw memory as something that fades and has to be refreshed. “The pictures drawn in our minds,” he wrote, “are laid in fading colours; and if not sometimes refreshed… vanish and disappear.” He pictured memory as a wall painting that dims unless you touch it up. But he assumed the touching-up was faithful — that you refreshed the same picture.

Refreshing a memory can change it. There’s evidence — first in animals, later in people — that pulling up a memory doesn’t leave it safe in its drawer while you look. Recalling it can make the trace soft again, editable, before it gets stored back down, and it can come back altered, with whatever your present mood and beliefs and imagination stirred in while it was open. The evidence in humans is real but messier and more argued-over than the tidy version admits, so don’t oversell it. And it genuinely cuts the other way too: most of the time, calling a memory up strengthens it. Practice the past and it sets harder, which is why a story told a hundred times can be the one you hold most firmly. That is the trap folded inside the gift. The memory you’ve revisited most is both the most firmly held and the most worked-over — rehearsed until it’s solid, and re-touched every time you rehearsed it. Firmness and fidelity come apart. The vivid childhood scene you’ve told a hundred times is held like granite and may be, by now, a hundredth-generation copy, each telling touched by the imagination that shares its machinery.

This is why the science-fiction version of the nightmare — the android in Blade Runner who has another person’s memories implanted and cannot tell they aren’t hers, and weeps when she learns — is not really science fiction. It’s a slightly dressed-up account of what a memory is. A picture, plus the feeling that it’s yours. Give a mind that picture and that feeling, however they got there, and it will have a memory, full stop. Rachael’s implanted childhood and Piaget’s kidnapping are the same object.

So the tidy picture we started with is gone. Memory is not a recording clerk who files the truth while imagination sits in the next office making things up. There is only one office. One faculty, building images, some of them stamped with a feeling of “this was real.” That faculty is how you know what happened to you. It is also how you know who you are. And it runs on the same equipment, and the same neutral pictures, as the part of you that dreams up giraffes in the kitchen.

The watch, returned

Go back to the boy in the pram.

The nurse’s letter did not just correct one fact. It exposed the machine. Piaget had a memory that was vivid, detailed, owned, and completely false, and the only reason he ever learned it was false is that someone happened to confess. No inner alarm had gone off. The feeling of truth had sat on the fantasy as comfortably as it sits on anything real. If the nurse had kept the watch and kept quiet, Piaget would have gone to his grave certain that a man once tried to steal him and a brave woman fought the man off — and he would have been just as certain of that as he was of anything he actually lived.

You have a pram memory too. Several of them. You don’t know which ones, and that is the whole point. Some of the scenes you are surest you lived, you assembled — out of photographs, out of family stories told and retold at the table, out of the person you needed yourself to have been. They carry the warmth. They feel owned. The feeling is telling you the truth about how the memory feels, and nothing at all about whether it happened.

Piaget got his correction by accident. Most of the time nobody writes the letter and nobody sends back the watch.