Elevator, Escalator, Lift: What Your Wrong Answers Taught Us About Riddles

One lock in Vault 8 reads like this:

I go from the first floor to the second floor and back again all day long, but I never move an inch. What am I?

It has come up four times so far. It hasn't opened once.

The answers were elevator, escalator and lift. The fourth try ran out the clock.

Look at those three for a second. Every one of them is a reasonable guess. Each is a thing that goes between floors all day long, and whoever typed them was thinking hard about the question. And every one of them breaks the last clue, the one that says "I never move an inch". Elevators move. That's the whole job.

The answer is stairs. Once you see it, it's hard to believe it ever hid from you.

We've been running Crack the Vault for a few weeks now. In that time, about 50 people have played around 130 runs and typed in a little over 560 answers. That's small, but it's enough to start seeing patterns, and the interesting ones are all in the wrong answers.

So this post is about those: what people typed, why it made sense at the time, and what it has changed about how we write locks.

Fair warning: this post gives away the answers to about a dozen locks. Every run picks its four to six locks from a pool of 30 per vault, so there will be plenty left to solve on your own.

First, how a lock decides you're right

When you type an answer, the server tidies it up before it compares anything. Capital letters don't matter. Spaces, dashes and punctuation get stripped out. A leading "a", "an" or "the" is dropped, so "the moon" and "moon" are the same answer.

Every question then has a list of answers we accept, not just one. The pine tree riddle accepts pine, pines, evergreen, fir, spruce and conifer. The horse riddle accepts horse, pony, donkey, mule and camel.

Longer words get a little slack, too. If the answer is six letters or more and you're one letter off, it still opens. "Umbrela" opens the umbrella lock, and "calender" opens the calendar lock. (Hold that thought. It comes back later.)

That's the machinery. The hard part is deciding what goes on the list.

Wrong answers live in the right neighbourhood

The thing we noticed first is that almost nobody types nonsense. The wrong answers are close. People find the right corner of the map and then take one wrong turn on the last street.

Here are some of the near misses we've seen so far.

Fingerprint. The riddle says: "You have ten of me, nobody else in the world has a set exactly like yours, and detectives dust for me." Players answered finger and fingers.

That's a good start. You do have ten fingers. But the next two clues don't fit. Lots of people have fingers like yours, and no detective has ever dusted for a finger. The first clue opened the door, and the guess walked through before the other two clues got a say.

Forgery. "I'm a fake painting passed off as the real thing, and museum experts work hard to catch me." One player tried copy, then duplicate, then replica, which is about as close as you can circle an answer without landing on it.

This one we argued about. A replica is a kind of fake painting, sort of. But a replica doesn't lie to anyone. It hangs in a gift shop with a label that says replica. A forgery pretends to be the real thing, and that pretending is the part museum experts are hunting for. So "fake" and "counterfeit" open this lock, and "replica" doesn't. We think that's the fair line, and we'll keep watching it.

Pine. "I stay green all winter long. My needles don't sew, and I drop little cones on the forest floor." Players answered tree and Christmas tree.

Both are true. Neither is what the riddle asks for. "Tree" is like answering "animal" when the riddle describes a penguin. The needles and the cones are there to narrow it down to one kind of tree, and that's why the lock wants pine, fir, spruce or something just as specific.

Avalanche. "I start as a whisper of loose snow high on the slope and end as a roaring white river racing down the mountain." Players answered glacier, igloo and waterfall.

Snow, mountains, ice, a river of something white. Every guess is built out of pieces of the clue. A glacier is the closest, and it fails on speed. Glaciers don't roar, and they certainly don't race.

Shadow. "I follow you all day when the sun is out, but I disappear when clouds roll in or the lights go off." One player answered sun.

This is our favourite kind of miss. The sun is in the clue, and it's even in the right place. It just isn't the thing doing the following.

River. "I have a bank but no money, a bed but I never sleep, and I run all the way to the sea without legs." Players answered seabed, ocean bed and seabank.

Here the player grabbed two words straight out of the riddle, "bed" and "sea", and glued them together. It's a smart move that works surprisingly often on harder puzzles. This time the riddle was using those words in their other meanings. A river has a bank and a bed. It runs to the sea. It isn't the sea.

Why clever people take the wrong turn

None of this is new to people who study problem solving. Psychologists have been watching smart people get stuck like this for more than 80 years.

In 1942, a psychologist named Abraham Luchins gave people a series of water jar puzzles. The first few could only be solved with one fiddly method, so people learned that method. Then he slipped in puzzles with a much simpler answer. Most people kept using the fiddly method anyway, and some couldn't solve a puzzle that only the simple method could crack. They weren't being slow. They were being efficient, reusing the first thing that worked. The name for it is the Einstellung effect, from the German word for a mindset.

Around the same time, Karl Duncker described what became known as the candle problem. You're given a candle, a box of drawing pins and some matches, and you have to fix the candle to a wall. People get stuck because they see the box as a container for the pins. The answer is to empty it and pin the box itself to the wall as a little shelf. That trap, seeing an object only as the thing it's normally for, is called functional fixedness.

Our wrong answers are small versions of both.

"Elevator" is the Einstellung effect. The clue "goes from the first floor to the second floor" fires off a strong, fast match, and the brain runs with it. The clue that should stop you, "never move an inch", arrives a second too late. By then you've already decided.

"Seabed" is closer to functional fixedness. The words in a riddle feel like parts you're meant to build with, so you build. But riddles love to hand you a word and mean something else by it. A bank that holds no money. A bed nobody sleeps in.

There's one more finding we think about a lot. In 1987, Janet Metcalfe and David Wiebe asked people to keep rating how close they felt to solving a problem while they worked on it. On ordinary algebra, the "I'm getting warmer" ratings climbed steadily toward the answer. On insight problems, the kind riddles are, they didn't. People felt cold, cold, cold, and then they had it.

That matches what we see. Riddle locks rarely get "almost solved" in a way the player can feel. You're either staring at the screen or you've got it. It explains why a good riddle feels so satisfying when it lands, and why a bad one feels unfair: if the jump never comes, you have nothing to show for all that thinking.

The answers we let in, and the ones we didn't

Before Crack the Vault went live, we ran a test on all 300 questions. We sent them out with the answers and hints removed and asked a set of blind solvers for their answer, plus every other one-word answer that honestly fit all the clues.

Every question had a findable answer. But 12 questions came back with alternatives we hadn't thought of, and they were good ones. "Reflection" for mirror. "Atlas" for map. "Shadows" for darkness. "Keypad" for keyboard. "Donkey", "mule" and "camel" for the horse riddle, because the riddle never says which animal Friday is. All of those now open their locks.

Ten questions got reworded, because they could fairly be read two ways. The best example is in Vault 1:

A combination lock has 4 dials, and each dial shows the digits 0 through 9. Counting every dial, how many digits are printed on the lock in total?

The answer is 40. Four dials, ten digits each. Our first version didn't say "counting every dial", so someone could reasonably say 10, since there are only ten different digits. Ten was a fair answer to a badly written question, so we added the words to close the gap.

It hasn't fully worked. This lock has come up five times. One answer was 40. Two ran out of time. The other two were 10 and 9.

Ten is the old ambiguity, still alive. Nine is the "don't forget zero" trap, which is exactly what the hint warns about. So this is the lock in Vault 1 we're now looking at hardest. If the wording still trips people up after more players reach it, it gets rewritten again.

That's the rule we try to hold ourselves to. An answer should open a lock if it fits every clue, not just the first one. "Fingers" fits one clue out of three. "Replica" fits the painting but not the lie. "Elevator" fits everything except the bit that matters. All of those stay shut. "Reflection" for a mirror riddle fits every clue, so it opens.

And when the right answer and a wrong answer both fit every clue equally well, the riddle is broken, not the player.

Two we changed, and one we didn't

Some wrong answers made us stop and look again. Two of them have already changed the game.

Password, for secret. "I'm yours to keep only until you tell someone. Once you share me, I'm not really me anymore." One player answered password, and honestly, that's pretty good. A password you've told someone is a password that no longer does its job. We argued about it for a while and then let it in. As of today, password opens that lock.

Swapped letters. Our typo rule forgives one wrong letter. But some of the most common slips aren't wrong letters, they're swapped ones. Players typed candel for candle, and calander for calendar four separate times. To our matching rule, those are two mistakes each, so they stayed shut, while "calender", one letter off, sailed straight through. On a phone keyboard, with a clock ticking, that felt harsher than it should. So we changed the rule. On answers of six letters or more, one swapped pair of letters is now forgiven too, so candel and calander both open their locks. Swap two pairs and it's still a no.

Cities, for Maine. The Vault 10 riddle asks for the only US state with a one-syllable name. One player tried Chicago, then Los Angeles, then got there: Maine. The riddle is fine. It's a reminder that under a timer, people often answer the question they expected, not the one on the screen. Also, that it's fine to be wrong twice.

What else 560 answers told us

A few numbers from the first few weeks:

• About 60% of locks open. That's true for both the riddles and the maths, which surprised us. We expected one kind to be clearly harder.
• About one in seven locks times out. We can't see why, and we'd guess a few of those are a phone call or a kettle rather than a riddle winning.
• One run in three uses a hint. We're glad. The hints are written to nudge, not to give it away. "Don't forget zero" is a nudge.
• Vault 1 gets cracked about half the time. Vault 10 has been tried ten times and cracked once. That's roughly the curve we were aiming for when we sorted all 300 questions from easiest to hardest last week.

Again, these are small numbers. One player having a good night can move them. But they're real people, not our test runs, and they're already changing what we write.

Why we're telling you this

Most games treat wrong answers as noise. You got it wrong, you lose a life, move on.

We think they're the most useful thing a puzzle game collects. A right answer tells us you got there. A wrong answer tells us how you were thinking, and that tells us whether the riddle was playing fair with you.

So here's what we do with them. We look at the answers, never at who typed them. When the same wrong answer keeps coming up, we ask one question: does it fit every clue? If it does, it gets added to the list. If two answers fit equally well, the riddle gets rewritten. If it only fits part of the riddle, the lock stays shut, and we try to make the clue that rules it out a little harder to miss.

If you think a lock turned you down unfairly, tell us. Every lock has a small Report this question link under it for exactly this. We read every one.

Your turn

The stairs riddle is still waiting in Vault 8. So far it's four tries to zero, and now you know the answer, so you'll have to find a different lock to be the first to crack.

The other 299 locks are built the same way, with every clue there for a reason, usually to rule out the first answer that comes to mind.

Start with Vault 1. It's free, it needs no sign-up, and it takes a few minutes. Read every clue. Then read the last one again.