37% RULE - Why 'Optimal Stopping' is one of the best way to avoid further bad decisions in life?

Updated: Sep 10

Whether you’re choosing a candidate or a spouse or a storefront, this math-based trick can help you pick well.


When do you give up on information gathering and decide you have enough data to make following example of choices?

  • How many people should you date before deciding to ‘settle down’ and get married?

  • How many interviewee should we look at before deciding on "The One"?

  • How many flats should I look upon before saying "Yes" to one of the offers?

  • How many hotels should I look before deciding on the one to settle into?

  • How many internships should you do before deciding on a job offer?

  • How many careers should you try out before you figure out which your favourite is?

Above are the classic examples of a situation where you feel:

  • Spend too long probing your options and you fall prey to analysis paralysis and let promising opportunities go by.(Overly Exploratory)

  • Close out the process too quickly and you risk regretting options you never considered.(Overly Exploitative)

The answer to above situations is to follow a single mathematically derived magic number 37, no matter what domain your decision falls in.


“Be prepared to immediately commit to the first thing you see that's better than what you saw in that first 37%.” - Brian Christian


37% Rule
Image by Author - 37% Rule

MAIN IDEA


The 37% rule comes from ‘optimal stopping’ theory in mathematics, which determines the optimal time to take a particular action in order to maximize reward and minimize cost (aka, the best time to stop seeking more options and pull the trigger). According to mathematicians, that point is right after you’ve seen or explored the first 37% of your options.

Of course, you need to come up with your total universe first, to calculate what 37% of your options will be. You can do that either by setting a maximum cap or a time-based deadline.


Lets consider the example:


If you’re hunting for a new car, and decide you’d like to see 10 cars before making a decision, you should plan to see the first 3-4 with zero intention of buying them. After exploring those, your exploratory period has reached a point of diminishing returns, and the next car that is better than those initial three is the keeper.


It is true that matters of the heart aren't logical, and this rule only maximizes probabilities. It doesn't guarantee success. So if lightning strikes or the angels sing and you're absolutely sure you've found the perfect match for you, don't let abstract formulas stand in the way.


But even if it's unwise to blindly follow the 37 percent rule in every instance, that doesn't make it invalid or unhelpful. Knowing that spending roughly the first third of your decision-making process gathering information is generally optimal can help you resist both FOMO and analysis paralysis. Knowing that you should snap up the first candidate who beats the options you previously explored can help you muster the courage to pull the trigger.


The trick to great decision making is balancing exploration and decisiveness, and that's exactly what the 37 percent rule helps you do.


Of course, when it comes to large financial decisions and matters of the heart, this rule doesn’t account for feelings, “gut instincts,” instant chemistry, and the power of recommendation. (E.g., when there is a very attractive, hilarious person sitting across from your very single self who was vetted and suggested by a close friend, even if they are the first date you’ve gone on in a year.)


TECHNICAL (1/e Stopping Rule)

This oddly precise recommendation actually has a formula behind it and mathematicians insist this rule gives you the highest probability of making the optimum choice. If you're into math, it's actually 1/e ( 1 over e, where e is the basis of the natural logarithm), which comes out to 0.368, or 36.8 percent or 37% as the number of choices increases.


PSYCHOLOGICAL TEST on Exploit or Explore Conundrum

Mathematics offers us the best answer to the “optimal stopping problem.” But there’s just one big issue with it: Humans are not rational probability-crunching machines. In fact, the opposite is usually true. We’re beautifully, infuriatingly, creatively, and messily chaotic. So, it falls on psychology to tell us about how we actually behave.

In psychology and economics, there is what’s known as a “explore/exploit” tradeoff. This asks whether you should go with a guaranteed “win” (the exploit) or risk going somewhere else for an unknown outcome (explore). The degree to which someone will explore or exploit will depend on a host of factors, and it ties in with how curious or risk-seeking we are.

According to research by Addicott, published in Nature, the extremes of being too explorative or too exploitative leave us disadvantaged. The person who exploits too much “could promote habit formation,” while the person who explores too much risks becoming a “jack of all trades, but master of none.” Over-exploiters become stuck in a rut, lacking motivation, and get bored. Over-explorers lack expertise and never fully experience anything in depth. What Addicott and his team concluded is “the most advantageous behaviors occurring around a point of balance between the two.”

Of course, different people are more explorative or exploitative at different times. Teenagers and entrepreneurs tend to explore more. Adults and managers exploit more. Try testing yourself with these three questions, to see where you fall:

  • If you are visiting a city you know vaguely well, will you go to a restaurant you know is nice, or will you try somewhere new?

  • If I tell you a gambling machine has a payout of $50, will you stay and play or explore to see if others have a bigger payout?

  • When you’re playing a game, do you tend to stick to the same tactics or mix it up each time?

“True optimization is the revolutionary contribution of modern research to decision processes.” - George Dantzig


USE CASES


1) SECRETARY PROBLEM OR HIRING PROBLEM

A historical problem that derived this rule called "Secretary problem" at around 1960.

Let’s suppose you want to hire a secretary. You want to find the best person for the job, but you won’t know how good someone is until you interview them. At that point, you can choose to hire them (a permanent decision) or to move on and interview someone else. The only catch is that once you’ve rejected someone, you’re not allowed to go back and hire them.

How do you maximise your chances of finding the right person?

If you're hiring a secretary and plan to interview 10 candidates, let the first three go by. Then choose the next person who beats that first bunch.

Secretary Problem
Image by Author - Secretary Problem

Basically if you knew you were interviewing 100 potential secretaries, you’d interview and reject the first 37, and you’d keep in mind who your best-so-far candidate was as a benchmark. From interview 38 onwards, you’d immediately hire the person better than your best-so-far candidate. This is apparently how you maximise your chances of hiring the best overall candidate.

Secretary Problem - Sampling
Image by Author - Secretary Problem - Sampling

Of course, real-world situations don’t always conform so neatly to math theories. Not knowing how many applicants you’ll get for an open position would make it hard to know when to stop interviewing and start hiring. Still, imposing some structure and limits to a process that can too often drag on can only benefit you — and will kill your FOMO.


2) MARRIAGE PROBLEM

So let's think over it for sometime, what essentially the solution says is that if you have a consideration set of around 10 girls before you decide which one to marry, you should not commit to anyone, until you have looked through at least 4 of them. Post which you should be ready to commit to anyone who is better than all previous ones.


If you're hoping to be settled down by 40 and start dating in high school, take the first third of your dating life -- up to around age 25 -- to casually check out your options.


The magical insight is from Logan Ury’s book (How to die not alone) is that we don’t need to think about it in terms of numbers - we can think about it in terms of time.


It’s very hard to know how many people we could end up feasibly dating, but it’s easy enough to estimate how much time we want to spend dating.


If for example, I’m open to dating between age 18-40 (and assuming there’s no radical change in the number of people I’m getting to know each year), the 37% rule says that when I hit the age of 26, I should marry the next best person.


3) REAL ESTATE PROBLEM

Brian Christian in his book, Algorithms to Live By: The Computer Science of Human Decisions, uses the 37% rule to help Macy to hunt best apartment for rent. As he writes: “If you want the best odds of getting the best apartment, spend 37% or first third of your apartment hunt (eleven days, if you’ve given yourself a month for the search) noncommittally exploring options. Leave the check book at home; you’re just calibrating. But after that point, be prepared to immediately commit—deposit and all—to the very first place you see that beats whatever you’ve already seen. This is not merely an intuitively satisfying compromise between looking and leaping. It is the provably optimal solution.”


CONCLUSION

Following this rule will save you from getting unnecessarily mired in information gathering and data analysis, get you into action, and maximize your probability of success.

While it won’t apply to every situation, and you shouldn’t discard what are clearly superior options simply because of a mathematical theory, if you have a tendency to make hasty but regrettable decisions, or spend too long investigating all your options, it’s a helpful rule of thumb. The next time you’re faced with competing choices, remember: Roughly the first third of your decision-making process should be information gathering, after which time, selecting the next great option you encounter is optimal.


I am sure that there are many who are already aware of this, but still publishing a post to reach out to as many as possible who don't know about Optimal Stopping problem and its solution, and are going through their lives trying to reinvent the wheel.


TLDR:

  • When trying to pick the best among many options, how many samples should you try before you commit? This is known as the optimal stopping problem.

  • Mathematicians tell us that, to maximize the chances of the best outcome, we ought to ditch the first 37% of any options.

  • In psychology, people tend to either "explore" or "exploit" more. But, sadly for us, relationships are a bit messier than probability would have it.


“Life is Quicker Than a Blink of an Eye” - Jimi Hendrix


“The best kinds of failures are quick, cheap, and early, leaving you plenty of time and resources to learn from the experiment and iterate your ideas.” - Tom Kelley


“Better a good decision quickly than the best decision too late.” - Harold Geneen


Mathematical proof of the Rule : https://plus.maths.org/content/solution-optimal-stopping-problem

Thanks for reading…


Found this post useful!!! Subscribe for free to join the community of curious subscribers to get update on the best curated posts from the internet.

200 views0 comments