Hidden Lessons: Survivor Bias at Camp

During World War II, the Statistical Research Group (SRG), a classified program filled with some of the brightest minds, including the famous mathematician Abraham Wald, were tasked with figuring out how to better protect the bombers from the horrific causality rates they were sustaining.

The initial approach was straightforward: Look at the planes that returned and reinforce the areas that were most commonly hit by enemy fire. The logic seemed impeccable — look at where bullets were damaging plans and reinforce these areas. But it was Wald who insightfully pointed out that the military should instead reinforce the areas where the returning planes were unscathed. His rationale was that the planes were returning despite being hit in the areas riddled with bullet holes, indicating that these sections could sustain damage. The areas without damage on the returning planes were likely the ones that, if hit, would prevent the planes from returning at all. Wald’s counterintuitive advice to armor the undamaged sections proved to be a significant breakthrough.

Survivor bias is therefore a logical error where we paid attention to only the entities that have passed through (Survived) a selective filter, which often leads us to an incorrect conclusion.  For you with a science background this is a type of sampling bias.

At Camp the selection filter is returning for another summer.  This could be staff or Campers.  But because a large group of people come back year after year, we think everything is great.  These are the campers and staff that we start to build relationships with, and their excitement becomes our opinions.  But are we up armoring the wrong parts of our camp plane?

When I came back to camping, I arrived mid-summer and everyone was busy.  No one really had time to on board me and I didn’t have much to do so I started talking to people.  Parents, kids, and staff.  Everyone loved the camp. But the Camp had experience 5 years of falling enrollment.  It didn’t make sense.

So, I put together a survey on Survey Monkey.  [My survey was very detailed and took about 10 minutes to complete which is far too long.  Make surveys quick and ask if people would be interested in doing a more in depth survey.]  it really dug into what was happening at camp.  And I sent it to the campers that had just left.  The results were very positive, with some expected detractors.  So I did the same at the end of the current session.  But new to my role I made an error pulling the email list and accidentally pulled the previous year’s email list.  It was a fucking disaster. Over half the surveys were negative and about half of those negative ones were actually angry.  Suddenly our falling enrollment and low return rate made sense.

I didn’t know what to do.  The results were truly devastating.  We had a new summer director and he had already made some changes, but I was reluctant to show people the results until I knew how to move forward.  The problems seemed wide ranging and deep.  I showed my new boss, who was also new in his role, he agreed we need to make changes rapidly. When we did and the results came quickly incredible.

But that’s not a tale for right now.  Because before this got better it got a lot worse.

Once we had this information, I decided to start calling families.  But only families that sent their campers once and never again.  By about my 20th call which I had spread out over a couple of days I had a pretty bleak picture of our camp.  I was emotionally spent.

A camp’s problems are always unique to that camp so its not worth going into what we discovered in any detail. The picture that emerged can only be described as a bifurcated camper experience.  If you fit the mold or came with a friend that had experienced camp, you loved it.  But some of the parents of kids that felt like ‘outsider’ were describing an experience was close to traumatic.

The camp had built it identify around being “in on it.”  There were jokes and traditions that were fun a lot of fun. If you knew what to expect.  Kids who had been to camp knew what to bring and wear on certain days.  They knew how program selection worked, and gamed it.  They knew the best bunks, best places to sit for meals.  They loved it.  But their joy and the staff that facilitated it was often at the expense of the kids that didn’t get it.  They were miserable unable to do the activities they wanted. Though staff would tell them they would be ready next year.  They simply didn’t come back.  It was bullying.

Our staff retention was low as well.  For many of the same reasons.  Potentially talented staff were not invited back or self-selected out because they were outsiders.

Please understand it wasn’t all terrible. There were incredible things happening at the camp, and it was doing great work.  But it wasn’t inclusive and the changes we made were to make the camp truly for all. Most of these changes happened before the summers end and our enrollment rates week to week started to climb.

The changes were small.  WE simply explained in advance, and in more detail how the kids selected their activities.  We increased the number of activities that the cabins did as a group so friendship could organically develop.  We reduced the number of kids you could arrange to be in a cabin with before arrival, and made sure both sets of parents were supportive.  Tiny changes that gave everyone equal standing

As I visit other camps and talk to Directors, I ask them if they ever call the kids that never come back.  They don’t, camps not for everyone.  But those that do realize that they two have survivor bias.  That they are looking at the returners, here their story’s and make that there narrative.  If camp is for all, then we need to make it a place for everyone.

As with so many things, camp teaches us a crucial lesson that extends far beyond the boundaries of summer experiences and into the very fabric of our daily lives and decision-making processes. Survivorship bias is not just a phenomenon in business, history, or war stories; it is a subtle, yet pervasive force that shapes our perceptions, decisions, and the narratives we choose to believe and propagate.

By recognizing the voices not heard, the stories not told, and the experiences overlooked, we open the door to a deeper understanding and a more inclusive approach to every aspect of our lives. Whether it's your camp program, or a string of bad relationships, the principle remains the same: true insight and progress lie in considering the full spectrum of experiences, not just your own and not just successful ones.

Call the people that don’t come back.

On a lighter note.  When your camp “X” breaks and you lament ‘oh they don’t make them like they used to.’  That’s survivor bias at play. The perception that something from years ago is still working today must mean that things from the 80’s were better and more durable and built to last longer. 

It doesn’t mean that.  It means your old camp director was cheap and didn’t upgrade things.  Technology has made quality control more effective.  The fact that one item lasted far longer than expected plus nostalgia leads us to think the past was better.  It wasn’t. The old days were not as good or as bad as your alumni tell you. Except the 90’s.  The 90’s were the pinnacle of human evolution.  The music, the movies, the people.  We have been in decline ever since.