Research: Reviewer Accuracy

Overview | Team | Plan | Reviewers | Program´╗┐

Highlights

  • Individuals with CEO experience (56% of the whole set) were 25% under-represented in the most accurate set.
  • Individuals with professional investing experience (41% of the whole set) were roughly on par for the top performers at spotting fails and overall accuracy, but were under-represented in the group that was best at picking winners.
  • Past Participants in the MCN program (16% of the whole set) were slightly better at picking companies that would go on to do well, and slightly worse at picking companies that would go on to do poorly.
  • Women (42% of the whole set) out-performed men in every category, most significantly in their ability to spot companies that would go on to fail.
  • Individuals based in the USA (88% of the whole set) were outperformed by our international brethren in their ability to spot failures, but were on par for their ability to pick winners.
  • Individuals with a legal education (6% of the whole set) did well in all categories.
  • Individuals with MBA degrees (41% of the whole set) were on par for their ability to spot failures, but did poorly compared to others in their ability to spot companies that went on to do well.
  • Individuals with any Masters degree or above (80% of the whole set) were on par for their ability to spot winner and failures, but slightly under par in representation in the most accurate category.
  • Individuals with doctorates (11% of the whole set) outperformed in every category to a significant degree.

Methodology

Within our time from of 2008 to 2018, 500 reviewers reviewed 5 or more business plans through our program. We compared the representation of individuals with the attributes listed below in the entire set to their representation in the top reviewers for picking successes and picking failures.

  • Past participation in the MCN Program as entrepreneurs
  • Gender
  • Formal education — JD, PHD, MBA, Any Masters Degree
  • USA-based or not USA-based
  • CEO Experience
  • Professional Investment Experience

A number of additional categories are tracked (non-binary genders, architecture / urban planning backgrounds, medical degrees, etc.), but none of them had enough representation to be statistically significant.

We tracked the success of those companies, measuring success by a formula that took into account longevity, revenue, funds raised, and employees hired. We then identified the top 10% individuals in each category who were the best at spotting eventual wins and failures,, as well as the top 10% most accurate overall, and compared the representation of the above attributes in that set to the total set, and accounting for overall optimism and pessimism.

Details

The percentages listed below are based on a set of 500 individuals who reviewed 4 or more companies for the MCN between 2008 and 2018. 100% means that the same percentage of individuals in that category were in the overall set and the listed top 10% set.

CategoryCEOInvestingPast ParticipantFemaleMaleUSALegalMBAPHDMasters & Up
Best at spotting failures82%98%90%133%76%89%125%102%130%101%
Best at spotting success89%68%115%100%100%102%125%68%148%93%
Best overall75%93%103%114%90%91%125%87%130%88%

Accuracy of Reviewers over Time

Reviewers get better the more they work with the MCN. A small increase in accuracy at spotting companies that go on to do well or poorly starts around the 5th cohort that reviewers work with the MCN, and continues until around the 15th cohort, after which it levels off.

Reviewer Bias

Our bias examinations take into account the for overall optimism / pessimism of the reviewer and the eventual success or failure of the company

  • Reviewers showed no significant bias for or against entrepreneurs that they felt were more (or less) like them in terms of shared values, skills, and mission.
  • We looked to see if our reviewers showed a bias for or against entrepreneurs when groups were compared by age, race, gender, and country of origin. In terms of mentor engagement (the resource that we distribute) there was little to no significant variance.