Anemone's Research Emporium
Where science and creativity collide
How big of a sample do you need?
Raters were asked to rate 34 female action heroes in films from the last 20 years on how professionally and practically they were dressed and how much the rater liked how they were dressed. (Study)
I had originally wanted to wait for between 20 and 30 ratings for each female action hero, because in my university Statistics for Biology class, we were told that was the optimal sample size to avoid skewing from unusual results. But somewhere in the teens would do in a pinch.
As it happens, that turns out to be plenty for most of the action heroes, though in some cases it looks like results haven't quite stabilized. And of course the results are only representative of the people who do the ratings. If a whole bunch of people from a more conservative (or easygoing, for that matter) segment of society showed up to do ratings, the numbers could shift in one direction or the other. At the same time, if they all shifted in the same way, the correlations probably wouldn't change much, since what matters is how the heroes are rated relative to each other.
The following are cumulative mean charts for some of the heroes in the sample. The first point is the average rating for the first rater; the second point is the average rating for the first two raters; the third point is the average rating for the first three raters; and so on. The average rating stabilizes after a while, so that future raters make less and less of a difference to the final average, even if their ratings are very different from other people's.
