Two decades after first beginning to experiment with the concept of publicly funded but independently managed charter schools, we still don’t know much about their effectiveness in creating a better model for learning, according to UC San Diego’s Division of Social Sciences.

“[M]ost policymakers don't have sufficient data on charter schools to decide whether they're successes to be replicated or disasters to be shut down,” says UCSD educational economist Julian Betts.

The problem in evaluating charter schools is the method with which most studies are performed – usually comparing test scores and student performance to those at a nearby traditional public school. But this method of evaluation is severely flawed, say Betts and others involved in his research, including University of California president emeritus Richard Atkinson and UCSD economist Emily Tang. Betts and Tang discarded 75 percent of the studies they encountered in an analysis of currently available literature on charter schools for failing to account for differences in the backgrounds and academic histories of students who chose charter schools and those who chose to attend the nearest traditional public school.

While most of these studies show charter schools outperforming the traditional model, the factor of self-selection is important, researchers say. The fact that some students (or their parents) have chosen to seek out additional education opportunities, and are willing to face challenges such as applying to and traveling to and from a charter that may be less accessible than a neighborhood school, may be an indicator itself that this group of students is likely to perform at a higher-than-average level. For this reason, a proper comparison would be between students who chose to go to a charter school and actually attended, versus those who did not.

For this reason, the best available data concerns charter school lotteries – instances where more pupils apply to attend a school than will be admitted. The peer group denied admission can then be studied and compared to those who do attend, providing a study group comprised of more similar subjects. Unfortunately only about 90 such studies are available, covering only about two percent of charter schools nationwide.

In these studies, the charter schools still come out ahead, with students performing better than their peers who didn’t “win the lottery” and gain admission. But there’s another caveat – schools that need to hold lotteries in order to limit student populations are likely to be superior to those who don’t have overcrowding problems, which is what makes them so desirable and also why it’s to be expected that their students would outperform others. A recent U.S. Department of Education survey, however, showed that only 130 of 492 charter middle schools needed to conduct admission lotteries, and only 77 of those agreed to make lottery information available to researchers.

“We need more lottery-based studies and we need to be able to do longitudinal work,” Betts says. He believes that higher-order learning skills and behaviors need to be taken into account when studying charters, as well as long-range follow-up including tracking such factors as college graduation rates. Further, Betts suggests that charter laws be revised so that all charter schools are compelled to share lottery data with state departments of education to facilitate better studies on what does and doesn’t work about charters.

“Taking these steps would improve research, not only on charter schools but on all public education,” concludes Betts.

Pictured: UCSD student tutoring at Preuss charter school