Uncontrolled by Jim Manzi

Boosterism for the history and application of randomized control trials.

James Lind is conventionally credited with executing the first crude but recognizable clinical trial. In 1747 he divided twelve scurvy stricken crew members on the British ship Salisbury into six treatment groups of two sailors each.

Credit for formally integrating this technique into clinical trials to create the modern randomized field trial (RFT), in which patients are assigned randomly to the test and control groups, is usually given to Sir Austin Bradford Hill, the statistician for a set of British clinical trials on the use of streptomycin to treat pulmonary tuberculosis in the 1940s.

the brilliant but erratic American polymath C. S. Peirce was the first person to discover the solution to this problem when he executed a randomized, blinded experiment in 1884 to test perceptions of small differences in weights (though the concept of assigning test treatments by drawing lots had been discussed hypothetically as early as the late 1600s by Flemish physician Van Helmont).

In a well-known 2005 paper, Dr. John Ioannidis, a professor at the Stanford School of Medicine, evaluated the reliability of forty-nine influential studies (each cited more than 1,000 times) published in major journals between 1990 and 2003 that reported effective interventions based on either experimental or nonexperimental methods. More than 80 percent of nonrandomized studies that had subsequent replications using stronger research designs were contradicted or had found stronger effects than the replication, whereas this was true for only 10 percent of findings shown initially in large RFTs.

His name was Rich Fairbank. In 1988 he and another SPA consultant named Nigel Morris went on to start the credit card company Capital One. Fairbank and Morris believed that applying the experimental method to business could go way beyond the factory, and set out to prove it, breaking down the process of operating a credit card company into discrete steps, and using constant controlled experimentation to improve performance.

in 1991 Heckman produced Randomization and Social Policy Evaluation, an article that after twenty years remains unsurpassed in its clarity and rigor in describing the limitations of the naive use of RFTs to measureand more important, predictprogram effects.

Leading Northwestern University education researcher and experimental expert Thomas D. Cook has identified only three randomized assignment experiments in education in the 1960s and 70s wave of social experiments: (1) the famous Perry Preschool experiment; (2) a school desegregation experiment with twelve students assigned to the test group; and (3) a test that provides some homes and not others with the TV program Sesame Street.

Probably the best-known randomized experiment in this period was the Tennessee STAR program, which tested the effect of smaller class sizes and is typically interpreted as showing some benefit from decreasing class sizes to about fifteen students in very early grades with experienced teachers. Like the original mandatory-arrest experiment, this program has been the purported basis for large expenditures in multiple states, but to my knowledge it has not been replicated in a large-scale RFT in the United States

As recently as 2001, Professor Cook could assert that no randomized experiments had been completed for the effects of most of the key contemporary schooling interventions: standards-setting, effective schools, movement to Catholic schools, Accelerated Schools, total quality management, charter schools, and smaller schools. He called out school vouchers, in which parents receive a voucher to pay for tuition at any private school, as a partial exception.

The early results have been impressive andconsistent with the introduction of randomized experiments to therapeutics, business, criminology, and social welfaremostly negative. The IES Directors Biennial Report to Congress in November 2008 called out a handful of programs that have shown some success, in a sea of widely touted programs that have not demonstrated any improvement at all.

For example, the IES sponsored a series of RFTs that tested fourteen well-known preschool curricula, and found only one curriculum that demonstrated some causal gains in performance that persisted through kindergarten (in one trial).

many of the research projects at the Ed Labs at Harvard, which under economics professor Roland Fryer Jr. conducts RFTs of school interventions;

Many things about APT turned out differently than we had expected. Settling on the SaaS delivery method was just one example and in fact was not even the most central; it is just a simple one to explain. The Hayekian knowledge problem is not a mere abstraction. Our innovations that have driven the greatest economic value uniformly arose from iterative collaboration between us and our customers to find new solutions to hard problems.

He can thrive as long as he keeps upgrading his skills rapidly enough to continue to add value as yesterdays innovations become tomorrows commodities.

Social evolution requires variation in institutions, plus a selection method that tends to retain more adaptive institutions. If we are to accelerate relevant learning, the first order of business is to encourage sufficient variation, cross-pollination of ideas, and selection pressure to encourage progress.