Rumours of the end of theory are greatly exaggerated

There was a suggestion that the use of theory is at an end, based on an uncited report from Google. There have been a number of rebuttals and comments scattered around, including here; but what I want to say here is that this has historical precedents that have to be watched closely.

The proposal is that by using enough statistical correlations, theory is no longer required. And in support it is pointed out that Google can now translate between natural languages, and serve up targeted advertisements merely by looking at the words one uses in pages and where one looks on the web, and the number of times certain words are used in close proximity. In other words, let’s go back to the serendipitous discovery techniques that held before the renaissance created the notions of underlying theory on which one can base future projections.

What a step backwards! It is exactly the mindset that was in operation during the Dark Ages, when knowledge seemed to be the accumulation of facts without much in the way of innovation, since a new invention needs some notion of predicted behaviour based on a, possibly intuitive, understanding of what is happening. Development was slow because it relied on the accidental discovery of phenomena in particular circumstances, like the smelting of iron, or the making of paper. There will always be a place for accidental observations (like the discovery of penicillin), but we no longer rely solely on them.

Theories are the foundation of modern scientific thought and are what have brought the engineering successes over the last 300 years. They will also be necessary to make progress in medical fields, and eventually, ethical fields, too. We must not lose them.