The Induction Controversy

The controversy is subtler than I explain here, but knowing a bit about the big picture is useful. If you want to delve deeper this is more the subject of philosophy than statistics, and certainly more the subject of philosophy than the statistics we study in this class. It revolves around the extent to which we can take what we observe - sometimes using many observations which is better as we see in Chapter 5 - and using those observations to know something of the world. For example I see the sun rise each morning, I might predict that it will continue to do so. And I would be correct, you really can learn something from observation. But there are situations where observation is not really enough.

The idea of learning everything from only careful observation, strongly adhered to by Sir Francis Bacon, was influential for a time. An example by Bertrand Russell exposes the problem with this approach. Russell suggested that the induction believer "Suppose oneself to be a turkey". A turkey on the farm would learn, through careful observation, that every day the farmer came to his pen in the morning it was to feed him. Day after day this would occur, the power of observation would make it clear that this is what a farmer does and this is how the turkey's world works. All observation suggests that this is true, but then comes Thanksgiving. The turkey is wrong!

The problem in the preceding example is that although being fed each morning is like the sun coming up each morning, there is actually a bit more going on as to why the observations are as they are. In the case of the turkey, there is additional meaning in why the farmer is feeding the turkeys, and one does not understand this by pure observation of just the feeding each day. In a sense this is correlation is not causality - understanding what is going on makes the every day feeding sensible and clear, but simply forecasting what will happen the next day requires an understanding (theory) of why all this is happening. We need this with the sun as well, but the story there still has the sun rising tomorrow. Issues of correlation vs causality are more for later in the course sequence, but we will discuss it.

Another weakness of observation only is that what we learn might not extend to other situations we want to learn about. For example many economists run experiments in villages in Africa or India to try and determine what types of help (money, resources, better education etc.) actually work in the sense of leading to better outcomes. If we only observe without theory, we might see that an intervention works well in a set of villages, but we are still unsure whether or not this same intervention will work in other villages, that might be for example in other parts of the world. This idea is known in statistics as 'extendability', we see it in Chapter 6. It is an important worry in rolling out programs and the answer really does not come from simple observation, we need more.

The link with machine learning is that most machine learning is purely predictive, and so does not neccessarily lead to insight about the problem. It is catching regularities, which is really useful and important. For example a machine learning algorithm that predicts that a Spanish speader saying 'Hola' is related to English 'Hello' is both correct and useful. But because they are predictions without really learning anything about underlying structures, there is much more to be done.

As for the actual philosophy part, I will leave that to the philosophers. Hume spend a good deal of thought worrying about how we can really believe beyond what we observe. In this course we make all of this mathematical.