4.3: The Geometry of OLS

The real purpose of this section is build on the previous section and to illustrate OLS in terms of geometry. This is because we have our model (\(\mathbf{\hat{Y}}\)) and reality (\(\mathbf{Y}\)) as in the figure below. The figure below gives a lot of insight into what is happening in OLS. Actually, this figure tells us that the OLS estimator is optimal in the sense of being closest to reality. All it takes is for us to understand geometry and linear algebra.

✦•················• ✦ •··················•✦

Figure \(\PageIndex{1}\): Schematic of OLS showing that it is optimal in the sense that the OLS estimators are the closest of all points in the reduced space to the true value.

The Inherent Limitation of Statistical Models

In any real-world scenario, an outcome we wish to understand (the dependent variable, \(\mathbf{Y}\)) is influenced by a vast and complex web of factors. Think of something like house prices, a patient's blood pressure, or a Ruritania's stock performance. In a perfect world, to fully explain \(\mathbf{Y}\), we would need to account for every single one of these influencing factors (the independent variables).

However, this is practically impossible. We can never identify or measure every possible influence. Therefore, when we build a statistical model — like an Ordinary Least Squares (OLS) regression — we are making a conscious choice to simplify reality. We select a limited set of what we believe are the most important variables to include in our model.

Note

In essence, we are restricting our view from the immense complexity of reality to a more manageable, but incomplete, subset of factors.

This observation gave rise to George E. P. Box's observation that all models are wrong, but some are useful.

A Geometric Interpretation of Our Model

The figure above offers a way to visualize this. Imagine that in "Reality," the dependent variable \(\mathbf{Y}\) is truly determined by three key independent variables. We can think of this full reality as existing in a three-dimensional space (3-space). Now, imagine that our statistical model, either by choice or by necessity, only uses only two of these three variables. Geometrically, this means our model is confined to a two-dimensional plane within the larger three-dimensional space.

• \(\mathbf{Y}\) represents the actual, observed value, which exists as a point in the full 3D space, above (or below) our model's 2-space plane.
• \(\mathbf{\hat{Y}}\) is the prediction made by our OLS model. By definition, \(\mathbf{\hat{Y}}\) must lie within the restricted 2-space defined by the two variables we chose.

The Best Possible Approximation

So, how does the OLS estimator find \(\mathbf{\hat{Y}}\)? It operates under the constraint of our model. It cannot leave the 2-space (the plane). Its goal is to find the point within that plane that is closest to the real-world \(\mathbf{Y}\). Here, this "closeness" is measured by the vertical distance between the point \(\mathbf{\hat{Y}}\) and the plane — this is the residual (or error) of our model.

Therefore, the OLS estimate \(\mathbf{\hat{Y}}\) is the optimal projection of reality onto our simplified model.

It is the best approximation we can get while remaining within the limitations we've imposed on ourselves. It acknowledges that our model is incomplete, but it ensures that, given its limitations, it is as accurate as it can possibly be. The gap between \(\mathbf{Y}\) and \(\mathbf{\hat{Y}}\) is a visual representation of all the influences we failed to include in our model.