11  Multivariate Regression

11.1 Introduction

Multivariate regression is a system of regression equations. Multivariate regression is used as reduced form models for instrumental variable estimation (Chaper 12), vector autoregressions (Chapter 15), demand systems (demand for multiple goods), and other contexts.

Multivariate regression is also called by the name systems of regression equations. Closely related is the method of Seemingly Unrelated Regressions (SUR) introduced in Section 11.7.

Most of the tools of single equation regression generalize to multivariate regression. A major difference is a new set of notation to handle matrix estimators.

11.2 Regression Systems

A univariate linear regression equation equals $Y=X^{\prime }\beta +e$ where $Y$ is scalar and $X$ is a vector. Multivariate regression is a system of $m$ linear regressions, and equals

$$Y_j=X_j^{\prime }{\beta }_j+e_j$$

for $j=1,\dots ,m$. Here we use the subscript $j$ to denote the $j^{th}$ dependent variable, not the $i^{th}$ individual. As an example, $Y_j$ could be expenditures by a household on good category $j$ (e.g., food, housing, transportation, clothing, recreation). The regressor vectors $X_j$ are $k_j\times 1$ and $e_j$ is an error. The coefficient vectors ${\beta }_j$ are $k_j\times 1$. The total number of coefficients are $\overline{k}=\sum _{j=1}^m{k}_j$. The regressors can be common across $j$ or can vary across $j$. In the household expenditure example the regressors $X_j$ are typically common across $j$, and include variables such as household income, number and ages of family members, and demographic characteristics. The regression system specializes to univariate regression when $m=1$.

Define the $m\times 1$ error vector $e={(e_1,\dots ,e_m)}^{\prime }$ and its $m\times m$ covariance matrix $\mathrm{\Sigma }=\mathbb{E}\left[e{e}^{\prime }\right]$. The diagonal elements are the variances of the errors $e_j$ and the off-diagonals are the covariances across variables.

We can group the $m$ equations (11.1) into a single equation as follows. Let $Y={(Y_1,\dots ,Y_m)}^{\prime }$ be the $m\times 1$ vector of dependent variables. Define the $m\times \overline{k}$ matrix of regressors

$$\overline{X}=\left(\begin{array}{cccc}{X}_1^{\prime }& 0& \cdots & 0\\ ⋮& X_2^{\prime }& & ⋮\\ 0& 0& \cdots & X_m^{\prime }\end{array}\right)$$

and the $\overline{k}\times 1$ stacked coefficient vector

$$\beta =\left(\begin{array}{c}{\beta }_1\\ ⋮\\ {\beta }_m\end{array}\right)$$

The $m$ regression equations can be jointly written as

$$Y=\overline{X}\beta +e.$$

This is a system of $m$ equations.

For $n$ observations the joint system can be written in matrix notation by stacking. Define

$$\mathit{Y}=\left(\begin{array}{c}{Y}_1\\ ⋮\\ Y_n\end{array}\right),\mathit{e}=\left(\begin{array}{c}{e}_1\\ ⋮\\ e_n\end{array}\right),\bar{\mathit{X}}=\left(\begin{array}{c}{\overline{X}}_1\\ ⋮\\ {\overline{X}}_n\end{array}\right)$$

which are $mn\times 1,mn\times 1$, and $mn\times \overline{k}$, respectively. The system can be written as $\mathit{Y}=\bar{\mathit{X}}\beta +\mathit{e}$.

In many applications the regressor vectors $X_j$ are common across the variables $j$, so $X_j=X$ and $k_j=k$. By this we mean that the same variables enter each equation with no exclusion restrictions. Several important simplifications occur in this context. One is that we can write (11.2) using the notation

$$Y={\mathit{B}}^{\prime }X+e$$

where $\mathit{B}=({\beta }_1,{\beta }_2,\cdots ,{\beta }_m)$ is $k\times m$. Another is that we can write the joint system of observations in the $n\times m$ matrix notation $\mathit{Y}=\mathit{X}\mathit{B}+\mathit{E}$ where

$$\mathit{Y}=\left(\begin{array}{c}{Y}_1^{\prime }\\ ⋮\\ Y_n^{\prime }\end{array}\right),\mathit{E}=\left(\begin{array}{c}{e}_1^{\prime }\\ ⋮\\ e_n^{\prime }\end{array}\right),\mathit{X}=\left(\begin{array}{c}{X}_1^{\prime }\\ ⋮\\ X_n^{\prime }\end{array}\right)$$

Another convenient implication of common regressors is that we have the simplification

$$\overline{X}=\left(\begin{array}{cccc}{X}^{\prime }& 0& \cdots & 0\\ 0& X^{\prime }& & 0\\ ⋮& ⋮& & ⋮\\ 0& 0& \cdots & X^{\prime }\end{array}\right)={\mathit{I}}_m\otimes X^{\prime }$$

where $\otimes$ is the Kronecker product (see Appendix A.21).

11.3 Least Squares Estimator

The equations (11.1) can be estimated by least squares. This takes the form

$${\hat{\beta }}_j={\left(\sum _{i=1}^n{X}_{ji}{X}_{ji}^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_{ji}{Y}_{ji}\right).$$

An estimator of $\beta$ is the stacked vector

$$\hat{\beta }=\left(\begin{array}{c}{\hat{\beta }}_1\\ ⋮\\ {\hat{\beta }}_m\end{array}\right).$$

We can alternatively write this estimator using the systems notation

$$\hat{\beta }={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left({\bar{\mathit{X}}}^{\prime }\mathit{Y}\right)={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{Y}_i\right)$$

To see this, observe that

$$\begin{array}{rl}{\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}& =\left(\begin{array}{ccc}{\overline{X}}_1^{\prime }& \cdots & {\overline{X}}_n^{\prime }\end{array}\right)\left(\begin{array}{c}{\overline{X}}_1\\ ⋮\\ {\overline{X}}_n\end{array}\right)\\ & =\sum _{i=1}^n{\overline{X}}_i^{\prime }{\overline{X}}_i\\ & =\sum _{i=1}^n\left(\begin{array}{cccc}{X}_{1i}& 0& \cdots & 0\\ ⋮& X_{2i}& & ⋮\\ 0& 0& \cdots & X_{mi}\end{array}\right)\left(\begin{array}{cccc}{X}_{1i}^{\prime }& 0& \cdots & 0\\ ⋮& X_{2i}^{\prime }& & ⋮\\ 0& 0& \cdots & X_{mi}^{\prime }\end{array}\right)\\ & =\left(\begin{array}{ccc}\sum _{i=1}^n{X}_{1i}{X}_{1i}^{\prime }& & 0\\ ⋮& & \sum _{i=1}^n{X}_{2i}{X}_{2i}^{\prime }& & & \\ 0& & 0& \cdots & \sum _{i=1}^n{X}_{mi}{X}_{mi}^{\prime }\end{array}\right)\end{array}$$

and

$$\begin{array}{rl}{\bar{\mathit{X}}}^{\prime }\mathit{Y}& =\left(\begin{array}{ccc}{\overline{X}}_1^{\prime }& \cdots & {\overline{X}}_n^{\prime }\end{array}\right)\left(\begin{array}{c}{Y}_1\\ ⋮\\ Y_n\end{array}\right)\\ & =\sum _{i=1}^n{\overline{X}}_i^{\prime }{Y}_i\\ & =\sum _{i=1}^n\left(\begin{array}{cccc}{X}_{1i}& 0& \cdots & 0\\ ⋮& X_{2i}& & ⋮\\ 0& 0& \cdots & X_{mi}\end{array}\right)\left(\begin{array}{c}{Y}_{1i}\\ ⋮\\ Y_{mi}\end{array}\right)\\ & =\left(\begin{array}{c}\sum _{i=1}^n{X}_{1i}{Y}_{1i}\\ ⋮\\ \sum _{i=1}^n{X}_{mi}{Y}_{mi}\end{array}\right).\end{array}$$

Hence

$$\begin{array}{rl}{\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left({\bar{\mathit{X}}}^{\prime }\mathit{Y}\right)& ={\left(\sum _{i=1}^n{\overline{X}}_i{\overline{X}}_i^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i{Y}_i\right)\\ & =\left(\begin{array}{c}{\left(\sum _{i=1}^n{X}_{1i}{X}_{1i}^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_{1i}{Y}_{1i}\right)\\ ⋮\\ {\left(\sum _{i=1}^n{X}_{mi}{X}_{mi}^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_{mi}{Y}_{mi}\right)\end{array}\right)\\ & =\hat{\beta }\end{array}$$

as claimed. The $m\times 1$ residual vector for the $i^{th}$ observation is ${\hat{e}}_i=Y_i-{\bar{\mathit{X}}}_i^{\prime }\hat{\beta }$. The least squares estimator of the $m\times m$ error covariance matrix is

$$\hat{\mathrm{\Sigma }}=\frac{1}{n}\sum _{i=1}^n{\hat{e}}_i{\hat{e}}_i^{\prime }.$$

In the case of common regressors, the least squares coefficients can be written as

$${\hat{\beta }}_j={\left(\sum _{i=1}^n{X}_i{X}_i^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_i{Y}_{ji}\right)$$

and

$$\hat{\mathit{B}}=({\hat{\beta }}_1,{\hat{\beta }}_2,\cdots ,{\hat{\beta }}_m)={\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1}\left({\mathit{X}}^{\prime }\mathit{Y}\right).$$

In Stata, multivariate regression can be implemented using the mvreg command.

11.4 Expectation and Variance of Systems Least Squares

We can calculate the finite-sample expectation and variance of $\hat{\beta }$ under the conditional expectation assumption

$$\mathbb{E}[e\mid X]=0$$

where $X$ is the union of the regressors $X_j$. Equation (11.7) is equivalent to $\mathbb{E}\lfloor Y_j\mid X\rfloor =X_j^{\prime }{\beta }_j$, which means that the regression model is correctly specified.

We can center the estimator as

$$\hat{\beta }-\beta ={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left({\bar{\mathit{X}}}^{\prime }\mathit{e}\right)={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{e}_i\right)$$

Taking conditional expectations we find $\mathbb{E}[\hat{\beta }\mid \mathit{X}]=\beta$. Consequently, systems least squares is unbiased under correct specification.

To compute the variance of the estimator, define the conditional covariance matrix of the errors of the $i^{th}$ observation $\mathbb{E}[e_i{e}_i^{\prime }\mid X_i]={\mathrm{\Sigma }}_i$ which in general is a function of $X_i$. If the observations are mutually independent then

$$\mathbb{E}[\mathit{e}{\mathit{e}}^{\prime }\mid \mathit{X}]=\mathbb{E}[\left(\begin{array}{cccc}{e}_1{e}_1^{\prime }& e_1{e}_2^{\prime }& \cdots & e_1{e}_n^{\prime }\\ ⋮& \ddots & & ⋮\\ e_n{e}_1^{\prime }& e_n{e}_2^{\prime }& \cdots & e_n{e}_n^{\prime }\end{array}\right)\mid \mathit{X}]=\left(\begin{array}{cccc}{\mathrm{\Sigma }}_1& 0& \cdots & 0\\ ⋮& \ddots & & ⋮\\ 0& 0& \cdots & {\mathrm{\Sigma }}_n\end{array}\right)\text{. }$$

Also, by independence across observations,

$$\mathrm{var}[\sum _{i=1}^n{\overline{X}}_i^{\prime }{e}_i\mid \mathit{X}]=\sum _{i=1}^n\mathrm{var}[{\overline{X}}_i^{\prime }{e}_i\mid X_i]=\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}_i{\overline{X}}_i.$$

It follows that

$$\mathrm{var}[\hat{\beta }\mid \mathit{X}]={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}_i{\overline{X}}_i\right){\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}.$$

When the regressors are common so that ${\overline{X}}_i={\mathit{I}}_m\otimes X_i^{\prime }$ then the covariance matrix can be written as

$$\mathrm{var}[\hat{\beta }\mid \mathit{X}]=({\mathit{I}}_m\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1})\left(\sum _{i=1}^n({\mathrm{\Sigma }}_i\otimes X_i{X}_i^{\prime })\right)({\mathit{I}}_m\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1})$$

If the errors are conditionally homoskedastic

$$\mathbb{E}[e{e}^{\prime }\mid X]=\mathrm{\Sigma }$$

then the covariance matrix simplifies to

$$\mathrm{var}[\hat{\beta }\mid \mathit{X}]={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }\mathrm{\Sigma }{\overline{X}}_i\right){\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}$$

If both simplifications (common regressors and conditional homoskedasticity) hold then we have the considerable simplication

$$\mathrm{var}[\hat{\beta }\mid \mathit{X}]=\mathrm{\Sigma }\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1}.$$

11.5 Asymptotic Distribution

For an asymptotic distribution it is sufficient to consider the equation-by-equation projection model in which case

$$\mathbb{E}\left[X_j{e}_j\right]=0.$$

First, consider consistency. Since ${\hat{\beta }}_j$ are the standard least squares estimators, they are consistent for the projection coefficients ${\beta }_j$.

Second, consider the asymptotic distribution. Our single equation theory implies that the ${\hat{\beta }}_j$ are asymptotically normal. But this theory does not provide a joint distribution of the ${\hat{\beta }}_j$ across $j$, which we now derive. Since the vector

$${\overline{X}}_i^{\prime }{e}_i=\left(\begin{array}{c}{X}_{1i}{e}_{1i}\\ ⋮\\ X_{mi}{e}_{mi}\end{array}\right)$$

is i.i.d. across $i$ and mean zero under (11.9), the central limit theorem implies

$$\frac{1}{\sqrt{n}}\sum _{i=1}^n{\overline{X}}_i^{\prime }{e}_i\underset{d}{⟶}\mathrm{N}(0,\mathrm{\Omega })$$

where

$$\mathrm{\Omega }=\mathbb{E}\left[{\overline{X}}_i^{\prime }{e}_i{e}_i^{\prime }{\overline{X}}_i\right]=\mathbb{E}\left[{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}_i{\overline{X}}_i\right].$$

The matrix $\mathrm{\Omega }$ is the covariance matrix of the variables $X_{ji}{e}_{ji}$ across equations. Under conditional homoskedasticity (11.8) the matrix $\mathrm{\Omega }$ simplifies to

$$\mathrm{\Omega }=\mathbb{E}\left[{\overline{X}}_i^{\prime }\mathrm{\Sigma }{\overline{X}}_i\right]$$

(see Exercise 11.1). When the regressors are common it simplies to

$$\mathrm{\Omega }=\mathbb{E}[e{e}^{\prime }\otimes X{X}^{\prime }]$$

(see Exercise 11.2). Under both conditions (homoskedasticity and common regressors) it simplifies to

$$\mathrm{\Omega }=\mathrm{\Sigma }\otimes \mathbb{E}\left[X{X}^{\prime }\right]$$

(see Exercise 11.3).

Applied to the centered and normalized estimator we obtain the asymptotic distribution. Theorem 11.1 Under Assumption 7.2, $\sqrt{n}(\hat{\beta }-\beta )\underset{d}{⟶}\mathrm{N}(0,{\mathit{V}}_{\beta })$ where ${\mathit{V}}_{\beta }=$ $Q^{-1}\mathrm{\Omega }{Q}^{-1}$ and

$$\mathit{Q}=\mathbb{E}\left[{\overline{X}}^{\prime }\overline{X}\right]=\left(\begin{array}{cccc}\mathbb{E}\left[X_1{X}_1^{\prime }\right]& 0& \cdots & 0\\ ⋮& \ddots & & ⋮\\ 0& 0& \cdots & \mathbb{E}\left[X_m{X}_m^{\prime }\right]\end{array}\right)$$

For a proof, see Exercise 11.4.

When the regressors are common the matrix $\mathit{Q}$ simplifies as

$$\mathit{Q}={\mathit{I}}_m\otimes \mathbb{E}\left[X{X}^{\prime }\right]$$

(See Exercise 11.5).

If both the regressors are common and the errors are conditionally homoskedastic (11.8) then we have the simplification

$${\mathit{V}}_{\beta }=\mathrm{\Sigma }\otimes {\left(\mathbb{E}\left[X{X}^{\prime }\right]\right)}^{-1}$$

(see Exercise 11.6).

Sometimes we are interested in parameters $\theta =r({\beta }_1,\dots ,{\beta }_m)=r(\beta )$ which are functions of the coefficients from multiple equations. In this case the least squares estimator of $\theta$ is $\hat{\theta }=r(\hat{\beta })$. The asymptotic distribution of $\hat{\theta }$ can be obtained from Theorem $11.1$ by the delta method.

Theorem 11.2 Under Assumptions $7.2$ and $7.3,\sqrt{n}(\hat{\theta }-\theta )\underset{d}{⟶}\mathrm{N}(0,{\mathit{V}}_{\theta })$ where ${\mathit{V}}_{\theta }={\mathit{R}}^{\prime }{\mathit{V}}_{\beta }\mathit{R}$ and $\mathit{R}=\frac{\partial }{\partial \beta }r(\beta {)}^{\prime }.$

For a proof, see Exercise 11.7.

Theorem $11.2$ is an example where multivariate regression is fundamentally distinct from univariate regression. Only by treating least squares as a joint estimator can we obtain a distributional theory for a function of multiple equations. We can thereby construct standard errors, confidence intervals, and hypothesis tests.

11.6 Covariance Matrix Estimation

From the finite sample and asymptotic theory we can construct appropriate estimators for the variance of $\hat{\beta }$. In the general case we have

$${\hat{\mathit{V}}}_{\hat{\beta }}={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{e}}_i{\hat{e}}_i^{\prime }{\overline{X}}_i\right){\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}.$$

Under conditional homoskedasticity (11.8) an appropriate estimator is

$${\hat{\mathit{V}}}_{\hat{\beta }}^0={\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}_i\right){\left({\bar{\mathit{X}}}^{\prime }\bar{\mathit{X}}\right)}^{-1}.$$

When the regressors are common then these estimators equal

$${\hat{\mathit{V}}}_{\hat{\beta }}=({\mathit{I}}_m\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1})\left(\sum _{i=1}^n({\hat{e}}_i{\hat{e}}_i^{\prime }\otimes X_i{X}_i^{\prime })\right)({\mathit{I}}_m\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1})$$

and ${\hat{\mathit{V}}}_{\hat{\beta }}^0=\hat{\mathrm{\Sigma }}\otimes {\left({\mathit{X}}^{\prime }\mathit{X}\right)}^{-1}$, respectively.

Covariance matrix estimators for $\hat{\theta }$ are found as

$$\begin{array}{rl}{\hat{\mathit{V}}}_{\hat{\theta }}& ={\hat{\mathit{R}}}^{\prime }{\hat{\mathit{V}}}_{\hat{\beta }}\hat{\mathit{R}}\\ {\hat{\mathit{V}}}_{\hat{\theta }}^0& ={\hat{\mathit{R}}}^{\prime }{\hat{\mathit{V}}}_{\hat{\beta }}^0\hat{\mathit{R}}\\ \hat{\mathit{R}}& =\frac{\partial }{\partial \beta }r(\hat{\beta }{)}^{\prime }.\end{array}$$

Theorem 11.3 Under Assumption 7.2, $n{\hat{\mathit{V}}}_{\hat{\beta }}\underset{p}{\to }{\mathit{V}}_{\beta }$ and $n{\hat{\mathit{V}}}_{\hat{\beta }}^0{\overrightarrow{p}}^0{\mathit{V}}_{\beta }^0$

For a proof, see Exercise 11.8.

11.7 Seemingly Unrelated Regression

Consider the systems regression model under the conditional expectation and homoskedasticity assumptions

$$\begin{array}{rl}Y& =\overline{X}\beta +e\\ \mathbb{E}[e\mid X]& =0\\ \mathbb{E}[e{e}^{\prime }\mid X]& =\mathrm{\Sigma }.\end{array}$$

Since the errors are correlated across equations we consider estimation by Generalized Least Squares (GLS). To derive the estimator, premultiply (11.15) by ${\mathrm{\Sigma }}^{-1/2}$ so that the transformed error vector is i.i.d. with covariance matrix ${\mathit{I}}_m$. Then apply least squares and rearrange to find

$${\hat{\beta }}_{\mathrm{gls}}={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}^{-1}{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}^{-1}{Y}_i\right)$$

(see Exercise 11.9). Another approach is to take the vector representation

$$\mathit{Y}=\bar{\mathit{X}}\beta +\mathit{e}$$

and calculate that the equation error $\mathit{e}$ has variance $\mathbb{E}\left[\mathit{e}{\mathit{e}}^{\prime }\right]={\mathit{I}}_n\otimes \mathrm{\Sigma }$. Premultiply the equation by ${\mathit{I}}_n\otimes$ ${\mathrm{\Sigma }}^{-1/2}$ so that the transformed error has covariance matrix ${\mathit{I}}_{nm}$ and then apply least squares to find

$${\hat{\beta }}_{\mathrm{gls}}={\left({\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\mathrm{\Sigma }}^{-1})\bar{\mathit{X}}\right)}^{-1}\left({\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\mathrm{\Sigma }}^{-1})\mathit{Y}\right)$$

(see Exercise 11.10). Expressions (11.16) and (11.17) are algebraically equivalent. To see the equivalence, observe that

$$\begin{array}{rl}{\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\mathrm{\Sigma }}^{-1})\bar{\mathit{X}}& =\left(\begin{array}{lll}{\overline{X}}_1^{\prime }& \cdots & {\overline{X}}_n^{\prime }\end{array}\right)\left(\begin{array}{cccc}{\mathrm{\Sigma }}^{-1}& 0& \cdots & 0\\ ⋮& {\mathrm{\Sigma }}^{-1}& & ⋮\\ 0& 0& \cdots & {\mathrm{\Sigma }}^{-1}\end{array}\right)\left(\begin{array}{c}{\overline{X}}_1\\ ⋮\\ {\overline{X}}_n\end{array}\right)\\ & =\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}^{-1}{\overline{X}}_i\end{array}$$

and

$$\begin{array}{rl}{\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\mathrm{\Sigma }}^{-1})\mathit{Y}& =\left(\begin{array}{lll}{\overline{X}}_1^{\prime }& \cdots & {\overline{X}}_n^{\prime }\end{array}\right)\left(\begin{array}{cccc}{\mathrm{\Sigma }}^{-1}& 0& \cdots & 0\\ ⋮& {\mathrm{\Sigma }}^{-1}& & ⋮\\ 0& 0& \cdots & 0^{-1}\end{array}\right)\left(\begin{array}{c}{Y}_1\\ ⋮\\ Y_n\end{array}\right)\\ & =\sum _{i=1}^n{\overline{X}}_i^{\prime }{\mathrm{\Sigma }}^{-1}{Y}_i.\end{array}$$

Since $\mathrm{\Sigma }$ is unknown it must be replaced by an estimator. Using $\hat{\mathrm{\Sigma }}$ from (11.5) we obtain a feasible GLS estimator.

$$\begin{array}{rl}{\hat{\beta }}_{\mathrm{sur}}& ={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{Y}_i\right)\\ & ={\left({\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\hat{\mathrm{\Sigma }}}^{-1})\bar{\mathit{X}}\right)}^{-1}\left({\bar{\mathit{X}}}^{\prime }({\mathit{I}}_n\otimes {\hat{\mathrm{\Sigma }}}^{-1})\mathit{Y}\right).\end{array}$$

This is the Seemingly Unrelated Regression (SUR) estimator as introduced by Zellner (1962).

The estimator $\hat{\mathrm{\Sigma }}$ can be updated by calculating the SUR residuals ${\hat{e}}_i=Y_i-{\overline{X}}_i^{\prime }{\hat{\beta }}_{\text{sur }}$ and the covariance matrix estimator $\hat{\mathrm{\Sigma }}=\frac{1}{n}\sum _{i=1}^n{\hat{e}}_i{\hat{e}}_i^{\prime }$. Substituted into (11.18) we obtain an iterated SUR estimator. This can be iterated until convergence.

Under conditional homoskedasticity (11.8) we can derive its asymptotic distribution.

Theorem 11.4 Under Assumption $7.2$ and (11.8)

$$\sqrt{n}({\hat{\beta }}_{\text{sur }}-\beta )\underset{d}{⟶}\mathrm{N}(0,V_{\beta }^{\ast })$$

where ${\mathit{V}}_{\beta }^{\ast }={\left(\mathbb{E}\left[{\overline{X}}^{\prime }{\mathrm{\Sigma }}^{-1}\overline{X}\right]\right)}^{-1}$.

For a proof, see Exercise 11.11.

Under these assumptions, SUR is more efficient than least squares.

Theorem 11.5 Under Assumption $7.2$ and (11.8)

$${\mathit{V}}_{\beta }^{\ast }={\left(\mathbb{E}\left[{\overline{X}}^{\prime }{\mathrm{\Sigma }}^{-1}\overline{X}\right]\right)}^{-1}\le {\left(\mathbb{E}\left[{\overline{X}}^{\prime }\overline{X}\right]\right)}^{-1}\mathbb{E}\left[{\overline{X}}^{\prime }\mathrm{\Sigma }\overline{X}\right]{\left(\mathbb{E}\left[{\overline{X}}^{\prime }\overline{X}\right]\right)}^{-1}={\mathit{V}}_{\beta }$$

and thus ${\hat{\beta }}_{\text{sur }}$ is asymptotically more efficient than ${\hat{\beta }}_{\text{ols. }}$. For a proof, see Exercise 11.12.

An appropriate estimator of the variance of ${\hat{\beta }}_{\text{sur }}$ is

$${\hat{\mathit{V}}}_{\hat{\beta }}={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{\overline{X}}_i\right)}^{-1}$$

Theorem 11.6 Under Assumption $7.2$ and (11.8) $n{\hat{\mathit{V}}}_{\hat{\beta }}\underset{p}{⟶}{\mathit{V}}_{\beta }$.

For a proof, see Exercise 11.13.

In Stata, the seemingly unrelated regressions estimator is implemented using the sureg command.

11.8 Equivalence of SUR and Least Squares

When the regressors are common across equations $X_j=X$ it turns out that the SUR estimator simplifies to least squares.

To see this, recall that when regressors are common this implies that $\overline{X}={\mathit{I}}_m\otimes X^{\prime }$. Then

$$\begin{array}{rl}{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}& =({\mathit{I}}_m\otimes X_i){\hat{\mathrm{\Sigma }}}^{-1}\\ & ={\hat{\mathrm{\Sigma }}}^{-1}\otimes X_i\\ & =({\hat{\mathrm{\Sigma }}}^{-1}\otimes {\mathit{I}}_k)({\mathit{I}}_m\otimes X_i)\\ & =({\hat{\mathrm{\Sigma }}}^{-1}\otimes {\mathit{I}}_k){\overline{X}}_i^{\prime }.\end{array}$$

Thus

$$\begin{array}{rl}{\hat{\beta }}_{\mathrm{sur}}& ={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{Y}_i\right)\\ & ={\left(({\hat{\mathrm{\Sigma }}}^{-1}\otimes {\mathit{I}}_k)\sum _{i=1}^n{\overline{X}}_i^{\prime }{\overline{X}}_i\right)}^{-1}\left(({\hat{\mathrm{\Sigma }}}^{-1}\otimes {\mathit{I}}_k)\sum _{i=1}^n{\overline{X}}_i^{\prime }{Y}_i\right)\\ & ={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{Y}_i\right)={\hat{\beta }}_{\mathrm{ols}}\end{array}$$

A model where regressors are not common across equations is nested within a model with the union of all regressors included in all equations. Thus the model with regressors common across equations is a fully unrestricted model, and a model where the regressors differ across equations is a restricted model. Thus the above result shows that the SUR estimator reduces to least squares in the absence of restrictions, but SUR can differ from least squares otherwise.

Another context where SUR=OLS is when the variance matrix is diagonal, $\mathrm{\Sigma }=\mathrm{diag}\{{\sigma }_1^2,\dots ,{\sigma }_m^2\}$. In this case ${\mathrm{\Sigma }}^{-1/2}{\overline{X}}_i={\overline{X}}_i\mathrm{diag}\{{\mathit{I}}_{k_1}{\sigma }_1^{-1/2},\dots ,{\mathit{I}}_{k_m}{\sigma }_m^{-1/2}\}$ from which you can calculate that ${\hat{\beta }}_{\text{sur }}={\hat{\beta }}_{\text{ols }}$. The intuition is that there is no difference in systems estimation when the equations are uncorrelated, which occurs when $\mathrm{\Sigma }$ is diagonal.

11.9 Maximum Likelihood Estimator

Take the linear model under the assumption that the error is independent of the regressors and multivariate normally distributed. Thus $Y=\overline{X}\beta +e$ with $e\sim \mathrm{N}(0,\mathrm{\Sigma })$. In this case we can consider the maximum likelihood estimator (MLE) of the coefficients.

It is convenient to reparameterize the covariance matrix in terms of its inverse $S={\mathrm{\Sigma }}^{-1}$. With this reparameterization the conditional density of $Y$ given $X=x$ equals

$$f(y\mid x)=\frac{\det (\mathit{S}{)}^{1/2}}{(2\pi {)}^{m/2}}\exp (-\frac{1}{2}(y-x\beta {)}^{\prime }\mathit{S}(y-x\beta )).$$

The log-likelihood function for the sample is

$${\ell }_n(\beta ,\mathit{S})=-\frac{nm}{2}\log (2\pi )+\frac{n}{2}\log (\det (\mathit{S}))-\frac{1}{2}\sum _{i=1}^n{(Y_i-{\overline{X}}_i\beta )}^{\prime }S(Y_i-{\overline{X}}_i\beta ).$$

The maximum likelihood estimator $({\hat{\beta }}_{\text{mle }},{\hat{S}}_{\text{mle }})$ maximizes the log-likelihood function. The first order conditions are

$$0={\frac{\partial }{\partial \beta }{\ell }_n(\beta ,\mathit{S})|}_{\beta =\hat{\beta },\mathit{S}=\hat{\mathit{S}}}=\sum _{i=1}^n{\overline{X}}_i\hat{\mathit{S}}(Y_i-{\overline{X}}_i\hat{\beta })$$

and

$$0={\frac{\partial }{\partial \mathit{S}}{\ell }_n(\beta ,\mathrm{\Sigma })|}_{\beta =\hat{\beta },\mathit{S}=\hat{\mathit{S}}}=\frac{n}{2}{\hat{\mathit{S}}}^{-1}-\frac{1}{2}\mathrm{tr}\left(\sum _{i=1}^n(Y_i-{\overline{X}}_i\hat{\beta }){(Y_i-{\overline{X}}_i\hat{\beta })}^{\prime }\right)$$

The second equation uses the matrix results $\frac{\partial }{\partial S}\log (\det (\mathit{S}))={\mathit{S}}^{-1}$ and $\frac{\partial }{\partial \mathit{B}}\mathrm{tr}(\mathit{A}\mathit{B})={\mathit{A}}^{\prime }$ from Appendix A.20.

Solving and making the substitution $\hat{\mathrm{\Sigma }}={\hat{\mathit{S}}}^{-1}$ we obtain

$$\begin{array}{c}{\hat{\beta }}_{\mathrm{mle}}={\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{\overline{X}}_i\right)}^{-1}\left(\sum _{i=1}^n{\overline{X}}_i^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{Y}_i\right)\\ {\hat{\mathrm{\Sigma }}}_{\mathrm{mle}}=\frac{1}{n}\sum _{i=1}^n(Y_i-{\overline{X}}_i\hat{\beta }){(Y_i-{\overline{X}}_i\hat{\beta })}^{\prime }.\end{array}$$

Notice that each equation refers to the other. Hence these are not closed-form expressions but can be solved via iteration. The solution is identical to the iterated SUR estimator. Thus the iterated SUR estimator is identical to MLE under normality.

Recall that the SUR estimator simplifies to OLS when the regressors are common across equations. The same occurs for the MLE. Thus when ${\overline{X}}_i={\mathit{I}}_m\otimes X_i^{\prime }$ we find that ${\hat{\beta }}_{\mathrm{mle}}={\hat{\beta }}_{\text{ols }}$ and ${\hat{\mathrm{\Sigma }}}_{\text{mle }}={\hat{\mathrm{\Sigma }}}_{\text{ols }}$.

11.10 Restricted Estimation

In many multivariate regression applications it is desired to impose restrictions on the coefficients. In particular, cross-equation restrictions (for example, imposing Slutsky symmetry on a demand system) can be quite important and can only be imposed by a multivariate estimation method. Estimation subject to restrictions can be done by minimum distance, maximum likelihood, or the generalized method of moments.

Minimum distance is a straightforward application of the methods of Chapter 8 to the estimators presented in this chapter, as such methods apply to any asymptotically normal estimator.

Imposing restrictions on maximum likelihood is also straightforward. The likelihood is maximized subject to the imposed restrictions. One important example is explored in detail in the following section.

Generalized method of moments estimation of multivariate regression subject to restrictions will be explored in Section 13.18. This is a particularly simple and straightforward way to estimate restricted multivariate regression models and is our generally preferred approach.

11.11 Reduced Rank Regression

One context where systems estimation is important is when it is desired to impose or test restrictions across equations. Restricted systems are commonly estimated by maximum likelihood under normality. In this section we explore one important special case of restricted multivariate regression known as reduced rank regression. The model was originally proposed by Anderson (1951) and extended by Johansen (1995).

The unrestricted model is

$$\begin{array}{rl}Y& ={\mathit{B}}^{\prime }X+{\mathit{C}}^{\prime }Z+e\\ \mathbb{E}[e{e}^{\prime }\mid X,Z]& =\mathrm{\Sigma }\end{array}$$

where $\mathit{B}$ is $k\times m,\mathit{C}$ is $\ell \times m,Y\in {\mathbb{R}}^m,X\in {\mathbb{R}}^k$, and $Z\in {\mathbb{R}}^{\ell }$. We separate the regressors as $X$ and $Z$ because the coefficient matrix $\mathit{B}$ will be restricted while $\mathit{C}$ will be unrestricted.

The matrix $\mathit{B}$ is full rank if

$$\mathrm{rank}(\mathit{B})=min(k,m).$$

The reduced rank restriction is $\mathrm{rank}(\mathit{B})=r<min(k,m)$ for some known $r$.

The reduced rank restriction implies that we can write the coefficient matrix $\mathit{B}$ in the factored form $\mathit{B}=\mathit{G}{\mathit{A}}^{\prime }$ where $\mathit{A}$ is $m\times r$ and $\mathit{G}$ is $k\times r$. This representation is not unique as we can replace $\mathit{G}$ with $\mathit{G}\mathit{Q}$ and $\mathit{A}$ with $\mathit{A}{\mathit{Q}}^{-1\prime }$ for any invertible $\mathit{Q}$ and the same relation holds. Identification therefore requires a normalization of the coefficients. A conventional normalization is ${\mathit{G}}^{\prime }\mathit{D}\mathit{G}={\mathit{I}}_r$ for given $\mathit{D}$.

Equivalently, the reduced rank restriction can be imposed by requiring that $\mathit{B}$ satisfy the restriction $\mathit{B}{\mathit{A}}_{\perp }=\mathit{G}{\mathit{A}}^{\prime }{\mathit{A}}_{\perp }=0$ for some $m\times (m-r)$ coefficient matrix ${\mathit{A}}_{\perp }$. Since $\mathit{G}$ is full rank this requires that ${\mathit{A}}^{\prime }{\mathit{A}}_{\perp }=0$, hence ${\mathit{A}}_{\perp }$ is the orthogonal complement of $\mathit{A}$. Note that ${\mathit{A}}_{\perp }$ is not unique as it can be replaced by ${\mathit{A}}_{\perp }\mathit{Q}$ for any $(m-r)\times (m-r)$ invertible $\mathit{Q}$. Thus if ${\mathit{A}}_{\perp }$ is to be estimated it requires a normalization.

We discuss methods for estimation of $\mathit{G},\mathit{A},\mathrm{\Sigma },\mathit{C}$, and ${\mathit{A}}_{\perp }$. The standard approach is maximum likelihood under the assumption that $e\sim \mathrm{N}(0,\mathrm{\Sigma })$. The log-likelihood function for the sample is

$$\begin{array}{rl}{\ell }_n(\mathit{G},\mathit{A},\mathit{C},\mathrm{\Sigma })& =-\frac{nm}{2}\log (2\pi )-\frac{n}{2}\log (\det (\mathrm{\Sigma }))\\ & -\frac{1}{2}\sum _{i=1}^n{(Y_i-\mathit{A}{\mathit{G}}^{\prime }{X}_i-{\mathit{C}}^{\prime }{Z}_i)}^{\prime }{\mathrm{\Sigma }}^{-1}(Y_i-\mathit{A}{\mathit{G}}^{\prime }{X}_i-{\mathit{C}}^{\prime }{Z}_i).\end{array}$$

Anderson (1951) derived the MLE by imposing the constraint $\mathit{B}{\mathit{A}}_{\perp }=0$ via the method of Lagrange multipliers. This turns out to be algebraically cumbersome.

Johansen (1995) instead proposed the following straightforward concentration method. Treating $\mathit{G}$ as if it is known, maximize the log-likelihood with respect to the other parameters. Resubstituting these estimators we obtain the concentrated log-likelihood function with respect to $\mathit{G}$. This can be maximized to find the MLE for $\mathit{G}$. The other parameter estimators are then obtain by substitution. We now describe these steps in detail.

Given $\mathit{G}$ the likelihood is a normal multivariate regression in the variables ${\mathit{G}}^{\prime }X$ and $Z$, so the MLE for $\mathit{A},\mathit{C}$ and $\mathrm{\Sigma }$ are least squares. In particular, using the Frisch-Waugh-Lovell residual regression formula we can write the estimators for $\mathit{A}$ and $\mathrm{\Sigma }$ as

$$\hat{\mathit{A}}(\mathit{G})=\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right){\left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}^{-1}$$

and

$$\hat{\mathrm{\Sigma }}(\mathit{G})=\frac{1}{n}({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}-{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}\mathit{G}{\left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}^{-1}{\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}})$$

where $\tilde{\mathit{Y}}=\mathit{Y}-\mathit{Z}{\left({\mathit{Z}}^{\prime }\mathit{Z}\right)}^{-1}{\mathit{Z}}^{\prime }\mathit{Y}$ and $\tilde{\mathit{X}}=\mathit{X}-\mathit{Z}{\left({\mathit{Z}}^{\prime }\mathit{Z}\right)}^{-1}{\mathit{Z}}^{\prime }\mathit{X}$.

Substituting these estimators into the log-likelihood function we obtain the concentrated likelihood function, which is a function of $G$ only.

$$\begin{array}{rl}{\tilde{\ell }}_n(\mathit{G})& ={\ell }_n(\mathit{G},\hat{\mathit{A}}(\mathit{G}),\hat{\mathit{C}}(\mathit{G}),\hat{\mathrm{\Sigma }}(\mathit{G}))\\ & =\frac{m}{2}(n\log (2\pi )-1)-\frac{n}{2}\log \left[\det ({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}-{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}\mathit{G}{\left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}^{-1}{\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}})\right]\\ & =\frac{m}{2}(n\log (2\pi )-1)-\frac{n}{2}\log \left(\det \left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)\right)-\frac{n}{2}\log [\frac{\det \left({\mathit{G}}^{\prime }({\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}-{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\mathit{Y}}^{\prime }\tilde{\mathit{X}})\mathit{G}\right)}{\det \left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}.\end{array}$$

The third equality uses Theorem A.1.8. The MLE $\hat{\mathit{G}}$ for $\mathit{G}$ is the maximizer of ${\tilde{\ell }}_n(\mathit{G})$, or equivalently equals

$$\begin{array}{rl}\hat{\mathit{G}}& =\underset{\mathit{G}}{\mathrm{argmin}}\frac{\det \left({\mathit{G}}^{\prime }({\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}-{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\mathit{Y}}^{\prime }\tilde{\mathit{X}})\mathit{G}\right)}{\det \left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}\\ & =\underset{\mathit{G}}{\mathrm{argmax}}\frac{\det \left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\mathit{Y}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}{\det \left({\mathit{G}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\mathit{G}\right)}\\ & =\{v_1,\dots ,v_r\}\end{array}$$

which are the generalized eigenvectors of ${\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\mathit{Y}}^{\prime }\tilde{\mathit{X}}$ with respect to ${\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}$ corresponding to the $r$ largest generalized eigenvalues. (Generalized eigenvalues and eigenvectors are discussed in Section A.14.) The estimator satisfies the normalization ${\hat{\mathit{G}}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\hat{\mathit{G}}={\mathit{I}}_r$. Letting $v_j^{\ast }$ denote the eigenvectors of (11.20) we can also express $\hat{\mathit{G}}=\{v_m^{\ast },\dots ,v_{m-r+1}^{\ast }\}$.

This is computationally straightforward. In MATLAB, for example, the generalized eigenvalues and eigenvectors of a matrix $\mathit{A}$ with respect to $\mathit{B}$ are found using the command eig $(\mathrm{A},\mathrm{B})$.

Given $\hat{\mathit{G}}$, the MLE $\hat{\mathit{A}},\hat{\mathit{C}},\hat{\mathrm{\Sigma }}$ are found by least squares regression of $Y$ on ${\hat{\mathit{G}}}^{\prime }X$ and $Z$. In particular, $\hat{\mathit{A}}={\hat{\mathit{G}}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}$ because ${\hat{\mathit{G}}}^{\prime }{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\hat{\mathit{G}}={\mathit{I}}_r$ We now discuss the estimator ${\hat{\mathit{A}}}_{\perp }$ of ${\mathit{A}}_{\perp }$. It turns out that

$$\begin{array}{rl}{\hat{\mathit{A}}}_{\perp }& =\underset{\mathit{A}}{\mathrm{argmax}}\frac{\det \left({\mathit{A}}^{\prime }({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}-{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}{\left({\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\right)}^{-1}{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}})\mathit{A}\right)}{\det \left({\mathit{A}}^{\prime }{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\mathit{A}\right)}\\ & =\{w_1,\dots ,w_{m-r}\}\end{array}$$

the eigenvectors of ${\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}-{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}{\left({\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\right)}^{-1}{\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}$ with respect to ${\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}$ associated with the largest $m-r$ eigenvalues.

By the dual eigenvalue relation (Theorem A.5), equations (11.20) and (11.21) have the same non-zero eigenvalues ${\lambda }_j$ and the associated eigenvectors $v_j^{\ast }$ and $w_j$ satisfy the relationship

$$w_j={\lambda }_j^{-1/2}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}{v}_j^{\ast }$$

Letting $\mathrm{\Lambda }=\mathrm{diag}\{{\lambda }_m,\dots ,{\lambda }_{m-r+1}\}$ this implies

$$\{w_m,\dots ,w_{m-r+1}\}={\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}\{v_m^{\ast },\dots ,v_{m-r+1}^{\ast }\}\mathrm{\Lambda }={\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}\hat{\mathit{A}}\mathrm{\Lambda }.$$

The second equality holds because $\hat{\mathit{G}}=\{v_m^{\ast },\dots ,v_{m-r+1}^{\ast }\}$ and $\hat{\mathit{A}}={\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}\hat{\mathit{G}}$. Since the eigenvectors $w_j$ satisfy the orthogonality property $w_j^{\prime }{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}{w}_{\ell }=0$ for $j\ne \ell$, it follows that

$$0={\hat{\mathit{A}}}_{\perp }^{\prime }{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\{w_m,\dots ,w_{m-r+1}\}={\hat{\mathit{A}}}_{\perp }^{\prime }\hat{\mathit{A}}\mathrm{\Lambda }.$$

Since $\mathrm{\Lambda }>0$ we conclude that ${\hat{A}}_{\perp }^{\prime }\hat{A}=0$ as desired.

The solution ${\hat{A}}_{\perp }$ in (11.21) can be represented several ways. One which is computationally convenient is to observe that

$${\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}-{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}{\left({\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}\right)}^{-1}{\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{X}}={\mathit{Y}}^{\prime }{\mathit{M}}_{\mathit{X},\mathit{Z}}\mathit{Y}={\tilde{\mathit{E}}}^{\prime }\tilde{\mathit{E}}$$

where ${\mathit{M}}_{\mathit{X},\mathit{Z}}={\mathit{I}}_n-(\mathit{X},\mathit{Z}){((\mathit{X},\mathit{Z}{)}^{\prime }(\mathit{X},\mathit{Z}))}^{-1}(\mathit{X},\mathit{Z}{)}^{\prime }$ and $\tilde{\mathit{E}}={\mathit{M}}_{\mathit{X},\mathit{Z}}\mathit{Y}$ is the residual matrix from the unrestricted multivariate least squares regression of $Y$ on $X$ and $\mathit{Z}$. The first equality follows by the FrischWaugh-Lovell theorem. This shows that ${\hat{\mathit{A}}}_{\perp }$ are the generalized eigenvectors of ${\tilde{\mathit{E}}}^{\prime }\tilde{\mathit{E}}$ with respect to ${\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}$ corresponding to the $m-r$ largest eigenvalues. In MATLAB, for example, these can be computed using the eig $(\mathrm{A},\mathrm{B})$ command.

Another representation is to write ${\mathit{M}}_Z={\mathit{I}}_n-\mathit{Z}{\left({\mathit{Z}}^{\prime }\mathit{Z}\right)}^{-1}{\mathit{Z}}^{\prime }$ so that

$${\hat{A}}_{\perp }=\underset{A}{\mathrm{argmax}}\frac{\det \left({\mathit{A}}^{\prime }{\mathit{Y}}^{\prime }{\mathit{M}}_{\mathit{X},\mathit{Z}}\mathit{Y}\mathit{A}\right)}{\det \left({\mathit{A}}^{\prime }{\mathit{Y}}^{\prime }{\mathit{M}}_{\mathit{Z}}\mathit{Y}\mathit{A}\right)}=\underset{\mathit{A}}{\mathrm{argmin}}\frac{\det \left({\mathit{A}}^{\prime }{\mathit{Y}}^{\prime }{\mathit{M}}_{\mathit{Z}}\mathit{Y}\mathit{A}\right)}{\det \left({\mathit{A}}^{\prime }{\mathit{Y}}^{\prime }{\mathit{M}}_{\mathit{X},\mathit{Z}}\mathit{Y}\mathit{A}\right)}.$$

We summarize our findings. Theorem 11.7 The MLE for the reduced rank model (11.19) under $e\sim \mathrm{N}(0,\mathrm{\Sigma })$ is given as follows. Let $\tilde{\mathit{Y}}$ and $\tilde{\mathit{X}}$ be the residual matrices from multivariate regression of $\mathit{Y}$ and $\mathit{X}$ on $\mathit{Z}$, respectively. Then ${\hat{\mathit{G}}}_{\mathrm{mle}}=\{v_1,\dots ,v_r\}$, the generalized eigenvectors of ${\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{Y}}{\left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)}^{-1}{\mathit{Y}}^{\prime }\tilde{\mathit{X}}$ with respect to ${\tilde{\mathit{X}}}^{\prime }\tilde{\mathit{X}}$ corresponding to the $r$ largest eigenvalues ${\hat{\lambda }}_j.{\hat{\mathit{A}}}_{\text{mle }},{\hat{\mathit{C}}}_{\text{mle }}$ and ${\hat{\mathrm{\Sigma }}}_{\text{mle }}$ are obtained by the least squares regression

$$\begin{array}{rl}{Y}_i& ={\hat{\mathit{A}}}_{\mathrm{mle}}{\hat{\mathit{G}}}_{\mathrm{mle}}^{\prime }{X}_i+{\hat{\mathit{C}}}_{\mathrm{mle}}^{\prime }{Z}_i+{\hat{e}}_i\\ {\hat{\mathrm{\Sigma }}}_{\mathrm{mle}}& =\frac{1}{n}\sum _{i=1}^n{\hat{e}}_i{\hat{e}}_i^{\prime }\end{array}$$

Let $\tilde{\mathit{E}}$ be the residual matrix from a multivariate regression of $\mathit{Y}$ on $\mathit{X}$ and $\mathit{Z}$. Then ${\hat{\mathit{A}}}_{\perp }$ equals the generalized eigenvectors of $\tilde{\mathit{E}}{\tilde{\mathit{E}}}^{\prime }$ with respect to ${\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}$ corresponding to the $m-r$ smallest eigenvalues. The maximized likelihood equals

$${\ell }_n=\frac{m}{2}(n\log (2\pi )-1)-\frac{n}{2}\log \left(\det \left({\tilde{\mathit{Y}}}^{\prime }\tilde{\mathit{Y}}\right)\right)-\frac{n}{2}\sum _{j=1}^r\log (1-{\hat{\lambda }}_j).$$

An R package for reduced rank regression is “RRR”. I am unaware of a Stata command.

11.12 Principal Component Analysis

In Section $4.21$ we described the Duflo, Dupas, and Kremer (2011) dataset which is a sample of Kenyan first grade test scores. Following the authors we focused on the variable totalscore which is each student’s composite test score. If you examine the data file you will find other pieces of information about the students’ performance, including each student’s score on separate sections of the test, with the labels wordscore (word recognition), sentscore (sentence recognition), letterscore (letter recognition), spellscore (spelling), additions_score (addition), substractions_score (subtraction), multiplications_score (multiplication). The “total” score sums the scores from the individual sections. Perhaps there is more information in the section scores. How can we learn about this from the data?

Principal component analysis (PCA) addresses this issue by ordering linear combinations by their contribution to variance. Definition $11.1$ Let $X$ be a $k\times 1$ random vector.

The first principal component is $U_1=h_1^{\prime }X$ where $h_1$ satisfies

$$h_1=\underset{h^{\prime }h=1}{\mathrm{argmax}}\mathrm{var}\left[h^{\prime }X\right].$$

The second principal component is $U_2=h_2^{\prime }X$ where

$$h_2=\underset{h^{\prime }h=1,h^{\prime }{h}_1=0}{\mathrm{argmax}}\mathrm{var}\left[h^{\prime }X\right].$$

In general, the $j^{th}$ principal component is $U_j=h_j^{\prime }X$ where

$$h_j=\underset{h^{\prime }h=1,h^{\prime }{h}_1=0,\dots ,h^{\prime }{h}_{j-1}=0}{\mathrm{argmax}}\mathrm{var}\left[h^{\prime }X\right].$$

The principal components of $X$ are linear combinations $h^{\prime }X$ ranked by contribution to variance. By the properties of quadratic forms (Section A.15) the weight vectors $h_j$ are the eigenvectors of $\mathrm{\Sigma }=\mathrm{var}[X]$.

Theorem 11.8 The principal components of $X$ are $U_j=h_j^{\prime }X$, where $h_j$ is the eigenvector of $\mathrm{\Sigma }$ associated with the $j^{\text{th }}$ ordered eigenvalue ${\lambda }_j$ of $\mathrm{\Sigma }$.

Another way to see the PCA construction is as follows. Since $\mathrm{\Sigma }$ is symmetric the spectral decomposition (Theorem A.3) states that $\mathrm{\Sigma }=\mathit{H}\mathit{D}{\mathit{H}}^{\prime }$ where $\mathit{H}=[h_1,\dots ,h_k]$ and $\mathit{D}=\mathrm{diag}(d_1,\dots ,d_k)$ are the eigenvectors and eigenvalues of $\mathrm{\Sigma }$. Since $\mathrm{\Sigma }$ is positive semi-definite the eigenvalues are real, non-negative, and ordered $d_1\ge d_2\ge \cdots \ge d_k$. Let $U=(U_1,\dots ,U_k)$ be the principal components of $X$. By Theorem 11.8, $U={\mathit{H}}^{\prime }X$. The covariance matrix of $U$ is

$$\mathrm{var}[U]=\mathrm{var}\left[{\mathit{H}}^{\prime }X\right]={\mathit{H}}^{\prime }\mathrm{\Sigma }\mathit{H}=\mathit{D}$$

which is diagonal. This shows that $\mathrm{var}\left[U_j\right]=d_j$ and the principal components are mutually uncorrelated. The relative variance contribution of the $j^{th}$ principal component is $d_j/\mathrm{tr}(\mathrm{\Sigma })$.

Principal components are sensitive to the scaling of $X$. Consequently, it is recommended to first scale each element of $X$ to have mean zero and unit variance. In this case $\mathrm{\Sigma }$ is a correlation matrix.

The sample principal components are obtained by replacing the unknowns by sample estimators. Let $\hat{\mathrm{\Sigma }}$ be the sample covariance or correlation matrix and ${\hat{h}}_1,{\hat{h}}_2,\dots ,{\hat{h}}_k$ its ordered eigenvectors. The sample principal components are ${\hat{h}}_j^{\prime }{X}_i$.

To illustrate we use the Duflo, Dupas, and Kremer (2011) dataset. In Table $11.1$ we display the seven eigenvalues of the sample correlation matrix for the seven test scores described above. The seven eigenvalues sum to seven because we have applied PCA to the correlation matrix. The first eigenvalue is $4.0$, implying that the first principal component explains $57$ of the variance of the seven test scores. The second eigenvalue is $1.0$, implying that the second principal component explains $15$ of the variance. Together the first two components explain $72$ of the variance of the seven test scores.

In Table $11.2$ we display the weight vectors (eigenvectors) for the first two principal components. The weights for the first component are all positive and similar in magnitude. This means that the first Table 11.1: Eigenvalue Decomposition of Sample Correlation Matrix

|Eigenvalue|Proportion|

|:|———-|———-| |1| $4.02$ | $0.57$ | |2| $1.04$ | $0.15$ | |3| $0.57$ | $0.08$ | |4| $0.52$ | $0.08$ | |5| $0.37$ | $0.05$ | |6| $0.29$ | $0.04$ | |7| $0.19$ | $0.03$ |

Table 11.2: Principal Component Weight Vectors

	First	Second
words	$0.41$	$-0.32$
sentences	$0.32$	$-0.49$
letters	$0.40$	$-0.13$
spelling	$0.43$	$-0.28$
addition	$0.38$	$0.41$
subtraction	$0.35$	$0.52$
multiplication	$0.33$	$0.36$

principal component is similar to a simple average of the seven test scores. This is quite fascinating. This is consistent with our intuition that a simple average (e.g. the variable totalscore) captures most of the information contained in the seven test scores. The weights for the second component have a different pattern. The four literacy scores receive negative weight and the three math scores receive positive weight with similar magnitudes. This means that the second principal component is similar to the difference between a student’s math and verbal test scores. Taken together, the information in the first two principal components is equivalent to “average verbal” and “average math” test scores. What this shows is that $57$ of the variation in the seven section test scores can be explained by a simple average (e.g. totalscore), and $72$ can be explained by averages for the verbal and math halves of the test.

In Stata, principal components analysis can be implemented with the pca command. In $\mathrm{R}$ use prcomp or princomp. All three can be applied to either covariance matrices (unscaled data) or correlation matrices (normalized data) but they have different default settings. The Stata pca command by default normalizes the observations. The R commands by default do not normalize the observations.

11.13 Factor Models

Closely related to principal components are factor models. These are statistical models which decompose random vectors into common factors and idiosyncratic errors. Factor models are popular throughout the social sciences. Consequently a variety of estimation methods have been developed. In the next few sections we focus on methods which are popular among economists.

Let $X={(X_1,\dots ,X_k)}^{\prime }$ be a $k\times 1$ random vector (for example the seven test scores described in the previous section). Assume that the elements of $X$ are scaled to have mean zero and unit variance.

A single factor model for $X$ is

$$X=\lambda F+u$$

where $\lambda \in {\mathbb{R}}^k$ are factor loadings, $F\in \mathbb{R}$ is a common factor, and $u\in {\mathbb{R}}^k$ is a random error. The factor $F$ is individual-specific while the coefficient $\lambda$ is common across individuals. The model (11.22) specifies that correlation between the elements of $X$ is due to the common factor $F$. In the student test score example it is intuitive to think of $F$ as a student’s scholastic “aptitude”; in this case the vector $\lambda$ describes how scholastic aptitude affects the seven subject scores.

A multiple factor model has $r<k$ factors. We write the model as

$$X=\mathrm{\Lambda }F+u$$

where $\mathrm{\Lambda }$ is a $k\times r$ matrix of factor loadings and $F={(F_1,\dots ,F_r)}^{\prime }$ is an $r\times 1$ vector of factors. In the student test score example possible factors could be “math aptitude”, “language skills”, “social skills”, “artistic ability”, “creativity”, etc. The factor loading matrix $\mathrm{\Lambda }$ indicates the effect of each factor on each test score. The number of factors $r$ is taken as known. We discuss selection of $r$ later.

The error vector $u$ is assumed to be mean zero, uncorrelated with $F$, and (under correct specification) to have mutually uncorrelated elements. We write its covariance matrix as $\mathrm{\Psi }=\mathbb{E}\left[u{u}^{\prime }\right]$. The factor vector $F$ can either be treated as random or as a regressor. In this section we treat $F$ as random; in the next we treat $F$ as regressors. The random factors $F$ are assumed mean zero and are normalized so that $\mathbb{E}\left[F{F}^{\prime }\right]=$ ${\mathit{I}}_r$

The assumptions imply that the correlation matrix $\mathrm{\Sigma }=\mathbb{E}\left[X{X}^{\prime }\right]$ equals

$$\mathrm{\Sigma }=\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }.$$

The factor analysis literature often describes $\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }$ as the communality and the idiosyncratic error matrix $\mathrm{\Psi }$ as the uniqueness. The former is the portion of the variance which is explained by the factor model and the latter is the unexplained portion of the variance.

The model is often ${}^1$ estimated by maximum likelihood. Under joint normality of $(F,u)$ the distribution of $X$ is $\mathrm{N}(0,\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi })$. The parameters are $\mathrm{\Lambda }$ and $\mathrm{\Psi }=\mathrm{diag}({\psi }_1,\dots ,{\psi }_k)$. The log-likelihood function of a random sample $(X_1,\dots ,X_n)$ is

$${\ell }_n(\mathrm{\Lambda },\mathrm{\Psi })=-\frac{nk}{2}\log (2\pi )-\frac{n}{2}\log \det (\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi })-\frac{n}{2}\mathrm{tr}\left({(\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi })}^{-1}\hat{\mathrm{\Sigma }}\right).$$

The $\mathrm{MLE}(\hat{\mathrm{\Lambda }},\hat{\mathrm{\Psi }})$ maximizes ${\ell }_n(\mathrm{\Lambda },\mathrm{\Psi })$. There is not an algebraic solution so the estimator is found using numerical methods. Fortunately, computational algorithms are available in standard packages. A detailed description and analysis can be found in Anderson (2003, Chapter 14).

The form of the log-likelihood is intriguing. Notice that the log-likelihood is only a function of the observations through its correlation matrix $\hat{\mathrm{\Sigma }}$, and only a function of the parameters through the population correlation matrix $\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$. The final term in (11.25) is a measure of the match between $\hat{\mathrm{\Sigma }}$ and $\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$. Together, we see that the Gaussian log-likelihood is essentially a measure of the fit of the model and sample correlation matrices. It is therefore not reliant on the normality assumption.

It is often of interest to estimate the factors $F_i$. Given $\mathrm{\Lambda }$ the equation $X_i=\mathrm{\Lambda }{F}_i+u_i$ can be viewed as a regression with coefficient $F_i$. Its least squares estimator is ${\hat{F}}_i={\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i$. The GLS estimator (taking into account the covariance matrix of $u_i)$ is ${\hat{F}}_i={\left({\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}{X}_i$. This motivates the Bartlett scoring estimator

$${\tilde{F}}_i={\left({\hat{\mathrm{\Lambda }}}^{\prime }{\hat{\mathrm{\Psi }}}^{-1}\hat{\mathrm{\Lambda }}\right)}^{-1}{\hat{\mathrm{\Lambda }}}^{\prime }{\hat{\mathrm{\Psi }}}^{-1}{X}_i.$$

The idealized version satisfies

$${\hat{F}}_i={\left({\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}(\mathrm{\Lambda }{F}_i+u_i)=F_i+{\left({\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}{u}_i$$

${}^1$ There are other estimators used in applied factor analysis. However there is little reason to consider estimators beyond the MLE of this section and the principal components estimator of the next section. which is unbiased for $F_i$ and has variance ${\left({\mathrm{\Lambda }}^{\prime }{\mathrm{\Psi }}^{-1}\mathrm{\Lambda }\right)}^{-1}$. Thus the Barlett scoring estimator is typically described as “unbiased” though this is actually a property of its idealized version ${\hat{F}}_i$.

A second estimator for the factors can be constructed from the multivariate linear projection of $F$ on $X$. This is $F=AX+\xi$ where the coefficient matrix $\mathit{A}$ is $r\times k$. The coefficient matrix equals

$$\mathit{A}=\mathbb{E}\left[F{X}^{\prime }\right]\mathbb{E}{\left[X{X}^{\prime }\right]}^{-1}={\mathrm{\Lambda }}^{\prime }{\mathrm{\Sigma }}^{-1},$$

the second equation using $\mathbb{E}\left[F{X}^{\prime }\right]=\mathbb{E}[F(\mathrm{\Lambda }F+u{)}^{\prime }]=\mathbb{E}\left[F{F}^{\prime }\right]{\mathrm{\Lambda }}^{\prime }+\mathbb{E}\left[F{u}^{\prime }\right]={\mathrm{\Lambda }}^{\prime }$. The predicted value of $F_i$ is $F_i^{\ast }=\mathit{A}{X}_i={\mathrm{\Lambda }}^{\prime }{\mathrm{\Sigma }}^{-1}{X}_i$. This motivates the regression scoring estimator

$${\overline{F}}_i={\hat{\mathrm{\Lambda }}}^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}{X}_i.$$

The idealized version $F_i^{\ast }$ has conditional expectation ${\mathrm{\Lambda }}^{\prime }{\mathrm{\Sigma }}^{-1}\mathrm{\Lambda }{F}_i$ and is thus biased for $F_i$. Hence the regression scoring estimator ${\overline{F}}_i$ is often described as “biased”. Some algebraic manipulations reveal that $F_i^{\ast }$ has MSE ${\mathit{I}}_r-{\mathrm{\Lambda }}^{\prime }{({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }+\mathrm{\Psi })}^{-1}\mathrm{\Lambda }$ which is smaller (in a positive definite sense) than the MSE of the idealized Bartlett estimator ${\hat{F}}_i$.

Which estimator is preferred, Bartlett or regression scoring? The differences diminish when $k$ is large so the choice is most relevant for small to moderate $k$. The regression scoring estimator has lower approximate MSE, meaning that it is a more precise estimator. Thus based on estimation precision this is our recommended choice.

The factor loadings $\mathrm{\Lambda }$ and factors $F$ are not separately identified. To see this, notice that if you replace $(\mathrm{\Lambda },F)$ with ${\mathrm{\Lambda }}^{\ast }=\mathrm{\Lambda }\mathit{G}$ and $F^{\ast }={\mathit{G}}^{\prime }F$ where $\mathit{G}$ is $r\times r$ and orthonormal then the regression model is identical. Such replacements are called “rotations” in the factor analysis literature. Any orthogonal rotation of the factor loadings is an equally valid representation. The default MLE outputs are one specific rotation; others can be obtained by a variety of algorithms (which we do not review here). Consequently it is unwise to attribute meaning to the individual factor loading estimates.

Another important and tricky issue is selection of the number of factors $r$. There is no clear guideline. One approach is to examine the principal component decomposition, look for a division between the “large” and “small eigenvalues, and set $r$ to equal to the number of”large” eigenvalues. Another approach is based on testing. As a by-product of the MLE (and standard package implementations) we obtain the LR test for the null hypothesis of $r$ factors against the alternative hypothesis of $k$ factors. If the LR test rejects (has a small p-value) this is evidence that the given $r$ may be too small.

In Stata, the $\mathrm{MLE}(\hat{\mathrm{\Lambda }},\hat{\mathrm{\Psi }})$ can be calculated with the factor, ml factors (r) command. The factor estimates ${\tilde{F}}_i$ and ${\overline{F}}_i$ can be calculated by the predict command with either the barlett or regression option, respectively. In $R$, the command factanal ( $X$, factors=r, rotation=“none”) calculates the $\mathrm{MLE}(\hat{\mathrm{\Lambda }},\hat{\mathrm{\Psi }})$ and also calculates the factor estimates ${\tilde{F}}_i$ and/or ${\overline{F}}_i$ using the scores option.

11.14 Approximate Factor Models

The MLE of the previous section is a good choice for factor estimation when the number of variables $k$ is small and the factor model is believed to be correctly specified. In many economic applications of factor analysis, however, the number of variables is $k$ is large. In such contexts the MLE can be computationally costly and/or unstable. Furthermore it is typically not credible to believe that the model is correctly specified; rather it is more reasonable to view the factor model as a useful approximation. In this section we explore an approach known as the approximate factor model with estimation by principal components. The estimation method is justified by an asymptotic framework where the number of variables $k\to \mathrm{\infty }$ The approximate factor model was introduced by Chamberlain and Rothschild (1983). It is the same as (11.23) but relaxes the assumption on the idiosyncratic error $u$ so that the covariance matrix $\mathrm{\Psi }=$ $\mathbb{E}\left[u{u}^{\prime }\right]$ is left unrestricted. In this context the Gaussian MLE of the previous section is misspecified.

Chamberlain and Rothschild (and the literature which followed) proposed estimation by least squares. The idea is to treat the factors as unknown regressors and simultaneously estimate the factors $F_i$ and factor loadings $\mathrm{\Lambda }$. We first describe the estimation method.

Let $(X_1,\dots ,X_n)$ be a sample centered at sample means. The least squares criterion is

$$\frac{1}{n}\sum _{i=1}^n{(X_i-\mathrm{\Lambda }{F}_i)}^{\prime }(X_i-\mathrm{\Lambda }{F}_i).$$

Let $(\hat{\mathrm{\Lambda }},{\hat{F}}_1,\dots ,{\hat{F}}_n)$ be the joint minimizers. As $\mathrm{\Lambda }$ and $F_i$ are not separately identified a normalization is needed. For compatibility with the notation of the previous section we use $n^{-1}\sum _{i=1}^n{\hat{F}}_i{\hat{F}}_i^{\prime }={\mathit{I}}_r$.

We use a concentration argument to find the solution. As described in the previous section, each observation satisfies the multivariate equation $X_i=\mathrm{\Lambda }{F}_i+u_i$. For fixed $\mathrm{\Lambda }$ this is a set of $k$ equations with $r$ unknowns $F_i$. The least squares solution is ${\hat{F}}_i(\mathrm{\Lambda })={\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i$. Substituting this expression into the least squares criterion the concentrated least squares criterion for $\mathrm{\Lambda }$ is

$$\begin{array}{rl}\frac{1}{n}\sum _{i=1}^n{(X_i-\mathrm{\Lambda }{\hat{F}}_i(\mathrm{\Lambda }))}^{\prime }(X_i-\mathrm{\Lambda }{\hat{F}}_i(\mathrm{\Lambda }))& =\frac{1}{n}\sum _{i=1}^n{(X_i-\mathrm{\Lambda }{\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i)}^{\prime }(X_i-\mathrm{\Lambda }{\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i)\\ & =\frac{1}{n}\sum _{i=1}^n(X_i^{\prime }{X}_i-X_i^{\prime }\mathrm{\Lambda }{\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i)\\ & =\mathrm{tr}[\hat{\mathrm{\Sigma }}]-\mathrm{tr}\left[{\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }\hat{\mathrm{\Sigma }}\mathrm{\Lambda }\right]\end{array}$$

where $\hat{\mathrm{\Sigma }}=n^{-1}\sum _{i=1}^n{X}_i{X}_i^{\prime }$ is the sample covariance matrix. The least squares estimator $\hat{\mathrm{\Lambda }}$ minimizes this criterion. Let $\hat{\mathit{D}}$ and $\hat{\mathit{H}}$ be first $r$ eigenvalues and eigenvectors of $\hat{\mathrm{\Sigma }}$. Using the normalization ${\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }={\mathit{I}}_r$, from the extrema results of Section A.15 the minimizer of the least squares criterion is $\hat{\mathrm{\Lambda }}=\hat{\mathit{H}}$. More broadly any rotation of $\hat{\mathit{H}}$ is valid. Consider $\hat{\mathrm{\Lambda }}=\hat{\mathit{H}}{\hat{\mathit{D}}}^{1/2}$. Recall the expression for the factors ${\hat{F}}_i(\mathrm{\Lambda })=$ ${\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i$. We find that the estimated factors are

$${\hat{F}}_i={\left({\hat{\mathit{D}}}^{1/2}{\hat{\mathit{H}}}^{\prime }\hat{\mathit{H}}{\hat{\mathit{D}}}^{1/2}\right)}^{-1}{\hat{\mathit{D}}}^{1/2}{\hat{\mathit{H}}}^{\prime }{X}_i={\hat{\mathit{D}}}^{-1/2}{\hat{\mathit{H}}}^{\prime }{X}_i$$

We calculate that

$$n^{-1}\sum _{i=1}^n{\hat{F}}_i{\hat{F}}_i^{\prime }={\hat{\mathit{D}}}^{-1/2}{\hat{\mathit{H}}}^{\prime }\hat{\mathrm{\Sigma }}\hat{\mathit{H}}{\hat{\mathit{D}}}^{-1/2\prime }={\hat{\mathit{D}}}^{-1/2}\hat{\mathit{D}}{\hat{\mathit{D}}}^{-1/2\prime }={\mathit{I}}_r$$

which is the desired normalization. This shows that the rotation $\hat{\mathrm{\Lambda }}=\hat{\mathit{H}}{\hat{\mathit{D}}}^{1/2}$ produces factor estimates satisfying this normalization.

We have proven the following result.

Theorem 11.9 The least squares estimator of the factor model (11.23) under the normalization $n^{-1}\sum _{i=1}^n{\hat{F}}_i{\hat{F}}_i^{\prime }={\mathit{I}}_r$ has the following solution:

1. Let $\hat{\mathit{D}}=\mathrm{diag}[{\hat{d}}_1,\dots ,{\hat{d}}_r]$ and $\hat{\mathit{H}}=[{\hat{h}}_1,\dots ,{\hat{h}}_r]$ be the first $r$ eigenvalues and eigenvectors of the sample covariance matrix $\hat{\mathrm{\Sigma }}$.

2. $\hat{\mathrm{\Lambda }}=\hat{\mathit{H}}{\hat{\mathit{D}}}^{1/2}$.

3. ${\hat{F}}_i={\hat{\mathit{D}}}^{-1/2}{\hat{\mathit{H}}}^{\prime }{X}_i$. Theorem $11.9$ shows that the least squares estimator is based on an eigenvalue decomposition of the covariance matrix. This is computationally stable even in high dimensions.

The factor estimates are the principal components scaled by the eigenvalues of $\hat{\mathrm{\Sigma }}$. Specifically, the $j^{th}$ factor estimate is ${\hat{F}}_{ji}={\hat{d}}_j^{-1/2}{\hat{h}}_j^{\prime }X$. Consequently many authors call this estimator the “principalcomponent method”.

Unfortunately, $\hat{\mathrm{\Lambda }}$ is inconsistent for $\mathrm{\Lambda }$ if $k$ is fixed, as we now show. By the WLLN and CMT, $\hat{\mathrm{\Sigma }}\underset{p}{⟶}$ and $\hat{\mathit{H}}\underset{p}{⟶}\mathit{H}$, the first $r$ eigenvectors of $\mathrm{\Sigma }$. When $\mathrm{\Psi }$ is diagonal, the eigenvectors of $\mathrm{\Sigma }=\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$ do not lie in the range space of $\mathrm{\Lambda }$ except in the special case $\mathrm{\Psi }={\sigma }^2{\mathit{I}}_k$. Consequently the estimator $\hat{\mathrm{\Lambda }}$ is inconsistent.

This inconsistency should not be viewed as surprising. The sample has a total of $nk$ observations and the model has a total of $nr+kr-r(r+1)/2$ parameters. Since the number of estimated pararameters is proportional to sample size we should not expect estimator consistency.

As first recognized by Chamberlain and Rothschild, this deficiency diminishes as $k$ increases. Specifically, assume that $k\to \mathrm{\infty }$ as $n\to \mathrm{\infty }$. One implication is that the number of observations $nk$ increase at a rate faster than $n$, while the number of parameters increase at a rate proportional to $n$. Another implication is that as $k$ increases there is increasing information about the factors.

To make this precise we add the following assumption. Let ${\lambda }_{min}(\mathit{A})$ and ${\lambda }_{max}(\mathit{A})$ denote the smallest and largest eigenvalues of a positive semi-definite matrix $\mathit{A}$.

Assumption $11.1$ As $k\to \mathrm{\infty }$

1. ${\lambda }_{max}(\mathrm{\Psi })\le B<\mathrm{\infty }$.

2. ${\lambda }_{min}\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)\to \mathrm{\infty }$ as $k\to \mathrm{\infty }$.

Assumption 11.1.1 bounds the covariance matrix of the idiosyncratic errors. When $\mathrm{\Psi }=\mathrm{diag}({\sigma }_1^2,\dots ,{\sigma }_k^2)$ this is the same as bounding the individual variances. Effectively Assumption 11.1.1 means that while the elements of $u$ can be correlated they cannot have a correlation structure similar to that of a factor model. Assumption 11.1.2 requires the factor loading matrix to increase in magnitude as the number of variables increases. This is a fairly mild requirement. When the factor loadings are of similar magnitude across variables, ${\lambda }_{min}\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)\sim k\to \mathrm{\infty }$. Conceptually, Assumption 11.1.2 requires additional variables to add information about the unobserved factors.

Assumption $11.1$ implies that in the covariance matrix factorization $\mathrm{\Sigma }=\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$ the component $\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }$ dominates as $k$ increases. This means that for large $k$ the first $r$ eigenvectors of $\mathrm{\Sigma }$ are equivalent to those of $\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }$, which are in the range space of $\mathrm{\Lambda }$. This observation led Chamberlain and Rothschild (1983) to deduce that the principal components estimator is an asymptotic (large $k$ ) analog estimator for the factor loadings and factors. Bai (2003) demonstrated that the estimator is consistent as $n,k\to \mathrm{\infty }$ jointly. The conditions and proofs are technical so are not reviewed here.

Now consider the estimated factors

$${\hat{F}}_i={\mathit{D}}^{-1/2}{\mathit{H}}^{\prime }{X}_i={\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }{X}_i$$

where for simplicity we ignore estimation error. Since $X_i=\mathrm{\Lambda }{F}_i+u_i$ and ${\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }=\mathit{D}$ we can write this as

$${\hat{F}}_i=F_i+{\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }{u}_i.$$

This shows that ${\hat{F}}_i$ is an unbiased estimator for $F_i$ and has variance $\mathrm{var}\left[{\hat{F}}_i\right]={\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathrm{\Psi }\mathrm{\Lambda }{\mathit{D}}^{-1}$. Under Assumption 11.1, $\Vert \mathrm{var}\left[{\hat{F}}_i\right]\Vert \le B/{\lambda }_{min}\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)\to 0$. Thus ${\hat{F}}_i$ is consistent for $F_i$ as $k\to \mathrm{\infty }$. Bai (2003) shows that this extends to the feasible estimator as $n,k\to \mathrm{\infty }$.

In Stata, the least squares estimator $\hat{\mathrm{\Lambda }}$ and factors ${\hat{F}}_i$ can be calculated with the factor, pcf factors (r) command followed by predict. In $\mathrm{R}$ a feasible estimation approach is to calculate the factors by eigenvalue decomposition.

11.15 Factor Models with Additional Regressors

Consider the model

$$X=\mathrm{\Lambda }F+BZ+e$$

where $X$ and $e$ are $k\times 1,\mathrm{\Lambda }$ is $k\times r,F$ is $r\times 1,B$ is $k\times \ell$, and $Z$ is $\ell \times 1$.

The coefficients $\mathrm{\Lambda }$ and $\mathit{B}$ can be estimated by a combination of factor regression (either MLE or principal components) and least squares. The key is the following two observations:

1. Given $\mathit{B}$, the coefficient $\mathrm{\Lambda }$ can be estimated by factor regression applied to $X-\mathit{B}Z$.

2. Given the factors $F$, the coefficients $\mathrm{\Lambda }$ and $\mathit{B}$ can be estimated by multivariate least squares of $X$ on $F$ and $Z$.

Estimation iterates between these two steps. Start with a preliminary estimator of $\mathit{B}$ obtained by multivariate least squares of $X$ on $Z$. Then apply the above two steps and iterate under convergence.

11.16 Factor-Augmented Regression

In the previous sections we considered factor models which decompose a set of variables into common factors and idiosyncratic errors. In this section we consider factor-augmented regression, which uses such common factors as regressors for dimension reduction.

Suppose we have the variables $(Y,Z,X)$ where $Y\in \mathbb{R},Z\in {\mathbb{R}}^{\ell }$, and $X\in {\mathbb{R}}^k$. In practice, $k$ may be large and the elements of $X$ may be highly correlated. The factor-augmented regression model is

$$\begin{array}{rl}Y& =F^{\prime }\beta +Z^{\prime }\gamma +e\\ X& =\mathrm{\Lambda }F+u\\ \mathbb{E}[Fe]& =0\\ \mathbb{E}[Ze]& =0\\ \mathbb{E}\left[F{u}^{\prime }\right]& =0\\ \mathbb{E}[ue]& =0,\end{array}$$

The random variables are $e\in \mathbb{R},F\in {\mathbb{R}}^r$, and $u\in {\mathbb{R}}^k$. The regression coefficients are $\beta \in {\mathbb{R}}^k$ and $\gamma \in {\mathbb{R}}^{\ell }$. The matrix $\mathrm{\Lambda }$ are the factor loadings.

This model specifies that the influence of $X$ on $Y$ is through the common factors $F$. The idea is that the variation in the regressors is mostly captured by the variation in the factors, so the influence of the regressors can be captured through these factors. This can be viewed as a dimension-reduction technique as we have reduced the $k$-dimensional $X$ to the $r$-dimensional $F$. Interest typically focuses on the regressors $Z$ and its coefficients $\gamma$. The factors $F$ are included in the regression as “controls” and its coefficient $\beta$ is less typically of interest. Since it is difficult to interpret the factors $F$ only their range space is identified it is generally prudent to avoid intrepreting the coefficients $\beta$. The model is typically estimated in multiple steps. First, the factor loadings $\mathrm{\Lambda }$ and factors $F_i$ are estimated by factor regression. In the case of principal-components estimation the factor estimates are the scaled ${}^2$ principal components ${\hat{F}}_i={\hat{\mathit{D}}}^{-1}{\hat{\mathrm{\Lambda }}}^{\prime }{X}_i$. Second, $Y$ is regressed on the estimated factors and the other regressors to obtain the estimator of $\beta$ and $\gamma$. This second-step estimator equals (for simplicity assume there is no $Z$ )

$$\begin{array}{rl}\hat{\beta }& ={\left(\sum _{i=1}^n{\hat{F}}_i{\hat{F}}_i^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{\hat{F}}_i{Y}_i\right)\\ & ={\left({\hat{\mathit{D}}}^{-1}{\hat{\mathrm{\Lambda }}}^{\prime }\frac{1}{n}\sum _{i=1}^n{X}_i{X}_i^{\prime }{\hat{\mathrm{\Lambda }}}^{-1}\right)}^{-1}\left({\hat{\mathit{D}}}^{-1}{\hat{\mathrm{\Lambda }}}^{\prime }\frac{1}{n}\sum _{i=1}^n{X}_i{Y}_i\right).\end{array}$$

Now let’s investigate its asymptotic behavior. As $n\to \mathrm{\infty },\hat{\mathrm{\Lambda }}\underset{p}{\to }\mathrm{\Lambda }$ and $\hat{\mathit{D}}\underset{p}{\to }\mathit{D}$ so

$$\hat{\beta }\underset{p}{⟶}{\beta }^{\ast }={\left({\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathbb{E}\left[X{X}^{\prime }\right]\mathrm{\Lambda }{\mathit{D}}^{-1}\right)}^{-1}({\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathbb{E}[XY]).$$

Recall $\mathbb{E}\left[X{X}^{\prime }\right]=\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi }$ and ${\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }=\mathit{D}$. We calculate that

$$\mathbb{E}[XY]=\mathbb{E}[(\mathrm{\Lambda }F+u)(F^{\prime }\beta +e)]=\mathrm{\Lambda }\beta .$$

We find that the right-hand-side of (11.26) equals

$${\beta }^{\ast }={\left(D^{-1}{\mathrm{\Lambda }}^{\prime }(\mathrm{\Lambda }{\mathrm{\Lambda }}^{\prime }+\mathrm{\Psi })\mathrm{\Lambda }{\mathit{D}}^{-1}\right)}^{-1}\left({\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\beta \right)={({\mathit{I}}_r+{\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathrm{\Psi }\mathrm{\Lambda }{\mathit{D}}^{-1})}^{-1}\beta$$

which does not equal $\beta$. Thus $\hat{\beta }$ has a probability limit but is inconsistent for $\beta$ as $n\to \mathrm{\infty }$.

This deficiency diminishes as $k\to \mathrm{\infty }$. Indeed,

$$\Vert {\mathit{D}}^{-1}{\mathrm{\Lambda }}^{\prime }\mathrm{\Psi }\mathrm{\Lambda }{\mathit{D}}^{-1}\Vert \le B\Vert {\mathit{D}}^{-1}\Vert \to 0$$

as $k\to \mathrm{\infty }$. This implies ${\beta }^{\ast }\to \beta$. Hence, if we take the sequential asymptotic limit $n\to \mathrm{\infty }$ followed by $k\to$ $\mathrm{\infty }$, we find $\hat{\beta }\underset{p}{⟶}\beta$. This implies that the estimator is consistent. Bai (2003) demonstrated consistency under the more rigorous but technically challenging setting where $n,k\to \mathrm{\infty }$ jointly. The implication of this result is that factor augmented regression is consistent if both the sample size and dimension of $X$ are large.

For asymptotic normality of $\hat{\beta }$ it turns out that we need to strengthen Assumption 11.1.2. The relevant condition is $n^{-1/2}{\lambda }_{min}\left({\mathrm{\Lambda }}^{\prime }\mathrm{\Lambda }\right)\to \mathrm{\infty }$. This is similar to the condition that $k^2/n\to \mathrm{\infty }$. This is technical but can be interpreted as meaning that $k$ is large relative to $\sqrt{n}$. Intuitively, this requires that dimension of $X$ is larger than sample size $n$.

In Stata, estimation takes the following steps. First, the factor command is used to estimate the factor model. Either MLE or principal components estimation can be used. Second, the predict command is used to estimate the factors, either by Barlett or regression scoring. Third, the factors are treated as regressors in an estimated regression.

11.17 Multivariate Normal*

Some interesting sampling results hold for matrix-valued normal variates. Let $\mathit{Y}$ be an $n\times m$ matrix whose rows are independent and distributed $\mathrm{N}(\mu ,\mathrm{\Sigma })$. We say that $\mathit{Y}$ is multivariate matrix normal, and

${}^2$ The unscaled principal components can equivalently be used if the coefficients $\hat{\beta }$ are not reported. The coefficient estimates $\hat{\gamma }$ are unaffected by the choice of factor scaling. write $Y\sim \mathrm{N}(\overline{\mu },I_n\otimes \mathrm{\Sigma })$, where $\overline{\mu }$ is $n\times m$ with each row equal to ${\mu }^{\prime }$. The notation is due to the fact that $\mathrm{vec}((\mathit{Y}-\mu {)}^{\prime })\sim \mathrm{N}(0,{\mathit{I}}_n\otimes \mathrm{\Sigma })$

Definition 11.2 If $n\times mY\sim \mathrm{N}(\overline{\mu },I_n\otimes \mathrm{\Sigma })$ then $W=Y^{\prime }Y$ is distributed Wishart with $n$ degress of freedom and covariance matrix $\mathrm{\Sigma }$, and is written as $W\sim$ $W_m(n,\mathrm{\Sigma })$.

The Wishart is a multivariate generalization of the chi-square. If $W\sim W_1(n,{\sigma }^2)$ then $W\sim {\sigma }^2{\chi }_n^2$.

The Wishart arises as the exact distribution of a sample covariance matrix in the normal sampling model. The bias-corrected estimator of $\mathrm{\Sigma }$ is

$$\hat{\mathrm{\Sigma }}=\frac{1}{n-1}\sum _{i=1}^n(Y_i-\overline{Y}){(Y_i-\overline{Y})}^{\prime }.$$

Theorem 11.10 If $Y_i\sim \mathrm{N}(\mu ,\mathrm{\Sigma })$ are independent then $\hat{\mathrm{\Sigma }}\sim W_m(n-1,\frac{1}{n-1}\mathrm{\Sigma })$.

The following manipulation is useful.

Theorem 11.11 If $W\sim W_m(n,\mathrm{\Sigma })$ then for $m\times 1\alpha ,{\left({\alpha }^{\prime }{W}^{-1}\alpha \right)}^{-1}\sim \frac{{\chi }_{n-m+1}^2}{{\alpha }^{\prime }{\mathrm{\Sigma }}^{-1}\alpha }$

To prove this, note that without loss of generality we can take $\mathrm{\Sigma }={\mathit{I}}_m$ and ${\alpha }^{\prime }\alpha =1$. Let $\mathit{H}$ be $m\times m$ orthonormal with first row equal to $\alpha$. so that $\mathit{H}\alpha =\left(\begin{array}{l}1\\ 0\end{array}\right)$. Since the distribution of $\mathit{Y}$ and $\mathit{Y}\mathit{H}$ are identical we can without loss of generality set $\alpha =\left(\begin{array}{c}1\\ 0\end{array}\right)$. Partition $\mathit{Y}=[{\mathit{Y}}_1,{\mathit{Y}}_2]$ where ${\mathit{Y}}_1$ is $n\times 1,{\mathit{Y}}_2$ is $n\times (m-1)$, and they are independent. Then

$$\begin{array}{rl}{\left({\alpha }^{\prime }{W}^{-1}\alpha \right)}^{-1}& ={\left(\left(\begin{array}{ll}1& 0\end{array}\right){\left(\begin{array}{cc}{\mathit{Y}}_1^{\prime }{\mathit{Y}}_1& {\mathit{Y}}_1^{\prime }{\mathit{Y}}_2\\ {\mathit{Y}}_2^{\prime }{\mathit{Y}}_1& {\mathit{Y}}_2^{\prime }{\mathit{Y}}_2\end{array}\right)}^{-1}\left(\begin{array}{c}1\\ 0\end{array}\right)\right)}^{-1}\\ & ={\mathit{Y}}_1^{\prime }{\mathit{Y}}_1-{\mathit{Y}}_1^{\prime }{\mathit{Y}}_2{\left({\mathit{Y}}_2^{\prime }{\mathit{Y}}_2\right)}^{-1}{\mathit{Y}}_2^{\prime }{\mathit{Y}}_1\\ & ={\mathit{Y}}_1^{\prime }{\mathit{M}}_2{\mathit{Y}}_1\sim {\chi }_{n-(m-1)}^2\end{array}$$

where ${\mathit{M}}_2={\mathit{I}}_{m-1}-{\mathit{Y}}_2{\left({\mathit{Y}}_2^{\prime }{\mathit{Y}}_2\right)}^{-1}{\mathit{Y}}_2^{\prime }$. The final distributional equality holds conditional on ${\mathit{Y}}_2$ by the same argument in the proof of Theorem 5.7. Since this does not depend on ${\mathit{Y}}_2$ it is the unconditional distribution as well. This establishes the stated result.

To test hypotheses about $\mu$ a classical statistic is known as Hotelling’s $T^2$ :

$$T^2=n(\overline{Y}-\mu {)}^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}(\overline{Y}-\mu ).$$

Theorem 11.12 If $Y\sim \mathrm{N}(\mu ,\mathrm{\Sigma })$ then

$$T^2\sim \frac{m}{(n-m)(n-1)}F(m,n-m)$$

a scaled F distribution.

To prove this recall that $\overline{Y}$ is independent of $\hat{\mathrm{\Sigma }}$. Apply Theorem $11.11$ with $\alpha =\overline{Y}-\mu$. Conditional on $\overline{Y}$ and using the fact that $\hat{\mathrm{\Sigma }}\sim W_m(n-1,\frac{1}{n-1}\mathrm{\Sigma })$,

$$\begin{array}{rl}\frac{n}{T^2}& ={((\overline{Y}-\mathrm{\Sigma }{)}^{\prime }{\hat{\mathrm{\Sigma }}}^{-1}(\overline{Y}-\mathrm{\Sigma }))}^{-1}\\ & \sim \frac{{\chi }_{n-1-m+1}^2}{(\overline{Y}-\mu {)}^{\prime }{\left(\frac{1}{n-1}\mathrm{\Sigma }\right)}^{-1}(\overline{Y}-\mu )}\\ & \sim n(n-1)\frac{{\chi }_{n-m}^2}{{\chi }_m^2}\end{array}$$

Since the two chi-square variables are independent, this is the stated result.

A very interesting property of this result is that the $T^2$ statistic is a multivariate quadratric form in normal random variables, yet it has the exact $F$ distribution.

11.18 Exercises

Exercise 11.1 Show (11.10) when the errors are conditionally homoskedastic (11.8).

Exercise 11.2 Show (11.11) when the regressors are common across equations $X_j=X$.

Exercise 11.3 Show (11.12) when the regressors are common across equations $X_j=X$ and the errors are conditionally homoskedastic (11.8).

Exercise 11.4 Prove Theorem 11.1.

Exercise 11.5 Show (11.13) when the regressors are common across equations $X_j=X$.

Exercise 11.6 Show (11.14) when the regressors are common across equations $X_j=X$ and the errors are conditionally homoskedastic (11.8).

Exercise 11.7 Prove Theorem 11.2.

Exercise 11.8 Prove Theorem 11.3.

Exercise $11.9$ Show that (11.16) follows from the steps described.

Exercise 11.10 Show that (11.17) follows from the steps described.

Exercise $11.11$ Prove Theorem 11.4. Exercise 11.12 Prove Theorem 11.5.

Hint: First, show that it is sufficient to show that

$$\mathbb{E}\left[{\overline{X}}^{\prime }\overline{X}\right]{\left(\mathbb{E}\left[{\overline{X}}^{\prime }{\mathrm{\Sigma }}^{-1}\overline{X}\right]\right)}^{-1}\mathbb{E}\left[{\overline{X}}^{\prime }\overline{X}\right]\le \mathbb{E}\left[{\overline{X}}^{\prime }\mathrm{\Sigma }\overline{X}\right].$$

Second, rewrite this equation using the transformations $U={\mathrm{\Sigma }}^{1/2}\overline{X}$ and $V={\mathrm{\Sigma }}^{1/2}\overline{X}$, and then apply the matrix Cauchy-Schwarz inequality (B.33).

Exercise 11.13 Prove Theorem 11.6.

Exercise $11.14$ Take the model

$$\begin{array}{rl}Y& ={\pi }^{\prime }\beta +e\\ \pi & =\mathbb{E}[X\mid Z]={\mathrm{\Gamma }}^{\prime }Z\\ \mathbb{E}[e\mid Z]& =0\end{array}$$

where $Y$ is scalar, $X$ is a $k$ vector and $Z$ is an $\ell$ vector. $\beta$ and $\pi$ are $k\times 1$ and $\mathrm{\Gamma }$ is $\ell \times k$. The sample is $(Y_i,X_i,Z_i:i=1,\dots ,n)$ with ${\pi }_i$ unobserved.

Consider the estimator $\hat{\beta }$ for $\beta$ by OLS of $Y$ on $\hat{\pi }={\hat{\mathrm{\Gamma }}}^{\prime }Z$ where $\hat{\mathrm{\Gamma }}$ is the OLS coefficient from the multivariate regression of $X$ on $Z$.

1. Show that $\hat{\beta }$ is consistent for $\beta$.

2. Find the asymptotic distribution $\sqrt{n}(\hat{\beta }-\beta )$ as $n\to \mathrm{\infty }$ assuming that $\beta =0$.

3. Why is the assumption $\beta =0$ an important simplifying condition in part (b)?

4. Using the result in (c) construct an appropriate asymptotic test for the hypothesis ${\mathbb{H}}_0:\beta =0$.

Exercise $11.15$ The observations are i.i.d., $(Y_{1i},Y_{2i},X_i:i=1,\dots ,n)$. The dependent variables $Y_1$ and $Y_2$ are real-valued. The regressor $X$ is a $k$-vector. The model is the two-equation system

$$\begin{array}{rl}{Y}_1& =X^{\prime }{\beta }_1+e_1\\ \mathbb{E}\left[X{e}_1\right]& =0\\ Y_2& =X^{\prime }{\beta }_2+e_2\\ \mathbb{E}\left[X{e}_2\right]& =0.\end{array}$$

1. What are the appropriate estimators ${\hat{\beta }}_1$ and ${\hat{\beta }}_2$ for ${\beta }_1$ and ${\beta }_2$ ?

2. Find the joint asymptotic distribution of ${\hat{\beta }}_1$ and ${\hat{\beta }}_2$.

3. Describe a test for ${\mathbb{H}}_0:{\beta }_1={\beta }_2$.