\documentclass[11pt]{article}
\usepackage{amsgen,amsmath,amstext,amsbsy,amsopn,amssymb}
\textwidth 6.3in \textheight 8.8in \topmargin -0.5truein
\oddsidemargin .15truein
\parskip .1in
\renewcommand{\baselinestretch}{1.53} % double spaced
\newtheorem{theorem}{\bf Theorem}
\newcommand{\be}{\begin{equation}}
\newcommand{\ee}{\end{equation}}
\newcommand {\reals} {{\rm I \! R}}
\newcommand{\bbR}{\mathbb{R}}
\newcommand{\cC}{\mathcal{C}}
\newcommand{\cS}{\mathcal{S}}
\newcommand{\cT}{\mathcal{T}}
\newcommand{\cU}{\mathcal{U}}
\newcommand{\cV}{\mathcal{V}}
\newcommand{\Ra}{{\mathcal{R}}}
\newcommand{\0}{{\mathbf{0}}}
\renewcommand{\a}{{\mathbf{a}}}
\renewcommand{\b}{{\mathbf{b}}}
\newcommand{\e}{{\mathbf{e}}}
\newcommand{\f}{{\mathbf{f}}}
\renewcommand{\u}{{\mathbf{u}}}
\renewcommand{\v}{{\mathbf{v}}}
\newcommand{\x}{{\mathbf{x}}}
\newcommand{\y}{{\mathbf{y}}}
\newcommand{\z}{{\mathbf{z}}}
\newcommand{\A}{{\mathbf{A}}}
\newcommand{\B}{{\mathbf{B}}}
\newcommand{\C}{{\mathbf{C}}}
\newcommand{\D}{{\mathbf{D}}}
\newcommand{\I}{{\mathbf{I}}}
\renewcommand{\H}{{\mathbf{H}}}
\renewcommand{\P}{{\mathbf{P}}}
\newcommand{\Q}{{\mathbf{Q}}}
\newcommand{\R}{{\mathbf{R}}}
\renewcommand{\S}{{\mathbf{S}}}
\newcommand{\U}{{\mathbf{U}}}
\newcommand{\V}{{\mathbf{V}}}
\newcommand{\W}{{\mathbf{W}}}
\newcommand{\X}{{\mathbf{X}}}
\newcommand{\Y}{{\mathbf{Y}}}
\newcommand{\Z}{{\mathbf{Z}}}
\newcommand{\tX}{{\mathbf{\tilde{X}}}}
\newcommand{\tu}{{\mathbf{\tilde{u}}}}
\newcommand{\tv}{{\mathbf{\tilde{v}}}}
\newcommand{\bX}{{\mathbf{\bar{X}}}}
\newcommand{\by}{{\mathbf{\bar{y}}}}
\newcommand{\cR}{{\cal R}}
\newcommand{\cN}{{\cal N}}
\newcommand{\bOmega}{\boldsymbol{\Omega}}
\newcommand{\bDelta}{\boldsymbol{\Delta}}
\newcommand{\Bphi}{\boldsymbol{\phi}}
\newcommand{\Bpsi}{\boldsymbol{\psi}}
\newcommand{\cbind}{\textrm{cbind}}
\newcommand{\rank}{\textrm{rank}}
\newcommand{\var}{\textrm{var}}
\newcommand{\tr}{\textrm{tr}}
\newcommand{\diag}{\textrm{diag}}
\renewcommand{\sum}{\textrm{sum}}
\newcommand{\mean}{\textrm{mean}}
\newcommand{\Range}{\textbf{Range}}
\newcommand{\Null}{\textbf{Null}}
\newcommand{\pnorm}[1]{{[\![}#1{]\!]}}
\newcommand{\plangle}{{\langle\!\langle}}
\newcommand{\prangle}{{\rangle\!\rangle}}
\newcommand{\Answer}{{\bf Answer:~}}
\begin{document}
\begin{title}
{\Large\bf HOMEWORK 1, STAT 961: Lin. Alg. 1 \\ Due Wed 2019/09/18, 2:00pm}
\end{title}
\author{\bf Your Name: ... (replace this)}
\maketitle
%----------------------------------------------------------------
\noindent
{\bf Instructions:} Edit this LaTex file by inserting your solutions
after each problem statement. Generate a PDF file from it and e-mail
the PDF in an attachment with filename
\centerline{\bf hw01-Yourlastname-Yourfirstname.pdf}
\noindent
to the following class Gmail address:
\centerline{\bf stat961.at.wharton@gmail.com}
\noindent
with {\bf subject line exactly} as follows for easy gmail search:
\centerline{\bf Homework 1, 2019}
\noindent Rules to be strictly followed under honor code:
\begin{enumerate} \itemsep 0em
\item You must write your own solutions and not copy from anyone.
Verbatim copying from others or unlisted sources, no matter how
minor, will result in zero points for the whole homework.
\item Subject to the previous item, you may explain the problems, but
not the solutions, to each other in general terms.
\item Do not discuss the homework with previous years' students of
Stat 961/541.
\item Do not consult solutions of similar homeworks of previous
years.
\item Report here who you collaborated with and what sources you used.
(You do not need to report help with LaTex and English language.)
\begin{itemize}
\item {\bf My collaborators:} ... (replace this)
\item {\bf The complete list of my sources is as follows:} ... (replace this)
\end{itemize}
\end{enumerate}
Instructions for presentation and typesetting:
\begin{enumerate}
\item Give derivations where appropriate, but don't when
instructed to give the answer without derivation.
\item LaTex math you will need: Symbols for matrices and vectors are
locally defined, in particular $\X$, $\Y$ for general matrices, $\H$
for projection matrices, $\I_{...}$ for identity matrices,
$\0_{...}$ for zero vectors and matrices, and $\x$, $\y$, $\z$,
$\a$, $\b$ for vectors. \\
Geometry: $\langle \a, \b \rangle = \a^T \b$ for inner products,
$\|\a\|^2 = \a^T \a$ for squared Euclidean norms. \\
Range space and null space of a linear map $\H$: $\Range(\H)$,
$\Null(\H)$.
\item To mimic obvious R functions in LaTex math mode we use
$\cbind(...)$, $\sum(\a)$ and $\mean(\a)$.
\item Write actual R code in inline verbatim mode: \verb|abc %*% xyz|.
This avoids conflicts between control characters of the R and LaTex
languages, as in this example: R's matrix multiplication \verb|%*%|
would require backslashes \verb|\| to show the percentage signs in
LaTex, but inside verbatim mode the percentage sign is shown
without. Normally the percentage sign is a
control character in LaTex to start a comment to the end of the line. \\
Note that inline verbatim mode lets you choose the beginning and
ending delimiters: \verb[abc %*% xyz[ does the same thing.
(If you read the PDF, you can't see the difference; read the LaTex
source.)
\end{enumerate}
%----------------------------------------------------------------
\newpage
\centerline{\bf Problems for Review of Matrix Algebra, in Particular Projection Matrices}
\centerline{Homework 1, Stat 961, 2019C}
%----------------------------------------------------------------
\begin{enumerate}
\item Two interpretations of matrix products:
\begin{enumerate}
\item First, consider a $m \times p$ matrix
$\X = \cbind(\x_1,\x_2,...,\x_p)$ and a $n \times p$ matrix
$\Y = \cbind(\y_1,\y_2,...,\y_p)$, with columns
$\x_j \in \reals^m, \y_j \in \reals^n$. Explain to yourself that
the matrix product $\X \Y^T$ is well-defined and what its size is
(nothing to write).
Problems: Express $\X \Y^T$ in terms of the columns. The answer
will be a sum of simpler matrices. What are these simple matrices
called? What R function computes such simple matrices? (No
derivations, please, just answers.)
\begin{enumerate}
\item $\X \Y^T = ...$
\item The summand matrices are called ...
\item R code for summand matrix $j$ using \verb|X[,j]| and \verb|Y[,j]|: \\
\verb|...|
\end{enumerate}
\item Next, consider a $n \times p$ matrix
$\X = \cbind(\x_1,\x_2,...,\x_p)$ and an $n \times q$ matrix
$\Y = \cbind(\y_1,\y_2,...,\y_q)$, with columns
$\x_j, \y_k \in \reals^n$. This time, explain to yourself that
$\X^T \Y$ is well-defined and what its size is (nothing to write).
Problems: Express the element $(j,k)$ of $\X^T \Y$ in terms of the
column vectors, explain its meaning, and write R code to compute
it. (Again, no derivations, just answers.)
\begin{enumerate}
\item Express the element $(j,k)$ of $\X^T \Y$ in terms of the columns:
\[
(\X^T \Y)_{j,k} = ...
\]
\item What is the geometric meaning of $(\X^T \Y)_{j,k}$? \\
...
\item R code for computing $(\X^T \Y)_{j,k}$ using \verb|X[,j]| and
\verb|Y[,k]|: \\
\verb|...|
\end{enumerate}
\end{enumerate}
\item With a $n \times p$ matrix $\X = \cbind(\x_1,\x_2,...,\x_p)$ and
a $n \times q$ matrix $\Y = \cbind(\y_1,\y_2,...,\y_q)$ as in the
previous problem~(b), what do the following conditions mean?
\begin{enumerate}
\item $\X^T \Y = \0_{p \times q}$: ...
\item $\X^T \X = \I_{p \times p}$: ...
\end{enumerate}
\item Let $\x$ and $\y$ be $n$-vectors, and consider simple regression
through the origin of $\y$ on~$\x$, no intercept, that is, $\X = \x$
is of size $n \times 1$. For this special case do the following (no
derivations, just answers):
\begin{enumerate}
\item Write down the triple-$\X$ matrix. Indicate the size of the
triple-$\X$ matrix in the form $a \times b$. Use squared norm
notation.
\[
(\X^T \X)^{-1} \X^T ...
~~~~~~\textrm{Size:}~~...
\]
\item Write the single OLS coefficient/slope estimate $\hat{\beta}$
by matrix-multiplying the triple-$\X$ matrix with the response
vector~$\y$. Show the result using inner product and squared norm
notation.
\[
\hat{\beta} = ...
\]
\item Write down the quadruple-$\X$ matrix, again using squared norm
notation. Of what type is this matrix?
\[
\X (\X^T \X)^{-1} \X^T = ...
~~~~~~\textrm{Type: ...}
\]
\item Think like a physicist: What are the units of the elements of
this quadruple-$\X$ matrix? Explain. (To make it concrete, you
may assume the units of $\x$ are US dollars.) What happens to it
if the units are changed, from US dollars to Euros, say?
\begin{enumerate}
\item The elements ...
\item Explanation: ...
\item Under a change of units of $\x$, the quadruple-$\X$ matrix
...
\end{enumerate}
\item Construct another vector $\tilde{\x}$ from $\x$ that has the
same units as the quadruple-$\X$ matrix and also generates the
same quadruple-$\X$ matrix (no derivations).
\[
\tilde{\x} = ...
\]
Quadruple-$\X$ matrix in terms of $\tilde{\x}$: ~~~$...$
\item Consider next the case that $\x = \e = (1,1,...,1)^T$. What
is the OLS coefficient estimate~$\hat{\beta}$ for the response
$\y$ when the single regressor is $\x = \e$? Give a short
derivation in which you translate geometric concepts to R
functions (but in LaTex math mode, not verbatim mode).
\[
\hat{\beta} = ...
~~~\textrm{because}~~~ \|\e\|^2 = ...
\]
\item What is $\tilde{\e}$, the special case of $\tilde{\x}$ where
$\x = \e$, from the question before?
\[
\tilde{\e} = ...
\]
\end{enumerate}
\item Back to multiple regression: We now assume
$\X = \cbind(\x_0,\x_1,...,\x_p)$ is of size $n \times (p+1)$ and
the first column is $\x_0 = \tilde{\e}$ for a rescaled intercept
coefficient. Consider the case that
$\X^T \X = \I_{(p+1)\times(p+1)}$.
\begin{enumerate}
\item Triple-$\X$ matrix $= ...$
\item $\hat{\beta}_j = ...$
\item Quadruple-$\X$ matrix $= ...$
\end{enumerate}
\item Back to a general $\X$ matrix, only assuming it has full rank.
Show that its quadruple-$\X$ matrix $\H = \X (\X^T \X)^{-1} \X^T$ is
an orthogonal projection, that is, it is idempotent and symmetric.
\begin{eqnarray*}
\H \H &=& ... \\
&=& ... \\
&=& ... \\
&=& \H
\end{eqnarray*}
\begin{eqnarray*}
\H^T &=& ... \\
&=& ... \\
&=& ... \\
&=& ... \\
&=& \H
\end{eqnarray*}
\item Show that if $\X_1$ ($n \times p_1$) and $\X_2$ ($n \times p_2$)
satisfy $\X_1^T \X_2 = \0_{p_1 \times p_2}$, then the sum of the
corresponding hat matrices, $\H_1 + \H_2$, is also an orthogonal
projection.
Do not show any calculations; explain the reasons in English. (You
may use inline notation for simple expressions such as $\H_1 \H_2$
in your explanations.)
{\bf Idempotence:} ...
{\bf Symmetry:} ...
{\bf Make a Guess:} Is $\H_1 + \H_2$ also a quadruple-$\X$ matrix?
If so, what is this~$\X$? (No derivations, just a guess.)
\Answer: ...
\item If $\H$ ($n \times n$) is idempotent and symmetric, does the
same hold for $\I - \H$? Show derivations.
{\bf Idempotence:} $...$
{\bf Symmetry:} $...$
\item If $\H$ and $\I - \H$ are matrix-multiplied in either order,
what's the result?
\begin{eqnarray*}
\H (\I - \H) &=& ... \\
(\I - \H) \H &=& ...
\end{eqnarray*}
\item Geometrically, idempotence means projection: dropping/projecting
a point $\x$ into a subspace, $\x \mapsto \H \x \in \Range(\H)$, and
if the dropping/projecting is repeated, then the point $\H \x$ does
not move because it's already in that subspace. Call
$\x - \H \x = (\I - \H) \x$ the (reverse) projection direction.
Show that this direction is in the null space
$\Null(\H) = \{ \z | \H \z = \0 \}$. No derivation, just an
argument based on the previous question.
\Answer ...
\item The mystery that symmetry and orthogonality of projections are
equivalent: Intuitively, orthogonality of a projection means that
points get dropped into a subspace such that the direction of
dropping/projecting is orthogonal to the subspace. Now, the
subspace that $\H$ projects onto is $\Range(\H)$, and the (reverse)
direction of dropping/projecting for a point $\x$ is $\x - \H \x$.
Thus to understand orthogonality of projection, we have to
understand what it means that $\x - \H \x = (\I - \H) \x$ is
orthogonal to {\em all} points in $\Range(\H)$, that is,
$(\I - \H) \x \perp \H \z$ for {\em all} $\x$ and {\em all} $\z$
in~$\reals^n$:
\begin{eqnarray*}
0 &=& ... \\
&=& ... \\
&=& ...
\end{eqnarray*}
It follows that $\H = ...$. But any matrix of the form $...$ is
symmetric! Hence $\H$ is symmetric.
\end{enumerate}
%----------------------------------------------------------------
\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%