机器学习的数学基础（二）

三道试题 第二题

• 第二题{Positive definite matrices}
• 题目
• 第一问
• 解
• 第二问
• 解
• 第三问
• 解

第二题{Positive definite matrices}

题目

A matrix $A \in {R}^{n×n}​$A∈Rn×n​ is positive semi-definite (PSD), denoted $A \ge 0​$A≥0​, if $A = {A}^{T}​$A=AT​ and ${x}^T Ax \ge 0​$xTAx≥0​ for all $x \in {R}^{n}​$x∈Rn​. A matrix $A​$A​ is positive definite,denoted $A > 0​$A>0​,if $A={A}^{T}​$A=AT​ and ${x}^TAx>0​$xTAx>0​ for all $x\ne 0​$x̸​=0​, that is, all non-zero vectors x.The simplest example of a positive definite matrix is the identity I (the diagonal matrix with 1s on the diagonal and 0s elsewhere), which satisfies ${x}^TIx={\left\| x\right\|}_2^2 = \sum_{i=1}^{n}{{x}_{i}^2}​$xTIx=∥x∥22​=∑i=1n​xi2​​.

第一问

Let $z \in {R}^{n}$z∈Rn be an n-vector. Show that $A = z{z}^T$A=zzT is positive semi-definite.

解

首先:$A^T = (zz^T)^T = zz^T=A$AT=(zzT)T=zzT=A

$zz^T=\begin{bmatrix} z_1 \\ z_2 \\ \vdots \\ z_n \end{bmatrix} \begin{bmatrix} z_1 & z_2 & \cdots & z_n \end{bmatrix}=\begin{bmatrix} z_1z_1 & z_1z_2 & \cdots & z_1z_n \\ z_2z_1 & z_2z_2 & \cdots & z_2z_n \\ \vdots &\vdots &\ddots & \vdots \\ z_nz_1 & z_nz_2 & \cdots &z_nz_n\end{bmatrix}​$zzT=⎣⎢⎢⎢⎡​z1​z2​⋮zn​​⎦⎥⎥⎥⎤​[z1​​z2​​⋯​zn​​]=⎣⎢⎢⎢⎡​z1​z1​z2​z1​⋮zn​z1​​z1​z2​z2​z2​⋮zn​z2​​⋯⋯⋱⋯​z1​zn​z2​zn​⋮zn​zn​​⎦⎥⎥⎥⎤​​

$\therefore A_{ij} = z_iz_j$∴Aij​=zi​zj​

由第一题可知:
${x}^{T}Ax =\sum_{i=1}^{n}{\sum_{j=1}^{n}{{A}_{ij}{x}_{i}}{x}_{j}}=\sum_{i=1}^{n}{\sum_{j=1}^{n}{z_iz_j{x}_{i}}{x}_{j}}=\sum_{i=1}^{n}z_ix_i\sum_{j=1}^{n}z_ix_j={(\sum_{i=1}^{n}z_ix_i)}^2 \ge 0$xTAx=∑i=1n​∑j=1n​Aij​xi​xj​=∑i=1n​∑j=1n​zi​zj​xi​xj​=∑i=1n​zi​xi​∑j=1n​zi​xj​=(∑i=1n​zi​xi​)2≥0 for $x\in R^n$x∈Rn

所以：A is PSD。

第二问

Let $z \in R^n​$z∈Rn​ be a non-zero n-vector. Let $A = zz^T​$A=zzT​ . What is the null-space of A? What is the rank of A?

解

• 知识点 null-space
  The nullspace of A consists of all solutions to $Ax=0$Ax=0. These vectors x are in $R^n$Rn. The nullspace containing all solutions of $Ax=0$Ax=0 is denoted by $N(A)​$N(A)​.

因为：z为非0向量，所以： $Ax=0 \Rightarrow zz^Tx=0 \Rightarrow \begin{bmatrix} z_1z_1 & z_1z_2 & \cdots & z_1z_n \\ z_2z_1 & z_2z_2 & \cdots & z_2z_n \\ \vdots &\vdots &\ddots & \vdots \\ z_nz_1 & z_nz_2 & \cdots &z_nz_n\end{bmatrix}x=0 \Rightarrow \begin{bmatrix} z_1z_1 & z_1z_2 & \cdots & z_1z_n \\ 0 & 0 & \cdots & 0 \\ \vdots &\vdots &\ddots & \vdots \\ 0 & 0 & \cdots &0 \end{bmatrix}x=0 \Rightarrow \begin{bmatrix} z_1 & z_2 & \cdots & z_n \\ 0 & 0 & \cdots & 0 \\ \vdots &\vdots &\ddots & \vdots \\ 0 & 0 & \cdots &0 \end{bmatrix}x=0​$Ax=0⇒zzTx=0⇒⎣⎢⎢⎢⎡​z1​z1​z2​z1​⋮zn​z1​​z1​z2​z2​z2​⋮zn​z2​​⋯⋯⋱⋯​z1​zn​z2​zn​⋮zn​zn​​⎦⎥⎥⎥⎤​x=0⇒⎣⎢⎢⎢⎡​z1​z1​0⋮0​z1​z2​0⋮0​⋯⋯⋱⋯​z1​zn​0⋮0​⎦⎥⎥⎥⎤​x=0⇒⎣⎢⎢⎢⎡​z1​0⋮0​z2​0⋮0​⋯⋯⋱⋯​zn​0⋮0​⎦⎥⎥⎥⎤​x=0​

可得 $rank(A) = 1$rank(A)=1.

可知：$x_1$x1​为pivot variable，其他为free variables，所以 special solutions如下：

$s_1 = \begin{bmatrix} -z_2 \\ z_1 \\ 0\\ \vdots \\ 0 \end{bmatrix}$s1​=⎣⎢⎢⎢⎢⎢⎡​−z2​z1​0⋮0​⎦⎥⎥⎥⎥⎥⎤​

$s_2 = \begin{bmatrix} -z_3 \\ 0\\ z_1 \\ \vdots \\ 0 \end{bmatrix}$s2​=⎣⎢⎢⎢⎢⎢⎡​−z3​0z1​⋮0​⎦⎥⎥⎥⎥⎥⎤​

$\vdots​$⋮​

$s_{n-2}= \begin{bmatrix} -z_{n-1} \\ 0 \\ 0\\ \vdots \\ z_1\\ 0 \end{bmatrix}​$sn−2​=⎣⎢⎢⎢⎢⎢⎢⎢⎡​−zn−1​00⋮z1​0​⎦⎥⎥⎥⎥⎥⎥⎥⎤​​

$s_{n-1}= \begin{bmatrix} -z_n \\ 0 \\ 0\\ \vdots \\ 0 \\ z_1 \end{bmatrix}​$sn−1​=⎣⎢⎢⎢⎢⎢⎢⎢⎡​−zn​00⋮0z1​​⎦⎥⎥⎥⎥⎥⎥⎥⎤​​

A的Null Space是$s_1​$s1​​ $s_2​$s2​​到$s_{n-1}​$sn−1​​共n-1个special solutions的线性组合，即：

$N(A)= c_1s_1 + c_2s_2 + \cdots + c_{n-1}s_(n-1)= \begin{bmatrix} -\sum_{i=1}^{n-1}c_iz_{i+1} \\ c_1z_1\\ \vdots \\c_{n-1} z_1 \end{bmatrix}​$N(A)=c1​s1​+c2​s2​+⋯+cn−1​s(​n−1)=⎣⎢⎢⎢⎡​−∑i=1n−1​ci​zi+1​c1​z1​⋮cn−1​z1​​⎦⎥⎥⎥⎤​​

第三问

Let $A \in R^{n\times n}$A∈Rn×n be positive semidefinite and $B \in R^{m \times n}$B∈Rm×n be arbitrary, where $m, n \in N$m,n∈N. Is $BAB^T$BABT PSD? If so, prove it. If not, give a counterexample with explicit A, B.

解

1. 令$C=BAB^T​$C=BABT​,则$C^T = BAB^T​$CT=BABT​, 且$C \in R^{m \times m}​$C∈Rm×m​.

2. 令 $x \in R^m$x∈Rm, $x^TCx = x^TBAB^Tx=(B^Tx)^TA(B^Tx)$xTCx=xTBABTx=(BTx)TA(BTx) 设 $y = B^Tx$y=BTx, $y \in R^n$y∈Rn, 因为 A是PSD，所以: $y^TAy \ge 0$yTAy≥0。

   综合1,2,可得 C为PSD，即：$BAB^T$BABT为PSD。