linear algebra matrix sample 乘積 反矩陣 identity matrix transpose matrix inner change (change row) math to matrix reduced echelon form(gaussian elimination) vector equation and matrix equation span linear dependence linear depandence vs. span: echelon form vector equation span R^4 linear independence? transform one-to-one onto linear transformation vectorspace child space null space columns space rows space rank nullity rank nullity/kernel Eigenvector and Eigenvalue eigen value eigen vector diagonalization orthogonal dot |A| vector matrix orthogonal vs orthonormal orthogonal matrix orthonormal find orth basis(gram-schmidt process) derivative of a Matrix projection curve fitting | regression | normal equation way 1 way 2 way 3

linear algebra

matrix

Column 1Column 2Column 3
Row1valuevaluevalue
Row2valuevaluevalue
Row3valuevaluevalue

diagonal 對角 commutative law associative law distributive law

sample

A=[1234] A=\begin{bmatrix} 1&2\\ 3&4\\ \end{bmatrix} B=[1−22−4] B=\begin{bmatrix} 1&-2\\ 2&-4\\ \end{bmatrix}

乘積

A=[abcd]B=[1234]A∗B=[a∗1+b∗3a∗2+b∗4c∗1+d∗3c∗2+d∗4]A=[231−5]B=[4361−23]A∗B=[2∗4+3∗12∗3+3∗−22∗6+3∗31∗4+−5∗11∗3+−5∗−21∗6+−5∗3]=[11021−113−9]A[n×m]∗B[m×s]=C[n×s] A=\begin{bmatrix} a&b\\ c&d\\ \end{bmatrix} \\ B=\begin{bmatrix} 1&2\\ 3&4\\ \end{bmatrix} \\ A*B=\begin{bmatrix} a*1+b*3 & a*2+b*4\\ c*1+d*3& c*2+d*4\\ \end{bmatrix} \\ A=\begin{bmatrix} 2&3\\ 1&-5\\ \end{bmatrix} B=\begin{bmatrix} 4&3&6\\ 1&-2&3\\ \end{bmatrix} \\ A*B=\begin{bmatrix} 2*4+3*1 & 2*3+3*-2&2*6+3*3\\ 1*4+-5*1& 1*3+-5*-2 &1*6+-5*3\\ \end{bmatrix}= \begin{bmatrix} 11 & 0 &21\\ -1& 13 &-9\\ \end{bmatrix}\\ \underset{[n\times m]}{A}*\underset{[m\times s]}{B}=\underset{[n\times s]}{C}

反矩陣

A=[abcd]a∗d−b∗c=0  then  not  inverableA∗A−1=I=[1001]A−1=1∣abcd∣∗[d−b−ca]high level matrix[AI]=[A∗A−1I∗A−1]=[IA−1][100∣100110∣010111∣001]=[100∣100010∣−110001∣0−11][100∣100110∣010111∣001]=[100∣100010∣−110001∣0−11]A=\begin{bmatrix} a&b\\ c&d\\ \end{bmatrix}\\ {a*d-b*c=0} \;then\; not \;inverable\\ A * A^{-1}=I =\begin{bmatrix} 1&0\\ 0&1\\ \end{bmatrix} \\ A^{-1}=\frac{1} {\begin{vmatrix} a&b\\ c&d\\ \end{vmatrix}} *\begin{bmatrix} d&-b\\ -c&a\\ \end{bmatrix}\\ \text{high level matrix} \\ \begin{bmatrix} A&I\\ \end{bmatrix}= \begin{bmatrix} A*A^{-1}&I*A^{-1}\\ \end{bmatrix}= \begin{bmatrix} I&A^{-1}\\ \end{bmatrix}\\ \begin{bmatrix} 1&0&0 &| & 1&0&0\\ 1&1&0 &| & 0&1&0\\ 1&1&1 &| & 0&0&1\\ \end{bmatrix}= \begin{bmatrix} 1&0&0 &| & 1&0&0\\ 0&1&0 &| & -1&1&0\\ 0&0&1 &| & 0&-1&1\\ \end{bmatrix}\\ \begin{bmatrix} 1&0&0 &| & 1&0&0\\ 1&1&0 &| & 0&1&0\\ 1&1&1 &| & 0&0&1\\ \end{bmatrix}= \begin{bmatrix} 1&0&0 &| & 1&0&0\\ 0&1&0 &| & -1&1&0\\ 0&0&1 &| & 0&-1&1\\ \end{bmatrix}

identity matrix

I=[100010001] I=\begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1\\ \end{bmatrix}

transpose matrix

A=[1234]AT=[1324](A−1)T=(AT)−1 A=\begin{bmatrix} 1&2\\ 3&4\\ \end{bmatrix}\\ A^{T}=\begin{bmatrix} 1&3\\ 2&4\\ \end{bmatrix} (A^{-1})^T=(A^T)^{-1}

inner change (change row)

[123456]==[563412] \begin{bmatrix} 1&2\\ 3&4\\ 5&6\\ \end{bmatrix} ==\begin{bmatrix} 5&6\\ 3&4\\ 1&2\\ \end{bmatrix}

math to matrix

{1∗x1+2∗x2+3∗x3=52∗x1+3∗x2+4∗x3=7=[12352347] \begin{cases} 1*x_1&+&2*x_2&+&3*x_3=5\\ 2*x_1&+&3*x_2&+&4*x_3=7 \end{cases}= \begin{bmatrix} 1&2&3&5\\ 2&3&4&7\\ \end{bmatrix}

reduced echelon form(gaussian elimination)

[03−664−53−78−5893−912−9615]==[3−912−96153−78−58903−664−5]=[3−912−961502−442−603−664−5]=[3−912−961502−442−603−664−5]=[3−912−961502−442−6000014]=[3−912−961501−221−3000014]=[1−34−32501−221−3000014]=[1−34−30−301−220−7000014]=[10−2−30−2401−220−7000014]={1∗x1−2∗x3+−3∗x4=241∗x2−2∗x3+2∗x4=−7x5=4 \begin{bmatrix} 0& 3 &-6 &6 &4 &-5\\ 3& -7 &8 &-5 &8 &9\\ 3& -9 &12 &-9 &6 &15\\ \end{bmatrix}== \begin{bmatrix} 3& -9 &12 &-9 &6 &15\\ 3& -7 &8 &-5 &8 &9\\ 0& 3 &-6 &6 &4 &-5\\ \end{bmatrix}= \begin{bmatrix} 3& -9 &12 &-9 &6 &15\\ 0& 2 &-4 &4 &2 &-6\\ 0& 3 &-6 &6 &4 &-5\\ \end{bmatrix}\\= \begin{bmatrix} 3& -9 &12 &-9 &6 &15\\ 0& 2 &-4 &4 &2 &-6\\ 0& 3 &-6 &6 &4 &-5\\ \end{bmatrix}= \begin{bmatrix} 3& -9 &12 &-9 &6 &15\\ 0& 2 &-4 &4 &2 &-6\\ 0& 0 &0 &0 &1 &4\\ \end{bmatrix}\\= \begin{bmatrix} 3& -9 &12 &-9 &6 &15\\ 0& 1 &-2 &2 &1 &-3\\ 0& 0 &0 &0 &1 &4\\ \end{bmatrix}= \begin{bmatrix} 1& -3 &4 &-3 &2 &5\\ 0& 1 &-2 &2 &1 &-3\\ 0& 0 &0 &0 &1 &4\\ \end{bmatrix}\\= \begin{bmatrix} 1& -3 &4 &-3 &0 &-3\\ 0& 1 &-2 &2 &0 &-7\\ 0& 0 &0 &0 &1 &4\\ \end{bmatrix}= \begin{bmatrix} 1& 0 &-2 &-3 &0 &-24\\ 0& 1 &-2 &2 &0 &-7\\ 0& 0 &0 &0 &1 &4\\ \end{bmatrix}\\= \begin{cases} 1*x_1 -2*x_3 +-3*x_4 =24\\ 1*x_2 -2*x_3 +2*x_4 =-7\\ x_5=4 \end{cases}

vector equation and matrix equation

{x1+2x2−x3=3−2x1−2x2+4x3=02x2+2x3=6==x1[1−20]+x2[2−22]+x3[−142]=[306]A[12−1−2−24022]∗x[x1x2x3]=[306]Ax=bAmatrix∗xvector=bvector \begin{cases} &x_1 &+ &2x_2 &-&x_3 &=&3\\ -&2x_1 &- &2x_2 &+&4x_3 &=&0\\ & & &2x_2 &+&2x_3 &=&6\\ \end{cases}==\\ x_1 \begin{bmatrix} 1\\ -2\\ 0\\ \end{bmatrix} +x_2 \begin{bmatrix} 2\\ -2\\ 2\\ \end{bmatrix} +x_3 \begin{bmatrix} -1\\ 4\\ 2\\ \end{bmatrix}= \begin{bmatrix} 3\\ 0\\ 6\\ \end{bmatrix}\\ A\begin{bmatrix} 1& 2 &-1 \\ -2&-2 &4 \\ 0& 2 &2 \\ \end{bmatrix} *x\begin{bmatrix} x_1\\ x_2\\ x_3\\ \end{bmatrix} =\begin{bmatrix} 3\\ 0\\ 6\\ \end{bmatrix}\\ Ax=b\\ A_{matrix}*x_{vector}=b_{vector}

span

v1=[1−20]v2=[2−22]v3=[−142]set{v1,v2,v3}span: mean all the combination vector of can combine by v1、v2、v3.set{v1,v2,v3}span R3: mean all the vector can be combine by v1、v2、v3.b are the all the vectorx are the all the possible vectorexample:if{v1,v2,v3}span R3 thenA[12−1−2−24022]∗x[x1x2x3]=b[b1b2b3]and[12−10−2−2400220]have pivot position in every rowbut[12−1b1−2−24b2022b3]=[10−3b1011b2000b3]did not have pivot point in row3So{v1,v2,v3}not span R3 v_1=\begin{bmatrix} 1\\ -2\\ 0\\ \end{bmatrix} v_2=\begin{bmatrix} 2\\ -2\\ 2\\ \end{bmatrix} v_3=\begin{bmatrix} -1\\ 4\\ 2\\ \end{bmatrix}\\ set\lbrace \begin{matrix} &v_1,&v_2,&v_3 \end{matrix} \rbrace span \text{: mean all the combination vector of can combine by }v_1、v_2、v_3 .\\ \\ set\lbrace \begin{matrix} &v_1,&v_2,&v_3 \end{matrix} \rbrace span \ R^3 \text{: mean all the vector can be combine by }v_1、v_2、v_3 .\\ \text{b are the all the vector} \\ \text{x are the all the possible vector} \\ example:\\ if \lbrace \begin{matrix} &v_1,&v_2,&v_3 \end{matrix} \rbrace span \ R^3 \ then \\ A\begin{bmatrix} 1 &2 & -1 \\ -2 &-2 & 4 \\ 0 & 2 & 2 \\ \end{bmatrix} *x\begin{bmatrix} x_1\\ x_2\\ x_3\\ \end{bmatrix} =b\begin{bmatrix} b_1\\ b_2\\ b_3\\ \end{bmatrix}\\ \\ and\begin{bmatrix} 1 &2 & -1 &0\\ -2 &-2 & 4 &0\\ 0 & 2 & 2 &0\\ \end{bmatrix} \text{have pivot position in every row}\\ but\begin{bmatrix} 1 &2 & -1 &b_1\\ -2 &-2 & 4 &b_2\\ 0 & 2 & 2 &b_3\\ \end{bmatrix}= \begin{bmatrix} 1 &0 & -3 &b_1\\ 0 &1 & 1 &b_2\\ 0 & 0 & 0 &b_3\\ \end{bmatrix} \text{did not have pivot point in row3}\\ So \lbrace \begin{matrix} &v_1,&v_2,&v_3 \end{matrix} \rbrace not \ span \ R^3

linear dependence

if{v1,v2,v3} linear dependence then at least one of vector is linear combination of another.if \lbrace \begin{matrix} v_1 ,& v_2 ,& v_3 \end{matrix} \rbrace \text{ linear dependence then at least one of vector is linear combination of another.}

linear depandence vs. span:

Amatrix∗xvector=bvectorA[02−2111−20−4120]x[x1x2x3]=b[b1b2b3b4] A_{matrix}*x_{vector}=b_{vector}\\ A\begin{bmatrix} 0 &2 & -2 \\ 1 &1 & 1 \\ -2 &0 & -4 \\ 1 &2 & 0 \\ \end{bmatrix}x \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix}= b \begin{bmatrix} b_1 \\ b_2 \\ b_3 \\ b_4 \\ \end{bmatrix}

echelon form

[02−2b1111b2−20−4b3120b4]=[111b202−2b1−20−4b3120b4]=[111b202−2b102−22b2+b301−1−b2+b4]=[102b2+−b1202−2b10002b2+b3−b1000−b2+b4+−b12]\begin{bmatrix} 0 &2 & -2 &b_1 \\ 1 &1 & 1 &b_2\\ -2 &0 & -4 &b_3\\ 1 &2 & 0 &b_4\\ \end{bmatrix}= \begin{bmatrix} 1 &1 & 1 &b_2\\ 0 &2 & -2 &b_1 \\ -2 &0 & -4 &b_3\\ 1 &2 & 0 &b_4\\ \end{bmatrix}= \begin{bmatrix} 1 &1 & 1 &b_2\\ 0 &2 & -2 &b_1 \\ 0 &2 & -2 &2b_2+b_3\\ 0 &1 & -1 &-b_2+b_4\\ \end{bmatrix}=\\ \begin{bmatrix} 1 &0 & 2 &b_2+\frac{-b_1}{2}\\ 0 &2 & -2 &b_1 \\ 0 &0 & 0 &2b_2+b_3-b_1 \\ 0 &0 & 0 &-b_2+b_4+\frac{-b_1}{2}\\ \end{bmatrix}

vector equation

x1[1000]+x2[0200]+x3[2−200]=b[b2+−b12b12b2+b3−b1−b2+b4+−b12]x_1\begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \\ \end{bmatrix}+ x_2\begin{bmatrix} 0 \\ 2 \\ 0 \\ 0 \\ \end{bmatrix}+ x_3\begin{bmatrix} 2 \\ -2 \\ 0 \\ 0 \\ \end{bmatrix}= b\begin{bmatrix} b_2+\frac{-b_1}{2} \\ b_1 \\ 2b_2+b_3-b_1 \\ -b_2+b_4+\frac{-b_1}{2}\\ \end{bmatrix}

span R^4

NO,did not have pivot point in row3 and row4

linear independence?

NO,did not have pivot point in column 3

transform

one-to-one

T(x)=bfor all b only have one x that T(x)=bT(x)= b \\ \text{for all b only have one x that T(x)=b}

onto

T(x)=bfor all b have at least one x that T(x)=bT(x)= b \\ \text{for all b have at least one x that T(x)=b}

linear transformation

T(x)=bfor all b have at least one x that T(x)=bT(a+b)=T(a)+T(b)T(x)= b \\ \text{for all b have at least one x that T(x)=b}\\ T(a+b)=T(a)+T(b)

vectorspace

If A is vector space then{the zero vector or zero matrix must be in side of Aif u and v is inside A then (u+v) should be inside Aif v is inside A then n∈R and n*v should be also inside A\text{If A is vector space then}\\ \begin{cases} \text{the zero vector or zero matrix must be in side of A}\\ \text{if u and v is inside A then (u+v) should be inside A}\\ \text{if v is inside A then n} \in \text{R and n*v should be also inside A}\\ \end{cases}

child space

x=[224−224048]→rref[100012000]x=\begin{bmatrix} 2 & 2& 4 \\ -2 & 2& 4 \\ 0 & 4& 8 \\ \end{bmatrix} \xrightarrow[\text{}]{\text{rref}} \begin{bmatrix} 1 & 0& 0 \\ 0 & 1& 2 \\ 0 & 0& 0 \\ \end{bmatrix} Col(A)=Row(AT)Col(AT)=Row(A)Col(A)=Row(A^T)\\ Col(A^T)=Row(A)\\

null space

null space : all the solution make Av=0.orth to columns space{[0−21]\text{null space : all the solution make } Av=0.\\ \text{orth to columns space}\\ {\begin{cases} {\begin{bmatrix} 0\\ -2\\ 1\\ \end{bmatrix}} \end{cases}}

columns space

columns space :all the combination of column vector Ax=bThe pivot columns of the origin columns.{[2−20],[224]\text{columns space :all the combination of column vector }\\ Ax=b\\ \text{The pivot columns of the origin columns.}\\ {\begin{cases} \begin{bmatrix} 2\\ -2\\ 0\\ \end{bmatrix}, \begin{bmatrix} 2\\ 2\\ 4\\ \end{bmatrix} \end{cases}}

rows space

rows space :the space that is orth to columns spacethe pivot row{[224],[−224]or{[100],[012]\text{rows space :the space that is orth to columns space}\\ \text{the pivot row}\\ {\begin{cases} \begin{bmatrix} 2\\ 2\\ 4\\ \end{bmatrix}, \begin{bmatrix} -2\\ 2\\ 4\\ \end{bmatrix} \end{cases}}or {\begin{cases} \begin{bmatrix} 1\\ 0\\ 0\\ \end{bmatrix}, \begin{bmatrix} 0\\ 1\\ 2\\ \end{bmatrix} \end{cases}}

rank nullity

A∈Rm×nrank(A)+nullity(A)=nrank(AT)+nullity(AT)=mA \in R^{m\times n} \\ rank(A)+nullity(A)=n\\ rank(A^T)+nullity(A^T)=m

rank

the number of non-zero pivot column

rank(A)=dim(Col(A))rank(A)=rank(AT)rank(A)=dim(Col(A))\\ rank(A)=rank(A^T)

nullity/kernel

the number of zero pivot column

nullity(A)=dim(Nul(A))nullity(A)=dim(Nul(A))

Eigenvector and Eigenvalue

λ=eigenvalue (nature number could be zero)v=eigenvector(not zero vector)Av=λvAv=λIvAv−λIv=0∣A−λI∣=0\lambda= \text{eigenvalue (nature number could be zero)}\\ v=\text{eigenvector(not zero vector)}\\ Av=\lambda v\\ Av =\lambda Iv\\ Av − \lambda Iv = 0\\ | A − \lambda I | = 0\\

eigen value

A=[6923]det[6923]=6∗3−9∗2=0∣A−λI∣=0det[6−λ923−λ]=(6−λ)(3−λ)−18=0λ=0 or 9A=\begin{bmatrix} 6 & 9\\ 2 & 3\\ \end{bmatrix}\\ det\begin{bmatrix} 6 & 9\\ 2 & 3\\ \end{bmatrix}=6*3-9*2=0\\ | A − \lambda I | = 0\\ det\begin{bmatrix} 6-\lambda & 9\\ 2 & 3-\lambda \\ \end{bmatrix}=(6-\lambda)(3-\lambda)-18=0\\ \lambda =0 \ or \ 9

eigen vector

λ=0 or 9Av=λv[6923][xy]=9[xy]{6x+9y=9x2x+3y=9y6x=18y→x=−3yAv=λv[6923][xy]=0[xy]x∈Ry∈Rv=[31]or[ab]\lambda =0 \ or \ 9 \\ Av=\lambda v\\ \begin{bmatrix} 6 & 9\\ 2 & 3\\ \end{bmatrix} \begin{bmatrix} x\\ y\\ \end{bmatrix}=9 \begin{bmatrix} x\\ y\\ \end{bmatrix}\\ \begin{cases} 6x+9y=9x\\ 2x+3y=9y\\ \end{cases}\\ 6x=18y \rightarrow x=-3y \\ Av=\lambda v\\ \begin{bmatrix} 6 & 9\\ 2 & 3\\ \end{bmatrix} \begin{bmatrix} x\\ y\\ \end{bmatrix}=0 \begin{bmatrix} x\\ y\\ \end{bmatrix}\\ x \in R\\ y \in R\\ v=\begin{bmatrix} 3\\ 1\\ \end{bmatrix}or \begin{bmatrix} a\\ b\\ \end{bmatrix}

diagonalization

Find P and D thatA=PDP−1So A2=(PDP−1)(PDP−1)A2=PD2P−1An=PDnP−1Pn×m=[V1EigenVectorV2EigenVectorV3EigenVector...]Dn×m=[λ1EigenValue00...0λ2EigenValue0...00λ3EigenValue...⋮⋮⋮⋱]\text{Find P and D that}\\ A=PDP^{-1}\\ \text{So } A^2=(PDP^{-1})(PDP^{-1})\\ A^2=PD^2P^{-1}\\ A^n=PD^nP^{-1}\\ \underset{n\times m}{P}=\begin{bmatrix} V1_{EigenVector} & V2_{EigenVector}& & V3_{EigenVector} &...\\ \end{bmatrix}\\ \underset{n \times m}{D}=\begin{bmatrix} \lambda1_{EigenValue} &0 &0&... \\ 0 & \lambda2_{EigenValue} & 0&...\\ 0 & 0 &\lambda3_{EigenValue} &...\\ \vdots & \vdots &\vdots & \ddots \\ \end{bmatrix}

orthogonal

If a⋅b=0 then a⊥b (orthogonal)[12]⋅[−84]=1×−8+2×4=0[12]⊥[−84]\text{If } a\cdot b=0 \text{ then }a \perp b \text{ (orthogonal)}\\ \begin{bmatrix} 1\\ 2\\ \end{bmatrix}\cdot \begin{bmatrix} -8\\ 4\\ \end{bmatrix}=1\times-8+2\times 4=0\\ \begin{bmatrix} 1\\ 2\\ \end{bmatrix}\perp \begin{bmatrix} -8\\ 4\\ \end{bmatrix}

dot

An∗m⋅Bn∗mcolumns space vector: A⋅B=AT×Brows space vector: A⋅B=A×BT\underset{n*m}{A}\cdot \underset{n*m}{B}\\ \text{columns space vector: }A\cdot B=A^T\times B\\ \text{rows space vector: }A\cdot B=A\times B^T\\

|A|

vector

∣A∣=∑i=1n∑j=1m(Aij)2∣A−B∣=∣A∣−2A⋅B+∣B∣|A|=\sqrt{\sum_{i=1}^{n}\sum_{j=1}^{m}{{(A_{ij})}^2}}\\ |A-B|=|A|-2A\cdot B+|B|\\

matrix

link

∣A−B∣=B to A distance or Similarity between two matricesdistance=∑i=1n∑j=1m∣Aij−Bij∣distance=∑i=1n∑j=1m(Aij−Bij)2|A-B|=\text{B to A distance or Similarity between two matrices}\\ distance=\sum_{i=1}^{n}\sum_{j=1}^{m}{|A_{ij}-B_{ij}}|\\ distance=\sqrt{\sum_{i=1}^{n}\sum_{j=1}^{m}{{(A_{ij}-B_{ij})}^2}}\\

orthogonal vs orthonormal

orthogonal matrix

If column vector are orth to each other(dot product is 0)\text{If column vector are orth to each other(dot product is 0)}\\

orthonormal

If column vector are orth to each other and the length of column vector is 1 then is orthonormalthen is orthogonal matrix[12012]⊥[12120]⊥[01212][121200121212012 ]is orthogonal matrixif A is orthogonal matrix then AT×A=IAT=A−1det(A)2=1\text{If column vector are orth to each other and the length of column vector is 1 then is orthonormal}\\ \text{then is orthogonal matrix}\\ \begin{bmatrix} \frac{1}{\sqrt{2}} \\ 0 \\ \frac{1}{\sqrt{2}} \\ \end{bmatrix} \perp \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \\ 0 \\ \end{bmatrix} \perp \begin{bmatrix} 0 \\ \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \\ \end{bmatrix}\\ \\ \begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 0\\ 0 & \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \ \end{bmatrix}\text{is orthogonal matrix}\\ \text{if A is orthogonal matrix then } \\ A^T \times A=I \\ A^T=A^{-1}\\ det(A)^2=1\\

find orth basis(gram-schmidt process)

derivative of a Matrix

A=[1234]Ax=[1234][x1x2]=[1x1+2x23x1+4x2]f1(x1,x2)=1x1+2x2f2(x1,x2)=3x1+4x2ddxAx=[ddx1f1ddx2f1ddx1f2ddx2f2]=[1234]=A∴ddxAx=AXTAX=[x1x2][a11x1+a12x2a21x1+a22x2]=[a11x12+a12x1x2+a21x1x2+a22x22]f3(x1,x2)=a11x12+a12x1x2+a21x1x2+a22x22ddxXTAX=[ddx1f3ddx2f3]=[2x1+a12x22x2+a21x1]∵z≠xA=\begin{bmatrix} 1 &2\\ 3 &4\\ \end{bmatrix}\\ Ax=\begin{bmatrix} 1 &2\\ 3 &4\\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ \end{bmatrix}= \begin{bmatrix} 1x_1+2x_2 \\ 3x_1+4x_2 \\ \end{bmatrix}\\ f_1(x_1,x_2)=1x_1+2x_2\\ f_2(x_1,x_2)=3x_1+4x_2\\ \frac{d}{dx}Ax= \begin{bmatrix} \frac{d}{dx_1}f_1 &\frac{d}{dx_2}f_1 \\ \frac{d}{dx_1}f_2 &\frac{d}{dx_2}f_2 \\ \end{bmatrix}= \begin{bmatrix} 1&2 \\ 3&4 \\ \end{bmatrix}=A\\ \therefore\frac{d}{dx}Ax=A\\ X^TAX= \begin{bmatrix} x_1 & x_2 \\ \end{bmatrix} \begin{bmatrix} a_{11}x_1+a_{12}x_2 \\ a_{21}x_1+a_{22}x_2 \\ \end{bmatrix}=\begin{bmatrix} a_{11}{x_1}^2+a_{12}x_1x_2+a_{21}x_1x_2+a_{22}{x_2}^2 \\ \end{bmatrix}\\ f_3(x_1,x_2)=a_{11}{x_1}^2+a_{12}x_1x_2+a_{21}x_1x_2+a_{22}{x_2}^2\\ \frac{d}{dx}X^TAX= \begin{bmatrix} \frac{d}{dx_1}f_3 &\frac{d}{dx_2}f_3 \\ \end{bmatrix}= \begin{bmatrix} 2x_1+a_{12}x_2 & 2x_2+a_{21}x_1 \\ \end{bmatrix} \\ \because z \neq x \\

projection

y^=AATAATyATAdefine z=y−y^so{y=y^+zy^⊥zy^∈A columns space\hat{y}= \frac{A }{A^T A} \frac{A^Ty }{A^T A}\\ \text{define } z=y-\hat{y}\\ % z=y(1-\frac{y\cdot A}{y \cdot y})\\ so\begin{cases} y=\hat{y}+z\\ \hat{y}\perp z\\ \hat{y}\in \text{A columns space}\\ \end{cases}

curve fitting | regression | normal equation

(dim⁡β)n=2,(test number)m=4β0+xβ1=y,(x,y) β0+2β1=1,(2,1) β0+5β1=2,(5,2) β0+7β1=3,(7,3) β0+8β1=3,(8,3) A×β=y[12151718]A[m×n]×[β0β1]β[n×1]=[1233]y[m×1]y^=y projection on Az=y−y^z∈nul ATdistance of y to A columns space is z(because z⊥A)(\dim \beta)n=2, \text{(test number)}m=4\\ \beta_0+x\beta_1 =y,\text{(x,y) }\\ \beta_0+2\beta_1 =1,\text{(2,1) }\\ \beta_0+5\beta_1 =2,\text{(5,2) }\\ \beta_0+7\beta_1 =3,\text{(7,3) }\\ \beta_0+8\beta_1 =3,\text{(8,3) }\\ \\ A\times \beta=y\\ \underset{A[m\times n]}{\begin{bmatrix} 1&2 \\ 1&5 \\ 1&7 \\ 1&8 \\ \end{bmatrix}}\times \underset{\beta[n\times 1]}{\begin{bmatrix} \beta_0 \\ \beta_1 \\ \end{bmatrix}}= \underset{y[m\times 1]}{\begin{bmatrix} 1 \\ 2 \\ 3 \\ 3 \\ \end{bmatrix}}\\ \hat{y}=y \text{ projection on }A\\ z=y-\hat{y}\\ z\in \text{nul }A^T\\ \text{distance of y to A columns space is } z(\text{because } z \perp A)\\

way 1

Find β let ∣Aβ−y∣2havemin⁡∣Aβ−y∣2=(∣Aβ∣−2Aβ⋅y+∣y∣)2∣Aβ−y∣2=(∣Aβ∣−2Aβ⋅(y^+z)+∣y^+z∣)2∣Aβ−y∣2=(∣Aβ∣−2Aβ⋅(y^+z)+∣y^+z∣)2\text{Find } \beta \ let \ {|A\beta - y|}^2 \text{have} \min\\ {|A\beta - y|}^2=(|A\beta|-2A\beta\cdot y+|y|)^2\\ {|A\beta - y|}^2=(|A\beta|-2A\beta\cdot (\hat{y}+z)+|\hat{y}+z|)^2\\ {|A\beta - y|}^2=(|A\beta|-2A\beta\cdot (\hat{y}+z)+|\hat{y}+z|)^2\\

way 2

Find β let ∣Aβ−y∣2havemin⁡if derivative f′(a)=0 then f(a) is max or min value ddβ∣y−Aβ∣2=02∣y−Aβ∣×ddβ∣y−Aβ∣=02∣y−Aβ∣×ddβ(∣y∣−2y⋅Aβ+∣Aβ∣)=02∣y−Aβ∣×(A+∣Aβ∣)=0\text{Find } \beta \ let \ {|A\beta - y|}^2 \text{have} \min\\ \text{if derivative }f^{'}(a)=0 \ then \ f(a)\text{ is max or min value }\\ \frac{d}{d\beta}{|y-A\beta|}^2=0\\ 2{|y-A\beta|}\times\frac{d}{d\beta}|y-A\beta|=0\\ 2{|y-A\beta|}\times\frac{d}{d\beta}(|y|-2y\cdot A\beta+|A\beta|)=0\\ 2{|y-A\beta|}\times (A+|A\beta|)=0\\

way 3

Find β let ∣Aβ−y∣2havemin⁡∣ATy−ATAβ∣=AT×distance∣AT(y^+z)−ATAβ∣=AT×distance∣ATy^+ATz−ATAβ∣=AT×distance∣ATy^−ATAβ∣=AT×distancey^∈A columns space so can find Aβ=y^∣ATy^−ATAβ∣=0ATy=ATAββ=(ATA)−1ATy\text{Find } \beta \ let \ {|A\beta - y|}^2 \text{have} \min\\ \begin{aligned} |A^Ty-A^TA\beta|&=A^T\times distance\\ |A^T(\hat{y}+z)-A^TA\beta|&=A^T\times distance\\ |A^T\hat{y}+A^Tz-A^TA\beta|&=A^T\times distance\\ |A^T\hat{y}-A^TA\beta|&=A^T\times distance\\ \end{aligned}\\ \hat{y}\in \text{A columns space so can find }A\beta =\hat{y} \\ |A^T\hat{y}-A^TA\beta|=0\\ A^Ty=A^TA\beta\\ \beta=(A^T A)^{-1}A^T y