Fields

A field is a set $F$ equipped with two operations, addition and mulitplication, and containing two special members 0 and 1 ($0\neq 1$), such that for all $\{a, b, c\}\in F$

1. 1. $(a+b)\in F$
   2. $a+b=b+a$
   3. $(a+b)+c=a+(b+c)$
   4. $a+0=a$
   5. there exists $-a$ such that $a+-a=0$
2. 1. $ab\in F$
   2. $ab=ba$
   3. $abc=abc$
   4. $a·1=a$
   5. there exists $a^{(-1)}$ such that $aa^{(-1)}=1$
3. $a(b+c)=ab+ac$

1. $F$ is an abelian group under addition
2. $F$ is an abelian group under multiplication
3. multiplication distributes over addition

Vector Spaces

Let $F$ be a field, and $V$ a set. We say $V$ is a vector space over $F$ if there exist two operations, defined for all $a\in F$, $u\in V$ and $v\in V$:

• vector addition: ($u$, $v$) → $(u+v)\in V$
• scalar multiplication: ($a$,$v$) → $av\in V$

1. 1. $u+(v+w)=(u+v)+w$
   2. $u+v=v+u$
   3. $u+0=u$
   4. there exists $-u$ such that $u+-u=0$
2. 1. $a(u+v)=au+av$
   2. $(a+b)u=au+bu$
   3. $abu=abu$
   4. $1·u=u$

1. $V$ is an abelian group under plus
2. Natural properties of scalar multiplication

Euclidean Space

Throughout this course we will think of a signal as a vector $$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_N\end{array}\right)=\begin{pmatrix}{x}_1 & x_2 & \dots & x_N\end{pmatrix}^T$$ The samples $\{x_i\}$ could be samples from a finite duration, continuous time signal, for example.

A signal will belong to one of two vector spaces:

Subspaces

Let $V$ be a vector space over $F$.

A subset $S\subseteq V$ is called a subspace of $V$ if $S$ is a vector space over $F$ in its own right.

$V=\mathbb{R}^2$, $F=\mathbb{R}$, $S=\text{any line though the origin}$.

Are there other subspaces?

Linear Independence

Let $u_1,\dots ,u_k\in V$.

We say that these vectors are linearly dependent if there exist scalars $a_1,\dots ,a_k\in F$ such that

If [link] only holds for the case $a_1=\dots =a_k=0$, we say that the vectors are linearly independent.

Spanning Sets

Consider the subset $S=\{v_1, v_2, \dots , v_k\}$. Define the span of $S$ $$<S>\equiv \mathrm{span}(S)\equiv \{\sum_{i=1}^k a_i{v}_i\colon a_i\in F\}$$

Fact: $<S>$ is a subspace of $V$.

$V=\mathbb{R}^3$, $F=\mathbb{R}$, $S=\{v_1, v_2\}$, $v_1=\left(\begin{array}{c}1\\ 0\\ 0\end{array}\right)$, $v_2=\left(\begin{array}{c}0\\ 1\\ 0\end{array}\right)$ ⇒ $<S>=\text{xy-plane}$.

Bases

A set $B\subseteq V$ is called a basis for $V$ over $F$ if and only if

1. $B$ is linearly independent
2. $<B>=V$

Dimension

Let $V$ be a vector space with basis $B$. The dimension of $V$, denoted $\dim (V)$, is the cardinality of $B$.

$\implies \dim (V)$ is well-defined.

If $\dim (V)< \infty$, we say $V$ is finite dimensional.

Facts
• If $S$ is a subspace of $V$, then $\dim (S)\le \dim (V)$.
• If $\dim (S)=\dim (V)< \infty$, then $S=V$.

Direct Sums

Let $V$ be a vector space, and let $S\subseteq V$ and $T\subseteq V$ be subspaces.

We say $V$ is the direct sum of $S$ and $T$, written $V=\oplus (S, T)$, if and only if for every $v\in V$, there exist unique $s\in S$ and $t\in T$ such that $v=s+t$.

If $V=\oplus (S, T)$, then $T$ is called a complement of $S$.

Norms

Let $V$ be a vector space over $F$. A norm is a mapping $\to (V, F)$, denoted by $(·)$, such that forall $u\in V$, $v\in V$, and $\lambda \in F$

1. $(u)> 0$ if $u\neq 0$
2. $(\lambda u)=\left|\lambda \right|(u)$
3. $(u+v)\le (u)+(v)$

Inner products

Let $V$ be a vector space over $F$, $F=\mathbb{R}$ or $\mathbb{C}$. An inner product is a mapping $V\times V\to F$, denoted $·\cdot ·$, such that

1. $v\cdot v\ge 0$, and $\iff (v\cdot v=0, v=0)$
2. $u\cdot v=\overline{v\cdot u}$
3. $au+bv\cdot w=a(u\cdot w)+b(v\cdot w)$

Triangle Inequality

If we define $(u)=u\cdot u$, then $$(u+v)\le (u)+(v)$$Hence, every inner product induces a norm.

Cauchy-Schwarz Inequality

For all $u\in V$, $v\in V$, $$\left|u\cdot v\right|\le (u)(v)$$ In inner product spaces, we have a notion of the angle between two vectors: $$(\angle (u, v)=\arccos \left(\frac{u\cdot v}{(u)(v)}\right))\in \left[0 , 2\pi \right)$$

Orthogonality

$u$ and $v$ are orthogonal if $$u\cdot v=0$$Notation: $\perp (u, v)$.

If in addition $(u)=(v)=1$, we say $u$ and $v$ are orthonormal.

In an orthogonal (orthonormal) set, each pair of vectors is orthogonal (orthonormal).

Orthonormal Bases

An Orthonormal basis is a basis $\{v_i\}$ such that $$v_i\cdot v_i={\delta }_{ij}=\begin{cases}1 & \text{if $i=j$}\\ 0 & \text{if $i\neq j$}\end{cases}$$

Expansion Coefficients

If the representation of $v$ with respect to $\{v_i\}$ is $$v=\sum a_i{v}_i$$ then $$a_i=v_i\cdot v$$

Gram-Schmidt

Every inner product space has an orthonormal basis. Any (countable) basis can be made orthogonal by the Gram-Schmidt orthogonalization process.

Orthogonal Compliments

Let $S\subseteq V$ be a subspace. The orthogonal compliment $S$ is $$S^{\perp }=\{u\colon u\in V\land (u\cdot v=0)\land \forall v\colon v\in S\}$$ $S^{\perp }$ is easily seen to be a subspace.

If $\dim (v)< \infty$, then $V=\oplus (S, S^{\perp })$. AsideIf dim v, then in order to have V ⊕ S S ⊥ we require V to be a Hilbert Space.

Linear Transformations

Loosely speaking, a linear transformation is a mapping from one vector space to another that preserves vector space operations.

More precisely, let $V$, $W$ be vector spaces over the same field $F$. A linear transformation is a mapping $T:V\to W$ such that $$T(au+bv)=aT(u)+bT(v)$$ for all $a\in F$, $b\in F$ and $u\in V$, $v\in V$.

In this class we will be concerned with linear transformations between (real or complex) Euclidean spaces, or subspaces thereof.

Image

$$\mathop{\mathrm{image}}(T)=\{w\colon w\in W\land T(v)=w\text{for some}v\}$$

Nullspace

Also known as the kernel: $$\ker (T)=\{v\colon v\in V\land (T(v)=0)\}$$

Rank

$$\mathrm{rank}(T)=\dim (\mathop{\mathrm{image}}(T))$$

Nullity

$$\mathrm{null}(T)=\dim (\ker (T))$$

Rank plus nullity theorem

$$\mathrm{rank}(T)+\mathrm{null}(T)=\dim (V)$$

Matrices

Every linear transformation $T$ has a matrix representation. If $T:{𝔼}^N\to {𝔼}^M$, $𝔼=\mathbb{R}$ or $\mathbb{C}$, then $T$ is represented by an $M\times N$ matrix $$A=\begin{pmatrix}{a}_{11} & \dots & a_{1N}\\ ⋮ & \ddots & ⋮\\ a_{M1} & \dots & a_{MN}\end{pmatrix}$$ where $\left(\begin{array}{c}{a}_{1i}\\ \dots \\ a_{Mi}\end{array}\right)=T(e_i)$ and $e_i=\left(\begin{array}{c}0\\ \dots \\ 1\\ \dots \\ 0\end{array}\right)$ is the $i^{\mathrm{th}}$ standard basis vector. AsideA linear transformation can be represented with respect to any bases of 𝔼 N and 𝔼 M, leading to a different A. We will always represent a linear transformation using the standard bases.

Column span

$$\mathrm{colspan}(A)=<A>=\mathop{\mathrm{image}}(A)$$

Duality

If $A:{\mathbb{R}}^N\to {\mathbb{R}}^M$, then $$\ker (A)^{\perp }=\mathop{\mathrm{image}}(A^T)$$

If $A:{\mathbb{C}}^N\to {\mathbb{C}}^M$, then $$\ker (A)^{\perp }=\mathop{\mathrm{image}}((A))$$

Inverses

The linear transformation/matrix $A$ is invertible if and only if there exists a matrix $B$ such that $AB=BA=I$ (identity).

Only square matrices can be invertible.

Let $A:{𝔽}^N\to {𝔽}^N$ be linear, $𝔽=\mathbb{R}$ or $\mathbb{C}$. The following are equivalent:

1. $A$ is invertible (nonsingular)
2. $\mathrm{rank}(A)=N$
3. $\mathrm{null}(A)=0$
4. $\det A\neq 0$
5. The columns of $A$ form a basis.