A2.0-Vectors.rmd

# Linear Algebra



When formalizing intuitive mathematical ideas, we define a set of *objects* and *rules* for manipulating them—this structure is called an *algebra*. In particular, **linear algebra** focuses on **vectors** and the rules that govern how they can be **added** and **scaled**.  



<p align="center">
<img src="Figure2.1MML.png" alt="A mind map of the concepts introduced in this chapter, along with where they are used in other parts of the book." width="400">
</p>




<div class="definition">
A **vector** is a mathematical object that can be **added** to other vectors and **multiplied by scalars**, resulting in another vector of the same kind.  A vector $v \in  \mathbb{R}^n$ has the form    \[
  \mathbf{v} =
  \begin{bmatrix}
   v_1 \\ v_2 \\ \vdots \\ v_n
   \end{bmatrix},
   \]
where each $v_i \in \mathbb{R}$.   
</div>




---


### Vector Spaces

While most people are familiar with *geometric vectors* (arrows with direction and magnitude), vectors can also take more abstract forms—as long as they obey the two key operations:

1. **Addition:** \( \mathbf{a} + \mathbf{b} = \mathbf{c} \)
2. **Scalar multiplication:** \( \lambda \mathbf{a} = \mathbf{b} \)


<div class="definition">
A set \( V \) is a **vector space** over \( \mathbb{R} \) if for any \( \mathbf{u}, \mathbf{v} \in V \) and any scalar \( \lambda \in \mathbb{R} \):

\[
\mathbf{u} + \mathbf{v} \in V \quad \text{and} \quad \lambda \mathbf{u} \in V
\]
</div>


<div class="definition">
Let \( \mathbf{u}, \mathbf{v} \in \mathbb{R}^n \) be two vectors.  The *sum* of two vectors is obtained by adding their corresponding components:  
   \[
   \mathbf{u} + \mathbf{v} =
   \begin{bmatrix}
   u_1 \\ u_2 \\ \vdots \\ u_n
   \end{bmatrix}
   +
   \begin{bmatrix}
   v_1 \\ v_2 \\ \vdots \\ v_n
   \end{bmatrix}
   =
   \begin{bmatrix}
   u_1 + v_1 \\ u_2 + v_2 \\ \vdots \\ u_n + v_n
   \end{bmatrix}
   \]
</div>




<div class="example">
We can add two vectors componentwise: 
\[
\begin{bmatrix} 1 \\ 4 \\ 10\\ 20 \end{bmatrix} + \begin{bmatrix} 2 \\ 6 \\ 15\\ 30 \end{bmatrix} = \begin{bmatrix} 1+2 \\ 4+6 \\ 10+15\\ 20+30 \end{bmatrix} = \begin{bmatrix} 2 \\ 10 \\ 25\\ 50 \end{bmatrix}.
\]
</div>


<div class="definition">
Let \( \mathbf{u} \in \mathbb{R}^n \) be a vector, and let \( \lambda \in \mathbb{R} \) be a scalar.  The *product* of a scalar \( \lambda \) and a vector \( \mathbf{u} \) is obtained by multiplying each component of the vector by the scalar:  
   \[
   \lambda \mathbf{u} =
   \lambda
   \begin{bmatrix}
   u_1 \\ u_2 \\ \vdots \\ u_n
   \end{bmatrix}
   =
   \begin{bmatrix}
   \lambda u_1 \\ \lambda u_2 \\ \vdots \\ \lambda u_n
   \end{bmatrix}
   \]
</div>




<div class="example">
Scalar multiplication is applied to each term:
\[
5\begin{bmatrix} 1 \\ 4 \\ 10\\ 20 \end{bmatrix} = \begin{bmatrix} 5 \times 1 \\ 5 \times 4 \\ 5 \times 10\\ 5 \times 20 \end{bmatrix} = \begin{bmatrix} 5 \\ 20 \\ 50\\ 100 \end{bmatrix}.
\]
</div>

<div class="example">
The set of complex numbers $\mathbb{C}$ is a vector space.  To prove this, we need to show that it satisfies the two properties:

1.  For $u, v \in \mathbb{C}$, we have $u+v \in \mathbb{C}$.
2.  For $u \in \mathbb{C}$ and $\lambda \in \mathbb{R}$, we have $\lambda u \in \mathbb{C}$.

Let $u,v \in \mathbb{C}$.  Then $u = a+bi$ and $v = c + di$.

1.  **Vector addition:**  The usual complex addition is defined as:
$$
u + v = (a + bi) + (c + di) = (a + c) + (b + d)i \in \mathbb{C}
$$

2.  **Scalar multiplication:** For real scalars \( r \in \mathbb{R} \), scalar multiplication is defined as:
$$
r \cdot (a + bi) = (ra) + (rb)i \in \mathbb{C}.
$$
</div>


Other examples of Vector spaces include:

- **Geometric vectors:** Can be drawn in space and manipulated visually.  
- **Polynomials:** Can be added and scaled to form new polynomials.  
- **Audio signals:** Represented as sequences of numbers that can be combined or scaled.  
- **Tuples of real numbers** in \( \mathbb{R}^n \) 



Treating vectors as elements of \( \mathbb{R}^n \) aligns with how data is represented in computer programs—arrays of real numbers.   This makes linear algebra essential for computational work and for algorithms in machine learning and data science.




---


###  Closure

A central idea in mathematics, known as **closure**, asks what new elements can be formed by combining existing ones through defined operations.  In linear algebra, the set of all possible linear combinations of vectors forms a **vector space**, a foundational concept throughout machine learning.


<div class="definition">
The **closure property** (or simply *closure*) describes whether a set is **closed under an operation** — meaning that when the operation is applied to elements of the set, the result is also an element of the same set.  

Formally, a set \( S \) is **closed** under an operation \( \circ \) if for all \( a, b \in S \):
\[
a \circ b \in S
\]
</div>



<div class="example">
The set of real numbers \( \mathbb{R} \) is **closed under addition** because for any \( a, b \in \mathbb{R} \), the sum \( a + b \in \mathbb{R} \).
</div>

<div class="example">
The set of integers \( \mathbb{Z} \) is **not closed under division**, since \( 1 \div 2 = 0.5 \notin \mathbb{Z} \).
</div>

<div class="example">
The set of vectors of a set length, $n$, is closed since $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$ implies that \[\mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} + \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix} = \begin{bmatrix} a_1+b_1 \\ a_2+b_2 \\ \vdots \\ a_n+b_n \end{bmatrix}.
\]

However, the set of all vectors is not closed since $\begin{bmatrix} 1 \\ 2 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\3 \end{bmatrix}$ is undefined.
</div>




---


###  Other Properties of Vectors


Vectors obey a set of algebraic rules that make them fundamental in both geometry and linear algebra.  

<div class="lemma">
Let \( \mathbf{u}, \mathbf{v}, \mathbf{w} \) be vectors, and let \( c, d \) be scalars.  

1. **Commutativity of Addition**  
   \[
   \mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}
   \]

2. **Associativity of Addition**  
   \[
   (\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})
   \]

3. **Additive Identity**  
   There exists a **zero vector** \( \mathbf{0} \) such that  
   \[
   \mathbf{v} + \mathbf{0} = \mathbf{v}
   \]

4. **Additive Inverse**  
   For every vector \( \mathbf{v} \), there exists a vector \( -\mathbf{v} \) such that  
   \[
   \mathbf{v} + (-\mathbf{v}) = \mathbf{0}
   \]

5. **Distributive Properties**  
   \[
   c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}
   \]
   \[
   (c + d)\mathbf{v} = c\mathbf{v} + d\mathbf{v}
   \]

6. **Associativity of Scalar Multiplication**  
   \[
   c(d\mathbf{v}) = (cd)\mathbf{v}
   \]

7. **Multiplicative Identity**  
   \[
   1 \mathbf{v} = \mathbf{v}
   \]
</div>


Sometimes, we use row vectors rather than column vectors simply to save space or for aesthetic reasons.

<div class="example">
To prove the commutative rule for vector addition, write \(u=[u_1,\dots,u_n]\), \(v=[v_1,\dots,v_n]\). Addition is componentwise:
\[
u + v = [u_1+v_1,\dots,u_n+v_n].
\]
By commutativity in \(\mathbb{R}\),
\[
u_i + v_i = v_i + u_i \quad \forall i,
\]
so
\[
u + v = [v_1+u_1, \dots, v_n+u_n] = v + u.
\]
</div>


The **dot product** (also called the **inner product**) is an operation that takes two vectors and returns a single number. It measures how similar or aligned the two vectors are.


<div class="definition">
For vectors \( \mathbf{a}, \mathbf{b} \in \mathbb{R}^n \), the dot product is defined as:
\[
\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i
\]
</div>

<div class="example">
Compute the dot product of $\begin{bmatrix} 2 \\ 5 \\ 4 \end{bmatrix}$ and $\begin{bmatrix} -3\\ 0 \\ -2 \end{bmatrix}$.
\begin{align*}
\begin{bmatrix} 2 \\ 5 \\ 4 \end{bmatrix} \cdot \begin{bmatrix} -3\\ 0 \\ -2 \end{bmatrix} &= \sum_{i=1}^{3} a_i b_i\\
&= a_1b_1 + a_2 b_2 + a_3 b_3\\
&= 2(-3) + 5(0) + 4(-2)\\
&= -14
\end{align*}
</div>






---


### Geometric Interpretation of a Vector




Geometrically, vectors can be thought of as **arrows** that have both **magnitude** (length) and **direction**. They are often used to represent quantities such as displacement, velocity, or force.  

Vector addition corresponds to placing one arrow’s tail at the head (the *triangle rule*), resulting in a new vector that represents the combined effect of both. 


<p align="center">
<img src="TipToTail2.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>

Scalar multiplication stretches or shrinks a vector and can reverse its direction if the scalar is negative.  


<p align="center">
<img src="ScalarMultiplication2.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>


These geometric operations follow the same algebraic properties found in vector spaces—such as commutativity, associativity, and distributivity — allowing us to interpret abstract vector operations visually as movements and scalings in space.  





<div class="example">
Let $\mathbf{u} = [3,-2]$ and $\mathbf{v} = [-1,4]$.  Then $\mathbf{u} + \mathbf{v}$ and $\mathbf{u} - \mathbf{v}$ can be computed using vectors:
\[\mathbf{u} + \mathbf{v}= [2,2] \;\;\; \text{ and } \;\;\; \mathbf{u} - \mathbf{v} = [4,-6].\]

<p align="center">
<img src="VectorAdditionExample.png" alt="https://courses.lumenlearning.com/ccbcmd-math/chapter/performing-vector-addition-and-scalar-multiplication/" width="400">
</p>
</div>




---


### Exercises {.unnumbered .unlisted}


<div class="exercise">
Vector addition

<div style="text-align: right;">
[Solution]( )
</div> 

</div><div class="exercise">
Scalar Multiplication

<div style="text-align: right;">
[Solution]( )
</div> 
</div>

</div><div class="exercise">
Both

<div style="text-align: right;">
[Solution]( )
</div> 
</div>


<div class="exercise">
Show that each of the following are vector spaces:

-  $\mathbb{R}^n$  
-  Polynomials  
-  Continuous functions  
-  Sequences  

<div style="text-align: right;">
[Solution]( )
</div> 
</div>






<div class="exercise">
Let \( \mathbf{u}, \mathbf{v}, \mathbf{w} \) be vectors, and let \( c, d \) be scalars. Prove each of the following properties: 

1. **Commutativity of Addition**  
   \[
   \mathbf{u} + \mathbf{v} = \mathbf{v} + \mathbf{u}
   \]

2. **Associativity of Addition**  
   \[
   (\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v} + \mathbf{w})
   \]

3. **Additive Identity**  
   There exists a **zero vector** \( \mathbf{0} \) such that  
   \[
   \mathbf{v} + \mathbf{0} = \mathbf{v}
   \]

4. **Additive Inverse**  
   For every vector \( \mathbf{v} \), there exists a vector \( -\mathbf{v} \) such that  
   \[
   \mathbf{v} + (-\mathbf{v}) = \mathbf{0}
   \]

5. **Distributive Properties**  
   \[
   c(\mathbf{u} + \mathbf{v}) = c\mathbf{u} + c\mathbf{v}
   \]
   \[
   (c + d)\mathbf{v} = c\mathbf{v} + d\mathbf{v}
   \]

6. **Associativity of Scalar Multiplication**  
   \[
   c(d\mathbf{v}) = (cd)\mathbf{v}
   \]

7. **Multiplicative Identity**  
   \[
   1 \mathbf{v} = \mathbf{v}
   \]
<div style="text-align: right;">
[Solution]( )
</div> 
</div>



<div class="exercise">
Dot product example


<div style="text-align: right;">
[Solution]( )
</div> 
</div>

<div class="exercise">

<div style="text-align: right;">
[Solution]( )
</div> 
</div>



---