linear-algebra-and-numerical-analysis
Intro to linear algebra
Spans and linear combinations
A linear combination of two vectors (, ) is, where (a,b) are scalars:
The span of two vectors and is the set of all their linear combinations.
- In 2D, two non-collinear vectors typically span the whole plane.
- If two vectors line up (one is a scalar multiple of the other), they span only a line.
Based on span, we have the concept of linear dependence vs independence:
- Linearly dependent: one vector can be expressed as a linear combination of others (redundant).
- Linearly independent: no vector in the set is redundant.
Checking for linear independence
Given two vectors and , you test for linear independence by setting their linear combination equal to the zero vector.
- If the only solution is that both and are , then they are linearly dependent.
- If there is a solution with either and not being the zero vector, then they are linearly dependent.
Matrices as linear transformations
A matrix has its columns as basis vectors, and applies a transformation to vectors by using the matrix's basis vectors as the basis vectors for the vector it wants to transform, mapping that vector into the matrix's coordinate space.
Here is how a matrix translates a vector in standard coordinate space (basis vectors are and ) into a vector .
NOTE
has a solution if is in the span of the columns of .
You can describe a linear transformation by just recording what happens to the and basis vectors of the original coordinate space and how they get transformed.
For example, if a linear transformation scales to , then the matrix representation of the linear transformation is , or
For this example, let's assume and .
- The basis vectors of , let's call them and
- When we transform into the coordinate space of , basically we're going to change the linear combination representation of from using the standard coordinate space basis vectors ( and ) to the basis vectors of :
- But this is now in the "language" of the non-standard basis vectors and , and the only way for us to understand what truly happened to is to translate the basis vectors into the standard coordinate space and then rewrite everything as a linear combination in the standard space:
Matrix multiplication properties
- Not commutative: (AB \ne BA) in general.
- Associative: (A(BC)=(AB)C).
- applied right to left: Applying one transformation after another is composition. If you first apply (B) then apply (A), the combined transformation is (AB), because matrix multiplication is applied right to left.
Change of basis
Let's assume
Determinants and invertible matrices
The geometric meaning of a determinant
For a 2×2 matrix, () is how much areas scale.
- (): areas expand
- (): areas shrink
- (): orientation flips (a “mirror”/reflection component)
But what if the determinant is 0, meaning the basis vectors are collinear with each other? That's where determinants connect to linear independence and dependence and singular matrices:
- : space gets “squished” into a lower dimension (e.g., plane → line).
- linearly dependent: basis vectors become linearly dependent.
- singular: since the columns of the matrix (basis vectors) are linearly dependent, matrix is singular.
- : the parallelogram formed by the basis vectors has non-zero area
- linearly dependent: basis vectors are not collinear, since that is the only way for the parallelogram formed by the basis vectors to have non-zero area
- singular: since the columns of the matrix (basis vectors) are linearly dependent, matrix is singular.
Mathematical properties of determinants
-
multiplicative rule: if a matrix gets transformed via composition by matrix , then the area formed by its basis vectors () gets scaled by the area formed from the basis vectors of , namely .
-
inverse rule
-
additive rule
Invertible and singular matrices
To represent a system of equations, you can use a coefficient matrix, multiply it by a variable vector, and set it equal to a constants vector: .
In plain english:
We are looking for a vector that when transformed by , lands on .
To solve this system of equations, we can use inverse matrices: Inverse matrices are the inverse transformation of a matrix.
By applying a transformation, then subsequently applying the inverse of that transformation, you end up back where you started: .
- If a matrix rotates space clockwise by 90 degrees, an inverse matrix of that matrix would rotate space counterclockwise 90 degrees.
- doing a transformation then doing the transformation is the same thing as doing nothing.
By using the inverse, you can find the variable vector like so
A matrix is invertible/nonsingular if it has an inverse, else we call it singular if it does not have an inverse.
Singular matrices are noninvertible (have det = 0) while nonsingular matrices are invertible. In terms of G.E, this means the following:
- singular matrices: infinite solutions or no solutions. One row is just full zeros when reduced via G.E, so there is at least one free variable.
- nonsingular: Has one solution for , meaning there is a pivot in each row and along the diagonals in its correct spot.
when matrices don’t have an inverse
If the determinant of matrix is 0, then it does not have an inverse because it squishes all of space onto one line irreversibly. The area of the parallelogram formed by the basis vectors is 0. You cannot un-squish a line.
Also, a matrix transformation being injective and surjective plays a role in if its invertible or not.
- injective: A linear map is injective iff the only vector mapped to (0) is the zero vector.
- Basically, no transformed vector besides the zero vector should transform to the zero vector, otherwise the linear transformation is not injective.
- surjective: Once transformed, the new basis matrix has the same span as the original basis matrix. Basically, the transformation matrix must be full rank and a square matrix.
Transformation matrices will only have inverses if they are bijective - meaning both injective and surjective.
So in summary, here are the three cases where a transformation matrix will not have an inverse:
- : transformation matrix is not injective, because some vectors will get squished to the zero vector as a result of this transformation.
- nonsquare matrices: Not surjective, since their column space is not full rank.
- matrices that transform some vectors to the : These matrices are not injective by definition.
Rank, column space, null space
- Rank: number of dimensions in the output / the dimension of the column space.
- Column space: the span of the columns of ().
- null space: The null space is all vectors mapped to zero, which can be found through this equation:
rank
rank means the number of dimensions in the output of the transformation, and is used to differentiate between transformations that either take up all the possible dimensions or have a zero-determinant and thus squish all vectors into a lower dimension.
- When the output of a transformation is a line, the rank is 1.
- When the output of a transformation is a plane, the rank is 2.
- When the output of a transformation is all of space, the rank is 3.
- Full rank means that the number of dimensions matches the number of columns.
- example: full rank for an space is 4.
IMPORTANT
Matrices are full rank iff they are injective, meaning its null space only has the trivial solution .
column space
Column space is the span of the columns of the matrix, with the max column space for a matrix being .
- when a matrix is singular based on column space: If your column space dimensions are less than the dimensions of the matrix, meaning less dimensional than , then the matrix is singular.
null space
The null space is the set of all possible vectors that become null, or after a linear transformation.
Geometrically, it's the set of all vectors that land on the origin after a linear transformation, and has two types of solutions:
- trivial solution: the only vector that maps to the origin is the origin itself, , since by nature of a linear transformation the origin (zero vector) always stays fixed.
- nontrivial solution: there is a non-empty set of non-zero vectors called the kernel that gets transformed into the .
You can find the null space by solving a system of equations and setting the post-transformed vector to the .
NOTE
The kernel is the set of all solutions to the null space equation, and a trivial solution leads to a trivial kernel, and a nontrivial solution leads to a nontrivial kernel.
Cramer's rule
Cramer's rule comes into play when we have an equation , where and are known, and we want to find out what is.
Intuition
Consider that for the standard identity matrix with basis vectors and , we can represent the coordinates of a vector as two areas.
Let's form the following notation:
- : the area of the parallelogram formed by and
- : the area of the parallelogram formed by and
Let's dive into the calculations
- : the area of the parallelogram formed by and turns out to be .

- : the area of the parallelogram formed by and turns out to be .

Now that we found what and are in terms of basis vectors, we can swap those standard basis vectors out for any matrix, and understand that linear transformations will scale and stretch vectors evenly:

- , but after transformation from A, it becomes , and now find the signed area using the output vector , since , thus we deal with instead of in the transformed coordinate space of and luckily since we know , thus we arrive at this new formula:
