Linear Algebra

1. What is Linear Algebra?

Linear algebra is the branch of mathematics that deals with vectors (ordered lists of numbers), matrices (grids of numbers), and linear transformations (functions that move, stretch, and rotate space). If regular algebra is about solving for one unknown number, linear algebra is about solving for many unknowns at once -- and understanding the structure of the space they live in.

Think of it this way: a single variable x is just a number on a number line. But in the real world, data almost never comes as a single number. An image is a grid of pixel values. A point in 3D space has three coordinates. A neural network has millions of parameters. Linear algebra gives you the tools to work with all of that at once, efficiently and elegantly.

Why CS Needs Linear Algebra

2. Vectors

A vector is an ordered list of numbers. That's the whole definition. In 2D, a vector has two components. In 3D, three. In machine learning, vectors can have thousands or millions of components.

Geometrically, you can think of a 2D or 3D vector as an arrow pointing from the origin to a point. The vector [3, 2] points 3 units right and 2 units up. But vectors don't have to be spatial -- they can represent anything: colors, audio samples, user preferences, word meanings.

Vector Addition and Subtraction

To add or subtract vectors, you just add or subtract their corresponding components. Both vectors must have the same number of components.

Scalar Multiplication

A scalar is just a regular number (as opposed to a vector). Multiplying a vector by a scalar multiplies every component by that number.

Magnitude (Length)

The magnitude (or length or norm) of a vector tells you how long it is. It's the distance from the origin to the point the vector represents.

Unit Vectors

A unit vector is a vector with magnitude 1. It represents a pure direction with no scaling. To make any vector into a unit vector, divide it by its magnitude.

Dot Product

The dot product (or scalar product) takes two vectors and returns a single number. You multiply corresponding components and add them up.

Geometric Meaning of Dot Product

The dot product tells you how much two vectors point in the same direction. It connects to the angle between them through this formula:

• If dot product > 0: vectors point roughly the same direction (angle < 90 degrees)
• If dot product = 0: vectors are perpendicular (angle = 90 degrees)
• If dot product < 0: vectors point in roughly opposite directions (angle > 90 degrees)

Cross Product (3D Only)

The cross product takes two 3D vectors and returns a new vector that is perpendicular to both inputs. It's mainly used in 3D graphics for computing surface normals.

3. Matrices

A matrix is a rectangular grid of numbers arranged in rows and columns. Just as a vector is a list, a matrix is a table. We describe a matrix by its dimensions: an m × n matrix has m rows and n columns.

Matrices show up everywhere in CS: an image is a matrix of pixel values, a spreadsheet is a matrix, a neural network layer is defined by a weight matrix, and a 3D transformation is a 4×4 matrix.

Matrix Addition and Subtraction

Just like vectors, you add/subtract matrices element by element. Both matrices must have the same dimensions.

Scalar Multiplication

Multiply every element in the matrix by the scalar.

Matrix Multiplication

This is the big one. Matrix multiplication is not element-wise. It uses a "row times column" pattern. Each entry in the result is the dot product of a row from the first matrix with a column from the second matrix.

Identity Matrix

The identity matrix (I) is the matrix equivalent of the number 1. It's a square matrix with 1s on the diagonal and 0s everywhere else. Multiplying any matrix by I leaves it unchanged.

Transpose

The transpose of a matrix (written A^T) flips rows and columns. Row 1 becomes column 1, row 2 becomes column 2, and so on. An m×n matrix becomes n×m.

4. Matrix Operations for CS

Systems of Equations as Ax = b

One of the most powerful uses of matrices is representing and solving systems of linear equations. Instead of writing out multiple equations, you pack everything into a single matrix equation.

Gaussian Elimination (Row Reduction)

Gaussian elimination is a systematic method for solving systems of equations by transforming the matrix into a simpler form. You create an augmented matrix (A with b appended) and perform row operations until the solution is clear.

The three allowed row operations are:

1. Swap two rows
2. Multiply a row by a nonzero constant
3. Add a multiple of one row to another row

Row Echelon Form

The goal of Gaussian elimination is to reach row echelon form, where:

• All zero rows are at the bottom
• The first nonzero entry in each row (called the pivot) is to the right of the pivot in the row above
• The matrix forms a "staircase" pattern

5. Determinant

The determinant is a single number computed from a square matrix that tells you important things about the matrix. Think of it as a measure of how much the matrix "stretches" or "squishes" space.

2×2 Determinant

What the Determinant Means

Geometrically, the determinant represents the scaling factor for area (2D) or volume (3D) when you apply the matrix as a transformation.

• |det| > 1: The transformation expands area/volume
• |det| = 1: Area/volume is preserved (like rotation)
• 0 < |det| < 1: The transformation shrinks area/volume
• det = 0: The transformation collapses a dimension (the matrix is singular)
• det < 0: The transformation flips orientation (like a reflection)

3×3 Determinant

For larger matrices, you expand along a row or column (cofactor expansion). For a 3×3 matrix:

6. Inverse Matrix

The inverse of a matrix A, written A^-1, is the matrix that "undoes" A. When you multiply A by its inverse, you get the identity matrix.

Think of it like division for matrices. If multiplication by A transforms space in some way, multiplication by A^-1 reverses that transformation exactly.

When Does the Inverse Exist?

A matrix has an inverse if and only if its determinant is not zero. If det(A) = 0, the matrix is called singular and has no inverse. This makes intuitive sense: if A collapses a dimension (det = 0), you can't recover the lost information.

2×2 Inverse Formula

Using the Inverse to Solve Systems

If you have the equation Ax = b and you know A^-1, you can solve for x directly:

7. Eigenvalues and Eigenvectors

This is the concept that makes most students panic, but the idea is surprisingly simple. When you multiply most vectors by a matrix, they change both direction and magnitude. But some special vectors only get scaled -- they keep pointing in the same direction. These are eigenvectors, and the scaling factor is the eigenvalue.

In words: "When I apply transformation A to vector v, I get back the same vector v, just scaled by λ."

How to Find Eigenvalues

Starting from Av = λv, we rearrange:

Why CS Cares About Eigenvalues

8. Linear Transformations

Every matrix represents a linear transformation -- a function that takes vectors in, moves/stretches/rotates them, and outputs new vectors. When you multiply a vector by a matrix, you're applying that transformation. This is the connection between abstract matrix math and real visual/spatial effects.

Common 2D Transformation Matrices

Composition = Matrix Multiplication

Here's the elegant part: applying two transformations in sequence is the same as multiplying their matrices together. If you want to first scale, then rotate, you compute R * S and use the resulting matrix.

9. Practical CS Applications

Neural Networks

A neural network layer is fundamentally a matrix multiplication followed by an activation function. The weights between layers form a matrix. The forward pass (computing the output) is just repeated matrix-vector multiplication.

Training adjusts the weights using gradients (vectors of partial derivatives), and the gradient computation involves matrix transposes and chain-rule multiplications. The entire deep learning stack is linear algebra plus calculus.

Image Processing

A grayscale image is literally a matrix where each entry is a pixel brightness (0-255). Color images are three matrices stacked (R, G, B channels). Convolution -- the core operation in image filters and CNNs -- slides a small matrix (the kernel) across the image and computes dot products at each position.

Recommendation Systems

Netflix, Spotify, and Amazon use matrix factorization for recommendations. You have a giant matrix of users × items, mostly empty (users have only rated a few things). The trick: decompose this into two smaller matrices -- one capturing user preferences, one capturing item features. The product approximates the full matrix, filling in the blanks with predictions.

3D Graphics Pipeline

Every vertex in a 3D scene goes through a chain of 4×4 matrix transformations. Using 4×4 matrices (instead of 3×3) through a technique called homogeneous coordinates lets you represent translation as a matrix multiplication too.

Python / NumPy Code Example

Pythonimport numpy as np

# --- Vectors ---
a = np.array([2, 3, -1])
b = np.array([4, -1, 5])

print("Addition:", a + b)          # [6, 2, 4]
print("Dot product:", np.dot(a, b)) # 0 (perpendicular!)
print("Magnitude:", np.linalg.norm(a)) # 3.742

# --- Matrices ---
A = np.array([[1, 2],
              [3, 4]])
B = np.array([[5, 6],
              [7, 8]])

print("Matrix multiply:", A @ B)  # @ is matrix multiply
print("Transpose:", A.T)
print("Determinant:", np.linalg.det(A))  # -2.0
print("Inverse:", np.linalg.inv(A))

# --- Solving Ax = b ---
A = np.array([[2, 3],
              [4, -1]])
b = np.array([7, 1])

x = np.linalg.solve(A, b)  # Better than inv(A) @ b
print("Solution:", x)  # [1. 1.667]

# --- Eigenvalues ---
A = np.array([[4, 1],
              [2, 3]])
eigenvalues, eigenvectors = np.linalg.eig(A)
print("Eigenvalues:", eigenvalues)    # [5. 2.]
print("Eigenvectors:", eigenvectors)  # columns are eigenvectors

# --- 2D Rotation ---
import math
theta = math.radians(90)
R = np.array([[math.cos(theta), -math.sin(theta)],
              [math.sin(theta),  math.cos(theta)]])
point = np.array([3, 1])
print("Rotated:", R @ point)  # [-1, 3]

10. Practice Quiz

Test your understanding of the key linear algebra concepts. Click the answer you think is correct.