Linear Algebra and ML: 13 Key Concepts, Codes, and Applications

A two-part overview details 13 fundamental concepts of linear algebra useful for machine learning practitioners. Determinants, inverses, subspaces, ranks, and row reductions are presented with NumPy examples and direct links to PCA, attention, or optimization.

Transforming Space, Measuring Effect, and Inverting

Matrices are described as linear transformations that preserve vector addition and scalar multiplication. They play a role in feature transformations, dimensionality reduction, linear models, and data geometry. Typical geometric effects are shown, including scaling, a 90-degree rotation, reflection across the x-axis, and horizontal shearing. The determinant of the matrix [[2, 0], [0, 3]] is 6.0, and for a unit square, the transformed area equals the absolute value of this determinant, which is also 6.0. The inverse of this matrix is calculated using numpy.linalg.inv, and the product A @ A_inv is displayed for verification. The pseudo-inverse of a 3x2 matrix is used to solve a system in the least squares sense, with weights obtained via X_pinv @ y, followed by predictions y_pred calculated as X @ w.

How Data is Represented and Manipulated

Training data is described as vectors and matrices, with grayscale images stored in 2D matrices and color images in 3D matrices. Neural networks perform matrix multiplications at each forward pass, akin to linear transformations. Tensors, generalizations of scalars, vectors, and matrices to three dimensions and beyond, are the central structure of TensorFlow and PyTorch frameworks, which leverage automatic differentiation on GPUs. Matrices are two-dimensional arrays of numbers, vectors are one-dimensional arrays with magnitude and direction, while scalars, zero-dimensional entities, also describe metrics such as losses, means, and variances.

Basic Operations on Vectors and Matrices

The presented vector operations include addition, subtraction, scalar multiplication, dot product, and cross product. Vectors are used for rotation, scaling, and translation, and to quantify similarity through Euclidean distance or cosine similarity. On the matrix side, examples cover addition, subtraction, element-wise division, matrix multiplication, matrix-scalar multiplications, and matrix-vector products. Matrices are noted as ubiquitous in machine learning, graphics, physics, optimization, and in Google's PageRank algorithm.

Combinations, Subspaces, Independence, and Rank

Linear combinations are among the concepts to master. The linear span of a set of vectors corresponds to all vectors formed by these combinations. A subspace is defined as a subset containing the zero vector, closed under addition and scalar multiplication. The rank of a matrix, equal to the maximum number of linearly independent rows or columns, provides information about unique information, independent directions, and relevant components. A rank lower than the number of features comes with risks of multicollinearity and unstable linear regressions. PCA is presented as automatically reducing dimension to rank and seeking the subspace where data concentrates, while an encoder compresses data into a latent subspace.

Solving Systems and Row Reduction

The resolution of linear systems is illustrated with numpy.linalg.solve on a 2x2 matrix and a vector b. Row reduction transforms a matrix into a simpler form while preserving the entire set of solutions, through three elementary operations: swapping rows, multiplying a row by a non-zero constant, and adding or subtracting multiples of other rows. Each pivot fixes a variable and gives a stair-step form. A row [0 0 0 | 1] indicates an inconsistent system, while [0 0 0 | 0] indicates a redundant equation. This method addresses the existence, uniqueness, or infinity of solutions, the number of independent equations, and the rank, particularly via Gaussian elimination.

Explicit Links to ML Methods and Reading Papers

Several direct links to machine learning methods are recalled: PCA relies on eigenvectors and eigenvalues, SVD is a matrix factorization, and attention exploits the dot product. Neural networks are presented as matrix multiplication machines, and optimization involves gradients and Hessians with techniques like gradient descent, Lagrange multipliers, and Kuhn–Tucker conditions. Mastery of mathematical notation facilitates efficient reading of research papers. The four mathematical pillars highlighted are linear algebra, calculus, probability, and statistics. The author asserts that solid foundations are sufficient without resorting to highly advanced doctoral-level mathematics and that these foundations help break down complex derivations; linear algebra is described as the backbone of the field. This overview is structured in two parts covering 26 concepts, with 13 notions in this first part, each concept accompanied by intuitions, explanations, definitions, and applications. The post is attributed to Rajendran S, originally published on a platform dedicated to artificial intelligence, dated and updated on August 24, 2026.