From Columns to Solutions

Suppose we are given the matrix

As discussed previously, this matrix consists of three column vectors in ℝ³. A natural question to ask is whether these three vectors are able to span the entire three-dimensional space. Equivalently, we may ask:

Does the equation Ax=b has a solution for every possible right-hand side vector b ∈ ℝ³?

Let’s see some possible solutions:

One obvious solution is [ 0  0  0 ]^T. in fact, zero is the obvious solution of every Ax=b problem. But it is still important. Another solution that come to our mind is [ 1  2  3 ]^T. we reached this possible answer by assuming x=[ 1  0  0 ]^T.

In the same way, [ 1  0  1 ]^T can also be a solution for Ax=b. So, we say:

For this particular matrix, it turns out that the third column is a linear combination of the first two columns:

(Column3) = (column1) + 2(column2)

Therefore, the third column does not introduce a new direction.

Geometrically, all three columns lie in the same plane in ℝ³. As a result, the linear combinations of the columns can reach only the vectors lying in that plane, not the entire three-dimensional space. This is equivalent to say that Ax=b has a solution only for those vectors b that lie in this plane. We call this plane the column space of the matrix A.

In general, the column space of a matrix is defined as the set of all linear combinations of its columns. It is always a subspace of ℝ^m (where m is the number of rows of the matrix), and therefore it always contains the zero vector.

Notice that removing the third column of A does not change its column space, since that column is already generated by the first two. What happens if we remove that column?

Let’s do this and call it matrix B:

Again, since there are only two columns in ℝ³, their linear combinations only form a plane, not the entire space. Therefore, the column space of B is also a two-dimensional subspace of ℝ³.

As we engineers are interested in those b that are the solution of Ax=b, it is natural to ask if the column-space of A and B are the same, can they be considered equivalent? What is the difference between matrix A and B? Let’s solve the system of equations for both of them.

Case (a)

This is a system of three equations with two unknowns. In general, such systems do not have a solution.

However, in this case, the vector b lies in the column space of the coefficient matrix, so a solution exists. Equivalently we can say, one of the three equations is a linear combination of the other two, meaning the system is not over-constrained.

Performing row operations, we obtain:

This is the only solution (unique solution).

Again, note that if b was b=[ 0  2  3 ]^T, then the system had no solution.

Case (b)

 it’s a system of 3 equations and 3 unknowns. But as we previously examined, the third column is a combination of the first two columns and thus it is not independent.

However, since the vector b lies in the column space, the system does have solutions. And as we will see, there is more than one solution.

From what we previously saw in part (a), we know that x=[ 1  -1  0 ]^T is a solution to this system. Let’s call this particular solution, x_p.

But since column3 is also a combination of column1 and column2, so there are infinite linear combinations of columns that can lead to b. The question is how to compute all solutions?

Let's first find the solutions that satisfy Ax=0. for convenience, we denote these solutions with x_n.

now we claim all the solutions of the system Ax=b is x_p+x_n. because

to find x_n we actually should find the proportions that columns cancel out each other.

as column3 is a linear combination of the first two column, let's try different values for x₃.

Geometrically, x_n is the equation of a line in 3D space which passes through the zero point. Therefore, the complete solution, x_c, to the system would be

This is the equation of a line in 3D vector space which is parallel with x_n, but does not pass through the origin.

We call x_n, the null space of the coefficient matrix. just like the column-space, it is also an intrinsic property of the coefficient matrix. Both null-space and column-space play a central role in understanding linear systems in engineering applications.

• The column space of a matrix tells us which right-hand sides b lead to solvable systems.

• The null space describes all the combination of columns that lead to Ax=0. So, it tells us whether possible solutions are unique and describes the freedom in the solution.