Linear Spaces

Definition and basic properties

A linear space over a field \mathbb{F} is a set \mathcal{X}, endorsed with the operations of vector addition+” and scalar multiplication\cdot” which are required to fullfill certain properties, usually called axioms. To emphasize the importance of all ingredients, vector spaces are often written as tuples (\mathcal{X}, \mathbb{F}, +, \cdot). We always assume that \mathbb{F} = \mathbb{R} or \mathbb{C}.

In the following, we list the axioms, which are required to hold for arbitrary x, y, z \in \mathcal{X} and a, b \in \mathbb{F}.

Associativity of addition

(x + y) + z = (x + y) + z

Commutativity of addition

x + y = y + x

Existence of a neutral element of addition

0 + x = x + 0 = x

Existence of inverse elements of addition

\forall x\ \exists \bar x: \bar x + x = x + \bar x = 0

Compatibility of multiplications

a \cdot (b \cdot x) = (ab) \cdot x

Neutral scalar is the neutral element of scalar multiplication

1 \cdot x = x

Distributivity with respect to vector addition

a \cdot (x + y) = a \cdot x + a \cdot y

Distributivity with respect to scalar addition

(a + b) \cdot x = a \cdot x + b \cdot x

Of course, the inverse element \bar x is usually denoted with -x.

Metric spaces

The vector space (\mathcal{X}, \mathbb{F}, +, \cdot) is called a metric space if it is additionally endorsed with a distance function or metric

d: \mathcal{X} \times \mathcal{X} \to [0, \infty)

with the following properties for all x, y, z \in \mathcal{X}:

\begin{align*}
  & d(x, y) = 0 \quad \Leftrightarrow \quad x = y && \text{(identity of indiscernibles)} \\
  & d(x, y) = d(y, x)  && \text{(symmetry)} \\
  & d(x, y) \leq d(x, z) + d(z, y) && \text{(subadditivity)}
\end{align*}

We call the tuple (\mathcal{X}, \mathbb{F}, +, \cdot, d) a Metric space.

Normed spaces

A function on \mathcal{X} intended to measure lengths of vectors is called a norm

\lVert \cdot \rVert : \mathcal{X} \to [0, \infty)

if it fulfills the following conditions for all x, y \in \mathcal{X} and a \in \mathbb{F}:

\begin{align*}
  & \lVert x \rVert = 0 \Leftrightarrow x = 0 && \text{(positive definiteness)} \\
  & \lVert a \cdot x \rVert = \lvert a \rvert\, \lVert x \rVert && \text{(positive homegeneity)}
  \\
  & \lVert x + y \rVert \leq \lVert x \rVert + \lVert x \rVert && \text{(triangle inequality)}
\end{align*}

A tuple (\mathcal{X}, \mathbb{F}, +, \cdot, \lVert \cdot \rVert) fulfilling these conditions is called Normed vector space. Note that a norm induces a natural metric via d(x, y) = \lVert x - y \rVert.

Inner product spaces

Measure angles and defining notions like orthogonality requires the existence of an inner product

\langle \cdot, \cdot \rangle : \mathcal{X} \times \mathcal{X} \to \mathbb{F}

with the following properties for all x, y, z \in \mathcal{X} and a \in \mathbb{F}:

\begin{align*}
  & \langle x, x \rangle \geq 0 \quad \text{and} \quad \langle x, x \rangle = 0 \Leftrightarrow
  x = 0 && \text{(positive definiteness)} \\
  & \langle a \cdot x + y, z \rangle = a \, \langle x, z \rangle + a \, \langle y, z \rangle &&
  \text{(linearity in the first argument)} \\
  & \langle x, y \rangle = \overline{\langle x, y \rangle} && \text{(conjugate symmetry)}
\end{align*}

The tuple (\mathcal{X}, \mathbb{F}, +, \cdot, \langle \cdot \rangle) is then called an Inner product space. Note that the inner product induces the norm \lVert x \rVert = \sqrt{\langle x, x \rangle}.

Cartesian spaces

We refer to the space \mathbb{F}^n as the n-dimensional Cartesian space over the field \mathbb{F}. We choose this notion since Euclidean spaces are usually associated with the Euclidean norm and distance, which are just (important) special cases. Vector addition and scalar multiplication in \mathbb{F}^n are, of course, realized with entry-wise addition and scalar multiplication.

The natural inner product in \mathbb{F}^n is defined as

\langle x, y \rangle_{\mathbb{F}^n} := \sum_{i=1}^n x_i\, \overline{y_i}

and reduces to the well-known dot product if \mathbb{F} = \mathbb{R}. For the norm, the most common choices are from the family of p-norms

\lVert x \rVert_p &:= \left( \sum_{i=1}^n \lvert x_i \rvert^p \right)^{\frac{1}{p}}
\quad \text{if } p \in [1, \infty) \\[1ex]
\lVert x \rVert_\infty &:= \max\big\{\lvert x_i \rvert\,|\, i \in \{1, \dots, n\} \big\}

with the standard Euclidan norm for p = 2. As metric, one usually takes the norm-induced distance function, although other choices are possible.

Weighted Cartesian spaces

In the standard definition of inner products, norms and distances, all components of a vector are have the same weight. This can be changed by using weighted versions of those functions as described in the following.

Let A \in \mathbb{F}^{n \times n} be a Hermitian square and positive definite matrix, in short A = A^* \succeq 0. Then, a weighted inner product is defined by

\langle x, y \rangle_A := \langle Ax, y \rangle_{\mathbb{F}^n}.

Weighted norms can be defined in different ways. For a general norm \lVert \cdot \rVert, a weighted version is given by

\lVert x \rVert_A := \lVert Ax \rVert

For the p-norms with p < \infty, the definition is usually changed to

\lVert x \rVert_{p, A} := \lVert A^{1/p} x \rVert,

where A^{1/p} is the p-th root of the matrix A. The reason for this definition is that for p = 2, this version is consistent with the inner product since \langle Ax, x \rangle = \langle A^{1/2} x, A^{1/2} x \rangle =
\lVert A^{1/2} x \rVert^2.

Remark on matrices as operators

A matrix M \in \mathbb{F}^{m \times n} can be regarded as a linear operator

\mathcal{M} &: \mathbb{F}^n \to \mathbb{F}^m \\
\mathcal{M}(x) &:= M x

It is well known that in the standard case of a Euclidean space, the adjoint operator is simply defined with the conjugate transposed matrix:

\mathcal{M}^* &: \mathbb{F}^m \to \mathbb{F}^n \\
\mathcal{M}^*(y) &:= M^* y

However if the spaces \mathbb{F}^n and \mathbb{F}^m have weighted inner products, this identification is no longer valid. If \mathbb{F}^{n \times n} \ni A = A^* \succeq 0 and \mathbb{F}^{m \times m} \ni B = B^* \succeq 0 are the weighting matrices of the inner products, we get

\langle \mathcal{M}(x), y \rangle_B
&= \langle B\mathcal{M}(x), y \rangle_{\mathbb{F}^m}
= \langle M x, B y \rangle_{\mathbb{F}^m}
= \langle x, M^* B y \rangle_{\mathbb{F}^n} \\
&= \langle A^{-1} A x, M^* B y \rangle_{\mathbb{F}^n}
= \langle A x, A^{-1} M^* B y \rangle_{\mathbb{F}^n} \\
&= \langle x, A^{-1} M^* B y \rangle_A

Thus, the adjoint of the matrix operator between the weighted spaces is rather given as \mathcal{M}^*(y) = A^{-1} M^* B y.