Linear Spaces¶

Definition and basic properties¶

A linear space over a field $\mathbb{F}$ is a set $\mathcal{X}$ , endorsed with the operations of vector addition " $+$ " and scalar multiplication " $\cdot$ " which are required to fullfill certain properties, usually called axioms. To emphasize the importance of all ingredients, vector spaces are often written as tuples $(\mathcal{X}, \mathbb{F}, +, \cdot)$ . We always assume that $\mathbb{F} = \mathbb{R}$ or $\mathbb{C}$ .

In the following, we list the axioms, which are required to hold for arbitrary $x, y, z \in \mathcal{X}$ and $a, b \in \mathbb{F}$ .

Associativity of addition	$(x + y) + z = (x + y) + z$
Commutativity of addition	$x + y = y + x$
Existence of a neutral element of addition	$0 + x = x + 0 = x$
Existence of inverse elements of addition	$\forall x\ \exists \bar x: \bar x + x = x + \bar x = 0$
Compatibility of multiplications	$a \cdot (b \cdot x) = (ab) \cdot x$
Neutral scalar is the neutral element of scalar multiplication	$1 \cdot x = x$
Distributivity with respect to vector addition	$a \cdot (x + y) = a \cdot x + a \cdot y$
Distributivity with respect to scalar addition	$(a + b) \cdot x = a \cdot x + b \cdot x$

Of course, the inverse element $\bar x$ is usually denoted with $-x$ .

Metric spaces¶

The vector space $(\mathcal{X}, \mathbb{F}, +, \cdot)$ is called a metric space if it is additionally endorsed with a distance function or metric

$d: \mathcal{X} \times \mathcal{X} \to [0, \infty)$

with the following properties for all $x, y, z \in \mathcal{X}$ :

$\begin{align*} & d(x, y) = 0 \quad \Leftrightarrow \quad x = y && \text{(identity of indiscernibles)} \\ & d(x, y) = d(y, x) && \text{(symmetry)} \\ & d(x, y) \leq d(x, z) + d(z, y) && \text{(subadditivity)} \end{align*}$

We call the tuple $(\mathcal{X}, \mathbb{F}, +, \cdot, d)$ a Metric space.

Normed spaces¶

A function on $\mathcal{X}$ intended to measure lengths of vectors is called a norm

$\lVert \cdot \rVert : \mathcal{X} \to [0, \infty)$

if it fulfills the following conditions for all $x, y \in \mathcal{X}$ and $a \in \mathbb{F}$ :

$\begin{align*} & \lVert x \rVert = 0 \Leftrightarrow x = 0 && \text{(positive definiteness)} \\ & \lVert a \cdot x \rVert = \lvert a \rvert\, \lVert x \rVert && \text{(positive homegeneity)} \\ & \lVert x + y \rVert \leq \lVert x \rVert + \lVert x \rVert && \text{(triangle inequality)} \end{align*}$

A tuple $(\mathcal{X}, \mathbb{F}, +, \cdot, \lVert \cdot \rVert)$ fulfilling these conditions is called Normed vector space. Note that a norm induces a natural metric via $d(x, y) = \lVert x - y \rVert$ .

Inner product spaces¶

Measure angles and defining notions like orthogonality requires the existence of an inner product

$\langle \cdot, \cdot \rangle : \mathcal{X} \times \mathcal{X} \to \mathbb{F}$

with the following properties for all $x, y, z \in \mathcal{X}$ and $a \in \mathbb{F}$ :

$\begin{align*} & \langle x, x \rangle \geq 0 \quad \text{and} \quad \langle x, x \rangle = 0 \Leftrightarrow x = 0 && \text{(positive definiteness)} \\ & \langle a \cdot x + y, z \rangle = a \, \langle x, z \rangle + a \, \langle y, z \rangle && \text{(linearity in the first argument)} \\ & \langle x, y \rangle = \overline{\langle x, y \rangle} && \text{(conjugate symmetry)} \end{align*}$

The tuple $(\mathcal{X}, \mathbb{F}, +, \cdot, \langle \cdot \rangle)$ is then called an Inner product space. Note that the inner product induces the norm $\lVert x \rVert = \sqrt{\langle x, x \rangle}$ .

Cartesian spaces¶

We refer to the space $\mathbb{F}^n$ as the $n$ -dimensional Cartesian space over the field $\mathbb{F}$ . We choose this notion since Euclidean spaces are usually associated with the Euclidean norm and distance, which are just (important) special cases. Vector addition and scalar multiplication in $\mathbb{F}^n$ are, of course, realized with entry-wise addition and scalar multiplication.

The natural inner product in $\mathbb{F}^n$ is defined as

$\langle x, y \rangle_{\mathbb{F}^n} := \sum_{i=1}^n x_i\, \overline{y_i}$

and reduces to the well-known dot product if $\mathbb{F} = \mathbb{R}$ . For the norm, the most common choices are from the family of p-norms

$\lVert x \rVert_p &:= \left( \sum_{i=1}^n \lvert x_i \rvert^p \right)^{\frac{1}{p}} \quad \text{if } p \in [1, \infty) \\[1ex] \lVert x \rVert_\infty &:= \max\big\{\lvert x_i \rvert\,|\, i \in \{1, \dots, n\} \big\}$

with the standard Euclidan norm for $p = 2$ . As metric, one usually takes the norm-induced distance function, although other choices are possible.

Weighted Cartesian spaces¶

In the standard definition of inner products, norms and distances, all components of a vector are have the same weight. This can be changed by using weighted versions of those functions as described in the following.

Let $A \in \mathbb{F}^{n \times n}$ be a Hermitian square and positive definite matrix, in short $A = A^* \succeq 0$ . Then, a weighted inner product is defined by

$\langle x, y \rangle_A := \langle Ax, y \rangle_{\mathbb{F}^n}.$

Weighted norms can be defined in different ways. For a general norm $\lVert \cdot \rVert$ , a weighted version is given by

$\lVert x \rVert_A := \lVert Ax \rVert$

For the $p$ -norms with $p < \infty$ , the definition is usually changed to

$\lVert x \rVert_{p, A} := \lVert A^{1/p} x \rVert,$

where $A^{1/p}$ is the $p$ -th root of the matrix $A$ . The reason for this definition is that for $p = 2$ , this version is consistent with the inner product since $\langle Ax, x \rangle = \langle A^{1/2} x, A^{1/2} x \rangle = \lVert A^{1/2} x \rVert^2$ .

Remark on matrices as operators¶

A matrix $M \in \mathbb{F}^{m \times n}$ can be regarded as a linear operator

$\mathcal{M} &: \mathbb{F}^n \to \mathbb{F}^m \\ \mathcal{M}(x) &:= M x$

It is well known that in the standard case of a Euclidean space, the adjoint operator is simply defined with the conjugate transposed matrix:

$\mathcal{M}^* &: \mathbb{F}^m \to \mathbb{F}^n \\ \mathcal{M}^*(y) &:= M^* y$

However if the spaces $\mathbb{F}^n$ and $\mathbb{F}^m$ have weighted inner products, this identification is no longer valid. If $\mathbb{F}^{n \times n} \ni A = A^* \succeq 0$ and $\mathbb{F}^{m \times m} \ni B = B^* \succeq 0$ are the weighting matrices of the inner products, we get

$\langle \mathcal{M}(x), y \rangle_B &= \langle B\mathcal{M}(x), y \rangle_{\mathbb{F}^m} = \langle M x, B y \rangle_{\mathbb{F}^m} = \langle x, M^* B y \rangle_{\mathbb{F}^n} \\ &= \langle A^{-1} A x, M^* B y \rangle_{\mathbb{F}^n} = \langle A x, A^{-1} M^* B y \rangle_{\mathbb{F}^n} \\ &= \langle x, A^{-1} M^* B y \rangle_A$

Thus, the adjoint of the matrix operator between the weighted spaces is rather given as $\mathcal{M}^*(y) = A^{-1} M^* B y$ .