Reproducing Kernel Hilbert Space
Kernel Function
A function $K : X \times X \to \mathbb{C}$ is called a kernel if there exists a complex Hilbert space $\mathcal{H}$ and a feature map $\phi : X \to \mathcal{H}$ such that
\[K(x,y) = \langle \phi(x), \phi(y) \rangle_{\mathcal{H}} \quad \text{for all } x, y \in X.\]Equivalently, we say $K$ is a kernel function if it satisfies:
-
Hermitian symmetry: \(K(x,y) = \overline{K(y,x)}, \quad \forall x, y \in X.\)
-
Positive semidefiniteness (PSD): for any integer $n$, any points $x_1, \dots, x_n \in X$, and any complex coefficients $\xi_1, \dots, \xi_n \in \mathbb{C}$, \(\sum_{i=1}^n \sum_{j=1}^n \xi_i \,\overline{\xi_j}\, K(x_i, x_j) \;\ge\; 0.\)
\[\iint_{X \times X} K(x,y)\, f(x)\,\overline{f(y)} \; d\mu(x)\,d\mu(y) \;\ge\; 0.\]Alternatively, when $X$ is equipped with a measure space structure, one can use an integral quadratic form version: If $(X,\mu)$ is a measure space and $K$ is measurable, then for all measurable $f: X \to \mathbb{C}$ with compact support (or in $L^2(\mu)$, under suitable conditions),
Equivalences:
- Feature-map ↔ Hermitian + finite-sum PSD (via Moore–Aronszajn theorem)
- Finite-sum PSD ↔ Integral-form PSD when approximating general measures/functions under regularity conditions
Construction of RKHS
-
Starting Point: Given a kernel $K$ on $X$, we define the kernel section: \(K_x(\cdot) := K(\cdot, x), \quad x \in X.\) Consider the linear span of functions ${ K_x : x \in X }$, which we denote as $\mathcal{H}_0$.
-
Inner Product: Define an inner product on $\mathcal{H}_0$ by \(\left\langle \sum_{i=1}^n \alpha_i K_{x_i}, \sum_{j=1}^m \beta_j K_{y_j} \right\rangle_{\mathcal{H}_0} := \sum_{i=1}^n \sum_{j=1}^m \alpha_i \overline{\beta_j} K(x_i, y_j).\) This is well-defined due to the PSD property of $K$.
- Completion: The space $\mathcal{H}_0$ with this inner product is a pre-Hilbert space. Its completion, denoted $\mathcal{H}$, is a Hilbert space.
- Reproducing Property: For any $f \in \mathcal{H}$ and $x \in X$, we have \(f(x) = \langle f, K_x \rangle_{\mathcal{H}}.\)
- Uniqueness: The RKHS $\mathcal{H}$ is unique up to isometric isomorphism. \(\text{RKHS} \;\longleftrightarrow\; \text{Kernel}\) \(\mathcal{H} \;\longleftrightarrow\; K\)
- Boundedness of Evaluation Functional: For each $x \in X$, the evaluation functional $\delta_x : \mathcal{H} \to \mathbb{C}$ defined by $\delta_x(f) = f(x)$ is bounded, with norm $|\delta_x| = \sqrt{K(x,x)}$.
Conections with integral operators and Mercer Theorem
If $X$ has a measure $\mu$ and $K$ is measurable, we can define an integral operator $T_K : L^2(X, \mu) \to L^2(X, \mu)$ by \((T_K f)(x) := \int_X K(x,y) f(y) \, d\mu(y).\) The operator $T_K$ is self-adjoint and positive.
If $K$ is continuous and $X$ is compact, then by Mercer’s theorem, $T_K$ is a compact, self-adjoint, positive operator. It has an orthonormal basis of eigenfunctions ${e_n}$ with corresponding non-negative eigenvalues ${\lambda_n}$. The kernel can be expressed as \(K(x,y) = \sum_{n=1}^\infty \lambda_n e_n(x) \overline{e_n(y)},\) with convergence absolute and uniform.
