Fourier Series and Fourier Transform - II

Certain feelings in my body lead me to believel that I have to stduy Fourier Series and Fourier Transform for a better understanding of probability theory, measure theory,entroy and information theory.

In our last post, we have introduced the big picture of Fourier Series. In this post, we will continue to explore the Fourier Series.

Definition of Fourier Series

For a periodic function f(x)f(x) with period 11 (you can normalize the period to 11 by scaling the xx-axis), we can write it as a Fourier Series:

f(x)=n=cne2πinx f(x) = \sum_{n=-\infty}^{\infty} c_n e^{2\pi i n x}

where cnc_n is the Fourier coefficient, which is the projection of f(x)f(x) onto the basis e2πinxe^{2\pi i n x}:

cn=f(x),e2πinx=01f(x)e2πinxdx c_n =\langle f(x), e^{2\pi i n x} \rangle = \int_0^1 f(x) e^{-2\pi i n x} dx

I like using the inner product notation to represent the Fourier coefficient, because it is more intuitive to me. The inner product is the projection of f(x)f(x) onto the basis e2πinxe^{2\pi i n x}, which is the same as the Fourier coefficient.

Before we moving on, let’s discuss the properties of the Fourier coefficient cnc_n:

  1. cnc_n is a complex number, which can be written as cn=an+ibnc_n = a_n + i b_n, where ana_n and bnb_n are real numbers.
  2. when f(x)f(x) is a real function, cn=cnc_n = \overline{c_{-n}}, where cn\overline{c_{-n}} is the complex conjugate of cnc_{-n}. This is because the basis e2πinxe^{2\pi i n x} is a complex number, and the inner product of two complex numbers is a complex number. When f(x)f(x) is a real function, the inner product of f(x)f(x) and e2πinxe^{2\pi i n x} is a real number, which is the same as the inner product of f(x)f(x) and e2πinxe^{-2\pi i n x}.
  3. c0c_0 is the average of f(x)f(x), which is a real number:c0=01f(x)dx c_0 = \int_0^1 f(x) dx

  4. when f(x)f(x) is even, cnc_n is also even, which means cn=cnc_n = c_{-n}. cn=01f(x)e2πinxdx=01f(s)e2πinsdslet s=x=10f(s)e2πin(s)ds=cs=cn\begin{aligned}c_n & = \int_0^1 f(x) e^{-2\pi i n x} dx \\ & = - \int_0^{-1} f(-s) e^{2\pi i n s} ds \quad \text{let } s = -x \\ & = \int_{-1}^0 f(-s) e^{-2\pi i n (-s)} ds \\ & = c_{-s} = c_{-n} \end{aligned}

  5. when f(x)f(x) is odd, cnc_n is odd, which means cn=cnc_n = -c_{-n}.

  6. when f(x)f(x) is real and even, cnc_n is real and even, which means cn=cnc_n = c_{-n}.

  7. when f(x)f(x) is real and odd, cnc_n is imaginary and odd, which means cn=cnc_n = -c_{-n}.

cn=cnwhen f(x) is oddcn=cnwhen f(x) is realcn=cnonly possible when it is pure imaginary c_n = -c_{-n} \quad \text{when } f(x) \text{ is odd} \\ c_n = \overline{c_{-n}} \quad \text{when } f(x) \text{ is real} \\ \longrightarrow - c_{-n} = \overline{c_{-n}} \quad \text{only possible when it is pure imaginary}

Those properties are very useful when we calculate the Fourier coefficient as they could help us to verify the correctness of our calculation. Since most of signals are real, we can use those properties in practice.

Two Examples

Conside a square wave of period 11, which is defined as:

f(x)={10x<12112x<1 f(x) = \begin{cases} 1 & 0 \leq x < \frac{1}{2} \\ -1 & \frac{1}{2} \leq x < 1 \end{cases}

square wave
Figure 1. Illustration of square wave.

The Fourier coefficient of f(x)f(x) is:

cn=01f(x)e2πinxdx=012e2πinxdx121e2πinxdx=12πine2πinx01212πine2πinx121=12πin[eπin1]12πin[e2πineπin]=12πin[eπin1e2πin+eπin]=12πin[2eπin1(cos(2πn)+isin(2πn))]=12πin[2eπin11]=12πin[2eπin2]=1πin[eπin1]=1πin[1eπin] \begin{aligned} c_n & = \int_0^1 f(x) e^{-2\pi i n x} dx \\ & = \int_0^{\frac{1}{2}} e^{-2\pi i n x} dx - \int_{\frac{1}{2}}^1 e^{-2\pi i n x} dx \\ & = \frac{1}{-2\pi i n} e^{-2\pi i n x} \Big|_0^{\frac{1}{2}} - \frac{1}{-2\pi i n} e^{-2\pi i n x} \Big|_{\frac{1}{2}}^1 \\ & = \frac{1}{-2\pi i n}[e^{-\pi i n} - 1] - \frac{1}{-2\pi i n}[e^{-2\pi i n} - e^{-\pi i n}] \\ & = \frac{1}{-2\pi i n}[e^{-\pi i n} - 1 - e^{-2\pi i n} + e^{-\pi i n}] \\ & = \frac{1}{-2\pi i n}[2e^{-\pi i n} -1 - (\cos(-2\pi n) + i \sin(2\pi n))] \\ & = \frac{1}{-2\pi i n}[2e^{-\pi i n} -1 - 1] \\ & = \frac{1}{-2\pi i n}[2e^{-\pi i n} -2] \\ & = \frac{1}{-\pi i n}[e^{-\pi i n} -1] \\ & = \frac{1}{\pi i n}[1- e^{-\pi i n}] \\ \end{aligned}

Therefore, the fourier series of f(x)f(x) is:

f(x)=n=,n0cne2πinx=n=,n01πin(1eπin)e2πinx \begin{aligned} f(x) & = \sum_{n=-\infty, n\neq 0}^{\infty} c_n e^{2\pi i n x} \\ & = \sum_{n=-\infty, n \neq 0}^{\infty} \frac{1}{\pi i n}(1- e^{-\pi i n}) e^{2\pi i n x} \end{aligned}

Notice, f(x)f(x) is an odd function, so cnc_n is imaginary and odd, which means cn=cnc_n = -c_{-n}. Notice that

1eπin=1cos(πn)isin(πn)=1(1)nisin(πn)={0n is even2n is odd \begin{aligned} 1 - e^{-\pi i n} & = 1 - \cos(\pi n) - i \sin(\pi n) \\ & = 1 - (-1)^n - i \sin(\pi n) \\ & = \begin{cases} 0 & n \text{ is even} \\ 2 & n \text{ is odd} \end{cases} \end{aligned}

So the series can be simplified as:

f(x)=n=,n0cne2πinx=n is odd2πine2πinx \begin{aligned} f(x) & = \sum_{n=-\infty, n\neq 0}^{\infty} c_n e^{2\pi i n x} \\ & = \sum_{n \text{ is odd}} \frac{2}{\pi i n}e^{2\pi i n x} \\ \end{aligned}

Reflections
We have shown that when the function is real and odd, the fourier coefficients are pure imaginary and odd.

Now, we combine the positive and negative terms together:

e2πinxe2πinx=2isin(2πnx) e^{2\pi i n x} - e^{-2\pi i n x} = 2i \sin(2\pi n x)

let n=2k+1n = 2k+1, we have:

f(x)=n is odd2πine2πinx=2πi(2k+1)e2πi(2k+1)x=2πi(2k+1)(e2πi(2k+1)xe2πi(2k+1)x)=k=04π(2k+1)sin[2π(2k+1)x]=4πk=012k+1sin[2π(2k+1)x] \begin{aligned} f(x) & = \sum_{n \text{ is odd}} \frac{2}{\pi i n}e^{2\pi i n x} \\ & = \sum_{-\infty}^{\infty} \frac{2}{\pi i (2k+1)}e^{2\pi i (2k+1) x} \\ & = \sum_{-\infty}^{\infty} \frac{2}{\pi i (2k+1)}(e^{2\pi i (2k+1) x} - e^{-2\pi i (2k+1) x}) \\ & = \sum_{k=0}^{\infty} \frac{4}{\pi (2k+1)} \sin[2\pi (2k+1) x] \\ & = \frac{4}{\pi} \sum_{k=0}^{\infty} \frac{1}{2k+1} \sin[2\pi (2k+1) x] \end{aligned}

Here is the visualization of the Fourier Series of the square wave (when N=100N=100, you can click the right bottom corner to see the animation):

From the above example,we can see that the fourier series is ‘converging’ to the square wave. The more terms we add, the more similar it is to the square wave. However, we also see discontinuity at the jump points. This is called Gibbs phenomenon. Since both sine and cosine are continuous, the fourier series of a function is also continuous. Therefore the fourier series of a discontinuous function will have discontinuity at the jump points.

Now, let’s see another example - traingle wave - which is defined as:

f(t)=12t={12+t12t<012t0t<12 f(t) = \frac{1}{2} - |t| = \begin{cases} \frac{1}{2} + t & -\frac{1}{2} \leq t < 0 \\ \frac{1}{2} - t & 0 \leq t < \frac{1}{2} \end{cases}

triangle wave
Figure 2. Illustration of triangle wave.

The coefficient of f(t)f(t) is at n=0n=0 is the average of f(t)f(t), which is 1/41/4. For n0n \neq 0, we have:

cn=1/21/2f(t)e2πintdt=1/21/2(12t)e2πintdt=121/21/2e2πintdt1/21/2te2πintdt=1/21/2te2πintdt;since 1/21/2e2πintdt=0=(1/20te2πintdt+01/2te2πintdt)=1/20te2πintdt01/2te2πintdt \begin{aligned} c_n & = \int_{-1/2}^{1/2} f(t) e^{-2\pi i n t} dt \\ & = \int_{-1/2}^{1/2} (\frac{1}{2} - |t|) e^{-2\pi i n t} dt \\ & = \frac{1}{2} \int_{-1/2}^{1/2} e^{-2\pi i n t} dt - \int_{-1/2}^{1/2} |t| e^{-2\pi i n t} dt \\ & = - \int_{-1/2}^{1/2} |t| e^{-2\pi i n t} dt; \quad \text{since } \int_{-1/2}^{1/2} e^{-2\pi i n t} dt = 0 \\ & = - \bigg( \int_{-1/2}^{0} - t e^{-2\pi i n t} dt + \int_{0}^{1/2} t e^{-2\pi i n t} dt \bigg) \\ & = \int_{-1/2}^{0} t e^{-2\pi i n t} dt - \int_{0}^{1/2} t e^{-2\pi i n t} dt \\ \end{aligned}

Now, let A(n)A(n) be the first integral and we have:

A(n)=1/20te2πintdt \begin{aligned} A(n) & = \int_{-1/2}^{0} t e^{-2\pi i n t} dt \end{aligned}

It is easy to show that

A(n)=1/20te2πintdt=1/20se2πinsdslet s=t=1/20se2πinsds=01/2se2πinsds=A(n)=1/20te2πintdt \begin{aligned} A(-n) & = \int_{-1/2}^{0} t e^{2\pi i n t} dt \\ & = \int_{1/2}^0 -s e^{-2\pi i n s} - ds \quad \text{let } s = -t \\ & = \int_{1/2}^0 s e^{-2\pi i n s} ds \\ & = - \int_{0}^{1/2} s e^{-2\pi i n s} ds \\ & = - A(n) = - \int_{-1/2}^{0} t e^{-2\pi i n t} dt \\ \end{aligned}

Therefore, the fourier coefficient can be written as:

cn=A(n)+A(n) c_n = A(n) + A(-n)

Now, let’s integrate A(n)A(n) by parts:

A(n)=1/20te2πintdt=12πinte2πint1/201/2012πine2πintdt=12πinte2πint1/201(2πin)2e2πint1/20=12πin[0+12eπin]1(2πin)2[1eπin]=14πineπin+14π2n2[1eπin]=πin4π2n2eπin+14π2n2[1eπin]=14π2n2[1eπin+πineπin]=14π2n2[1+eπin(πin1)] \begin{aligned} A(n) & = \int_{-1/2}^{0} t e^{-2\pi i n t} dt \\ & = \frac{1}{-2\pi i n} t e^{-2\pi i n t} \Big|_{-1/2}^0 - \int_{-1/2}^{0} \frac{1}{-2\pi i n} e^{-2\pi i n t} dt \\ & = \frac{1}{-2\pi i n} t e^{-2\pi i n t} \Big|_{-1/2}^0 - \frac{1}{(2\pi i n)^2} e^{-2\pi i n t} \Big|_{-1/2}^0 \\ & = \frac{1}{-2\pi i n} [0 + \frac{1}{2} e^{\pi i n}] - \frac{1}{(2\pi i n)^2} [1 - e^{\pi i n}] \\ & = - \frac{1}{4\pi i n}e^{\pi in} + \frac{1}{4\pi^2n^2} [1 - e^{\pi i n}] \\ & = \frac{\pi i n}{4 \pi^2 n^2} e^{\pi i n} + \frac{1}{4\pi^2n^2} [1 - e^{\pi i n}] \\ & = \frac{1}{4\pi^2n^2} [1 - e^{\pi i n} + \pi i n e^{\pi i n}] \\ & = \frac{1}{4\pi^2n^2} [ 1 + e^{\pi i n} (\pi i n - 1)] \end{aligned}

Therefore, we could have

A(n)=14π2n2[1+eπin(πin1)] A(-n) = \frac{1}{4\pi^2n^2} [ 1 + e^{-\pi i n} (-\pi i n - 1)]

The fourier coefficient is:

cn=A(n)+A(n)=14π2n2[1+eπin(πin1)]+14π2n2[1+eπin(πin1)]=14π2n2[2+eπin(πin1)+eπin(πin1)]=14π2n2[2+(cos(πn)+isin(πn))(πin1)(cos(πn)isin(πn))(πin+1)]=14π2n2[2+cos(πn)(πin1)cos(πn)(πin+1)]=12π2n2(1cos(πn))={0n is even1π2n2n is odd \begin{aligned} c_n & = A(n) + A(-n) \\ & = \frac{1}{4\pi^2n^2} [ 1 + e^{\pi i n} (\pi i n - 1)] + \frac{1}{4\pi^2n^2} [ 1 + e^{-\pi i n} (-\pi i n - 1)] \\ & = \frac{1}{4\pi^2 n^2} [ 2 + e^{\pi i n} (\pi i n - 1) + e^{-\pi i n} (-\pi i n - 1)] \\ & = \frac{1}{4\pi^2 n^2} [ 2 + (\cos(\pi n) + i \sin(\pi n)) (\pi i n - 1) - (\cos(\pi n) - i \sin(\pi n)) (\pi i n + 1)] \\ & = \frac{1}{4\pi^2 n^2} [ 2 + \cos(\pi n)(\pi in - 1) - \cos(\pi n)(\pi in +1)] \\ & = \frac{1}{2\pi^2 n^2} (1 - \cos(\pi n)) \\ & = \begin{cases} 0 & n \text{ is even} \\ \frac{1}{\pi^2 n^2} & n \text{ is odd} \end{cases} \end{aligned}

Now, let’s write down the fourier series of f(t)f(t):

f(t)=n=cne2πint=n is odd1π2n2e2πint=01π2n2e2πint+11π2n2e2πint=cne2πint+cne2πint=cn(e2πint+e2πint)=2π2n2cos(2πnt)=14+k=01π2(2k+1)2cos[2π(2k+1)t] \begin{aligned} f(t) & = \sum_{n=-\infty}^{\infty} c_n e^{2\pi i n t} \\ & = \sum_{n \text{ is odd}} \frac{1}{\pi^2 n^2} e^{2\pi i n t} \\ & = \sum_{-\infty}^{0} \frac{1}{\pi^2 n^2} e^{2\pi i n t} + \sum_{1}^{\infty} \frac{1}{\pi^2 n^2} e^{2\pi i n t} \\ & = c_{-n} e^{-2\pi i n t} + c_n e^{2\pi i n t} \\ & = c_n (e^{2\pi i n t} + e^{-2\pi i n t}) \\ & = \frac{2}{\pi^2 n^2} \cos (2 \pi n t) \\ & = \frac{1}{4} + \sum_{k=0}^\infty \frac{1}{\pi^2 (2k+1)^2} \cos[2\pi (2k+1) t] \\ \end{aligned}

For this example, there is no joumping points, so there is no Gibbs phenomenon. The fourier series is converging to the triangle wave. However, since we have infinite terms, the fourier series is not a triangle wave. It is a smooth triangle wave. The fourier series is a smooth approximation of the triangle wave. The more terms we add, the more similar it is to the triangle wave.

This is due to the fact that the fourier series is a linear combination of the basis e2πinte^{2\pi i n t}. The basis e2πinte^{2\pi i n t} is a smooth function, so the fourier series is also a smooth function. Or put it in another way, both sines and cosines are differentiable to any order, so the fourier series is also differentiable to any order.

In summary, a discontinuoity in any order derivative of a periodic function will force an infinite number of terms in the fourier series to approximate the function.

Note also that for the triangle wave the coefficients decrease like 1/n21/n^2 while for the square wave they decrease like 1/n1/n. Or, it takes around N=100N=100 terms to approximate the square wave, but it only takes around N=10N=10 terms to approximate the triangle wave. This has exactly do do wit the fact that the square wave is discontinuous while the triangle wave is continuous but its derivative is discontinuous.

Reflections
I hope those two examples could give you the sense of how the fourier series works and how it converges to the original function in terms of the speed and the smoothness.

Convergence of Fourier Series

Until now, we have assumed that the period is always 11. Now, let’s assume ff is periodic at interval LL from [a,b][a, b], which means f(x+L)=f(x)f(x+L) = f(x). We can write the fourier series as:

cn=f^(n)=1Labf(x)e2πinx/Ldx,nZ(1) c_n = \hat{f}(n) = \frac{1}{L} \int_a^b f(x) e^{-2\pi i n x / L} dx, \quad n \in \mathbb{Z} \tag{1}

The NN-th partial sum of the fourier series is:

SN(f)(x)=n=NNf^(n)e2πinx/L(2) S_N(f)(x) = \sum_{n=-N}^{N} \hat{f}(n) e^{2\pi i n x / L} \tag{2}

Now, we try to answer the following questions:

Roughly speaking, there are three senses of convergence:

  1. Pointwise Convergence: SN(f)(x)S_N(f)(x) converges to f(x)f(x) for every xx.
  2. Uniform Convergence: SN(f)(x)S_N(f)(x) converges to f(x)f(x) uniformly. In words, when NN is large, the partial sum SN(f)(x)S_N(f)(x) is close to f(x)f(x) for every xx over the entire interval [a,b][a, b].
  3. Mean Square Convergence: SN(f)(x)S_N(f)(x) converges to f(x)f(x) in the mean square sense. In words, the average of the square of the difference between SN(f)(x)S_N(f)(x) and f(x)f(x) converges to 00 as NN \rightarrow \infty, meaning:

limNabSN(f)(x)f(x)2dx=0(3) \lim_{N \rightarrow \infty} \int_a^b |S_N(f)(x) - f(x)|^2 dx = 0 \tag{3}

We will not prove the convergence of the fourier series here. We refer the readers to the two examples we have shown above. The square wave is discontinuous, so the fourier series converges to the square wave in the mean square sense. The triangle wave is continuous, so the fourier series converges to the triangle wave uniformly. Generally speaking, uniform convergence is the strongest form of convergence. Pointwise convergence is the weakest form of convergence. Mean square convergence is in between, which is also very subtle to study.

Dirichlet Kernel

After introducing the partial sum, it is natural to ask how good is the partial sum SN(f)(x)S_N(f)(x) in approximating f(x)f(x). The answer is given by the Dirichlet kernel. Now, let’s examine the partial sum SN(f)(x)S_N(f)(x) (to simplify the notation, we assume L=1L=1):

SN(f)(x)=n=NNf^(n)e2πinx/L=n=NN1Labf(t)e2πint/Ldt e2πinx/L=abf(t)n=NNe2πin(tx)dt=abf(t)DN(tx)dt \begin{aligned} S_N(f)(x) & = \sum_{n=-N}^{N} \hat{f}(n) e^{2\pi i n x / L} \\ & = \sum_{n=-N}^{N} \frac{1}{L} \int_a^b f(t) e^{-2\pi i n t / L} dt \ e^{2\pi i n x / L} \\ & = \int_a^b f(t) \sum_{n=-N}^{N} e^{-2\pi i n (t-x) } dt \\ & = \int_a^b f(t) D_N(t-x) dt \\ \end{aligned}

where DN(x)D_N(x) is the Dirichlet kernel:

DN(x)=n=NNe2πinx=sin[(N+12)2πx]sin(πx)(4) D_N(x) = \sum_{n=-N}^{N} e^{-2\pi i n x} = \frac{\sin[(N+\frac{1}{2})2\pi x]}{\sin(\pi x)} \tag{4}

We will not discuss the derivation of the Dirichelt kernel here. We will learn more about the Dirichlet kernel in the future when we talk about the convolution.

Orthogonality of the basis

In the previous post, we have show that

en(t)=e2πint e_n(t) = e^{2\pi i n t}

is an orthogonal basis. From this, we could derive Pythagoras’s Theorem for the inner product:

f,g=abf(x)g(x)dx \langle f, g \rangle = \int_a^b f(x) \overline{g(x)} dx

For our basis, we have:

en,em=abe2πinte2πimtdt={1n=m0nm \begin{aligned} \langle e_n, e_m \rangle & = \int_a^b e^{2\pi i n t} \overline{e^{2\pi i m t}} dt \\ & = \begin{cases} 1 & n = m \\ 0 & n \neq m \end{cases} \end{aligned}

The Pythagoras’s Theorem for the inner product is:

n=NNen2=n=NNen2(5) \bigg | \bigg | \sum_{n=-N}^{N} e_n \bigg | \bigg |^2 = \sum_{n=-N}^{N} |e_n|^2 \tag{5}

Here is the proof:

n=NNen2=n=NNen,n=NNen=n=NNm=NNen,emby linearity=n=NNm=NN{<en,em>n=m0nm=n=NNen2 \begin{aligned} \bigg | \bigg | \sum_{n=-N}^{N} e_n \bigg | \bigg |^2 & = \bigg \langle \sum_{n=-N}^{N} e_n, \sum_{n=-N}^{N} e_n \bigg \rangle \\ & = \sum_{n=-N}^N \sum_{m=-N}^N \langle e_n, e_m \rangle \quad \text{by linearity} \\ & = \sum_{n=-N}^N \sum_{m=-N}^N \begin{cases} <e_n, e_m> & n = m \\ 0 & n \neq m \end{cases} \\ & = \sum_{n=-N}^N |e_n|^2 \end{aligned}

Some important inequalities

Before we finish this post, let’s introduce some important inequalities that are useful when we study the fourier series:

The norm of a function f(x)f(x) is defined as:

f(x)=f(x),f(x)=abf(x)2dx(9) ||f(x)|| = \sqrt{\langle f(x), f(x) \rangle} = \sqrt{\int_a^b |f(x)|^2 dx} \tag{9}

If you forget how we calculate the absolute value of a complex number, here is a quick review:

z=zz=a2+b2(10) |z| = \sqrt{z \overline{z}} = \sqrt{a^2 + b^2} \tag{10}

where z=a+biz = a + bi and z=abi\overline{z} = a - bi. Therefore,

f(x),f(x)=abf(x)f(x)dx=abf(x)2dx=f(x)2 \langle f(x), f(x) \rangle = \int_a^b f(x) \overline{f(x)} dx = \int_a^b |f(x)|^2 dx = ||f(x)||^2

With the defintion of norm, let’s prove the Bessel’s inequality. For the complex inner product, we have:

f+g2=f+g,f+g \begin{aligned} ||f + g||^2 & = \langle f + g, f + g \rangle \\ \end{aligned}

0f(x)n=NNf(x),e2πinxe2πinx2=f(x)n=NNf(x),e2πinxe2πinx,f(x)n=NNf(x),e2πinxe2πinx=f(x)2n=NNf(x),e2πinx2=f(x)2n=NNf^(n)2 \begin{aligned} 0 \leq \bigg | \bigg | f(x) - \sum_{n=-N}^{N} \langle f(x), e^{2\pi i n x} \rangle e^{2\pi i n x} \bigg | \bigg |^2 & = \langle f(x) - \sum_{n=-N}^{N} \langle f(x), e^{2\pi i n x} \rangle e^{2\pi i n x}, f(x) - \sum_{n=-N}^{N} \langle f(x), e^{2\pi i n x} \rangle e^{2\pi i n x} \rangle \\ & = ||f(x)||^2 - \sum_{n=-N}^{N} |\langle f(x), e^{2\pi i n x} \rangle|^2 \\ & = ||f(x)||^2 - \sum_{n=-N}^{N} |\hat{f}(n)|^2 \end{aligned}

This proves the Bessel’s inequality in equation (6). The complete proof of Bessel’s inequality can be found here.

Now, let’s derive the Rayleigh’s Identity. We will assume L=1L=1 for simplicity:

f,f=01f(x)f(x)dx=01f(x)2dx=n=f^(n)e2πinx,m=f^(m)e2πimx=n=f,enen,m=f,emem=n,mf,enf,emen,emusing linearity=n,mf,enf,emδn,musing orthogonality=nf,en2=nf^(n)2 \begin{aligned} \langle f, f \rangle & = \int_0^1 f(x) \overline{f(x)} dx \\ & = \int_0^1 |f(x)|^2 dx \\ & = \bigg \langle \sum_{n=-\infty}^{\infty} \hat{f}(n) e^{2\pi i n x}, \sum_{m=-\infty}^{\infty} \hat{f}(m) e^{2\pi i m x} \bigg \rangle \\ & = \bigg \langle \sum_{n=-\infty}^{\infty} \langle f, e_n \rangle e_n, \sum_{m=-\infty}^{\infty} \langle f, e_m \rangle e_m \bigg \rangle \\ & = \sum_{n, m} \langle f, e_n \rangle \overline{ \langle f, e_m \rangle} \langle e_n, e_m \rangle \quad \text{using linearity} \\ & = \sum_{n, m} \langle f, e_n \rangle \overline{ \langle f, e_m \rangle} \delta_{n, m} \quad \text{using orthogonality} \\ & = \sum_{n} |\langle f, e_n \rangle|^2 \\ & = \sum_{n} |\hat{f}(n)|^2 \end{aligned}

This proves the Rayleigh’s Identity in equation (7). This means the energy of the function f(x)f(x) is the sum of the energy of the fourier coefficients.

We will not prove the Cauchy-Schwarz inequality here as this one is so well-known. The proof can be found anywhere on the internet.

Application of Rayleigh’s Identity

Now, let’s use the Rayleigh’s Identity to prove the following identity:

n=11n2=π26(11) \sum_{n=1}^{\infty} \frac{1}{n^2} = \frac{\pi^2}{6} \tag{11}

Euler proved this identity in 1735. This is also a special case of zeta function, which is defined as:

ζ(s)=n=11ns(12) \zeta(s) = \sum_{n=1}^{\infty} \frac{1}{n^s} \tag{12}

All those equations are also realted to the Basel problem, which is fun to read.

To derive the identity, we first define a function f(x)f(x) as:

f(x)=xon (π,π)(13) f(x) = x \quad \text{on } (-\pi, \pi) \tag{13}

The period of f(x)f(x) is 2π2\pi. Therefore, we can write the fourier coefficient of f(x)f(x) as:

f^(n)=12πππxe2πinx/Ldx=12πππxeinxdx=12π(1inxeinxππππ1ineinxdx)=12π(1inxeinxππ1(in)2einxππ)=12π(1in(πeinπ(π)einπ)1(in)2(einπeinπ))=12π(1inπ(einπ+einπ)1(in)2sinnπ2i)=12π1inπ2cosnπ=cosnπin=(1)n+1inn0 \begin{aligned} \hat{f}(n) & = \frac{1}{2\pi} \int_{-\pi}^{\pi} x e^{-2\pi i n x/L} dx \\ & = \frac{1}{2\pi} \int_{-\pi}^{\pi} x e^{- i n x} dx \\ & = \frac{1}{2\pi} \bigg( \frac{1}{-i n} x e^{ -i n x} \Big|_{-\pi}^{\pi} - \int_{-\pi}^{\pi} \frac{1}{-i n} e^{ -i n x} dx \bigg) \\ & = \frac{1}{2\pi} \bigg( \frac{1}{-i n} x e^{ -i n x} \Big|_{-\pi}^{\pi} - \frac{1}{(i n)^2} e^{ -i n x} \Big|_{-\pi}^{\pi} \bigg) \\ & = \frac{1}{2\pi} \bigg( \frac{1}{i n} (\pi e^{ -i n \pi} - (-\pi) e^{i n \pi}) - \frac{1}{(i n)^2} (e^{ -i n \pi} - e^{- i n \pi}) \bigg) \\ & = \frac{1}{2\pi} \bigg( \frac{1}{-i n} \pi (e^{in\pi} + e^{-in\pi}) - \frac{1}{(i n)^2} \frac{\sin n \pi}{2i} \bigg) \\ & = \frac{1}{2\pi} \frac{1}{-i n} \pi \cdot 2 \cos n \pi \\ & = \frac{\cos n \pi}{-in} \\ & = \frac{(-1)^{n+1}}{in} \quad n \neq 0 \end{aligned}

Therefore we have the fourier series of f(x)f(x):

f(x)=n=f^(n)e2πinx/L=n=,n0(1)n+1ineinx \begin{aligned} f(x) & = \sum_{n=-\infty}^{\infty} \hat{f}(n) e^{2\pi i n x/L} \\ & = \sum_{n=-\infty, n\neq 0}^{\infty} \frac{(-1)^{n+1}}{in} e^{ i n x} \end{aligned}

From the above equation, we can see that f(x)f(x) is an odd function, so the fourier coefficient is imaginary and odd. Based on Rayleigh’s Identity, we have:

12πππf(x)2dx=n=f^(n)2=1f^(n)2+1f^(n)2=1(1)n+1in2+n=1(1)n+1in2=2n=11n2=12πππx2dx=12πx33ππ=π23 \begin{aligned} \frac{1}{2\pi} \int_{-\pi}^\pi |f(x)|^2 dx & = \sum_{n=-\infty}^{\infty} |\hat{f}(n)|^2 \\ & = \sum_{-\infty}^{-1} |\hat{f}(n)|^2 + \sum_{1}^{\infty} |\hat{f}(n)|^2 \\ & = \sum_{-\infty}^{-1} \bigg | \frac{(-1)^{n+1}}{in} \bigg |^2 + \sum_{n=1}^{\infty} \bigg | \frac{(-1)^{n+1}}{in} \bigg |^2 \\ & = 2 \sum_{n=1}^{\infty} \frac{1}{n^2} \\ & = \frac{1}{2\pi} \int_{-\pi}^\pi x^2 dx \\ & = \frac{1}{2\pi} \frac{x^3}{3} \Big|_{-\pi}^{\pi} \\ & = \frac{\pi^2}{3} \\ \end{aligned}

Therefore, we have:

n=11n2=π26 \sum_{n=1}^{\infty} \frac{1}{n^2} = \frac{\pi^2}{6}

Reflections
I have mentioned that learning Fourer Series will help us to understand probability theory. Here is an example. The above example is related to the Basel problem and Zeta function, which are related to Gamma function. When you study probability theory, you will see Gamma function a lot, for example, the Gamma distribution or the Gamma distribution as the conjugate prior of the Poisson distribution, etc.