# 1. Introduction

## What Geometric Deep Learning is all about?

**Gist:** Fundamental Principles underlying Deep Representation Learning architectures. Single word to convey gist of the course: **Symmetry**

> Symmetry, as wide or as narrow as you may define its meaning, is **one idea** by which man through the ages has tried to comprehend and create order, beauty and perfection. **H. Weyl 1952**

## Historical background

The term **symmetry** has **Greek** origin (*συμμετρία*). Symmetry literally translates as **same measure** and ancient greeks used this term to somehow **vaguely** convey the the **beauty** of proportion in arts and **harmony**.

**Plato** considered the **five regular polyhedra** (\~370 BC) what we now call the **platonic solids**. Originally thought as fundamental must be **building blocks** that shape the physical world. This idea was not very far from the truth. **Kepler** many centuries after attempted a rigorous analysis of the symmetry. He was in particular concerned with the symmetric shape of **water crystals** and he wrote a book titled **“On the six-cornered snowflake”** (1611)

Nowadays known as hexagonal packing of particles, it was an idea that clearly preceded the understanding of **how matter is formed** and the concept of atoms, molecules and crystals. It holds today as the basis of modern **crystallography**.

**Modern geometry** is also traced back to ancient Greece and the seminal work of **Euclide's elements**.

### Euclidean geometry

> In a plane, given a line and a point not on it, at most one line parallel to the given line can be drawn through the point. **Euclide (\~300 BC)**

This type of geometry was the only one known for **over 2000 years**. At the **core** of Euclidean geometry was a set of **5 basic assumptions**, or how Euclid called them, **postulates** that he used to derive certain **properties** and **prove** results. For hundred of years the **fifth postulate** of Euclidian geometry is stating that:

> It's possible to pass only **one line parallel** to a given line through a point that lies outside of it.

This fifth postulate subsequently defied any attempt to try to derive it from the other postulates of geometry.

### End of Euclid’s Monopoly

In the **19th** the Euclidian monopoly came to an end. The 19th century was really a remarkable burst of creativity that made **geometry** into probably one of the most exciting fields of mathematics. **First** it was the development of what is called **projective geometry** (J. V. Poncelet **1822**).

#### No exists notion of parallelism

In projective geometry, **points** and **lines** are **interchangeable** and there is **no** such thing as **parallelism**. Any two lines **intersect** at one exactly **point**. Nowadays it's very popular in **computer graphics**. **Non considered non-euclidian geometry strictly**. But it was probably the first one to undermine this euclidian concept of **parallelism**.

#### Hyperbolic Geometry

**First published construction of a non-euclidian geometry**. Is credited to **N. Lobachevsky** (1826), a Russian mathematician that considered the fifth axiom of euclidian geometry a completely **arbitrary limitation**.

> In geometry I find certain imperfections which I hold to be the reason why this science \[...] can as yet make no advance from that state in which it came to us from Euclid. As belonging to these imperfections, I consider \[...] the momentous gap in the theory of parallels, to fill which all efforts of mathematicians have so far been in vain. **N. Lobachevsky 1826**

**Alternative proposed postulate**: **More** than **one** line can pass through a **point** that is **parallel** to a given one. Such construction required a **space** with **negative curvature**. This type of space is now called a **hyperbolic space**.

**Academia rejection of the proposal**

This idea was so **unconventional** and **theoretical** at the time of the publication that he was openly derided by colleagues at his university for writing and publishing such nonsense. Some mathematicians of his time said that this was at the level of some **school teacher** not a university professor.

**Different persons came to the same idea from different backgrounds**

**J. Bolyai** (1832), a Hungarian mathematician apparently came to the same ideas together with Lobachevsky.

> I have discovered such wonderful things that I was amazed...out of nothing I have created a strange new universe. — **Jánus Bolyai to his father**

> To praise it would amount to praising myself. For the entire content of the work...coincides almost exactly with my own meditations \[in the] past thirty or thirty-five years.” — **Gauss to Farkas Bolyai**

#### Riemann (Differential) Geometry

> **Manifolds** in which, as in the plane and in space, the line-element may be reduced to the form,
>
> $$
> \sqrt{\sum{dx^2}}
> $$
>
> are therefore only a particular case of the manifolds to be here investigated; they require a special name, and therefore these manifolds in which the square of the line-element may be expressed as the sum of the squares of complete differentials I will call **flat**.” **Riemann**

**Last nail in Euclid's coffin**

**Gauss's** own Ph.D student **Bernard Riemann**. In his lecture on the hypothesis on which geometry is based, he basically formulated what is nowadays called **differential geometry of surfaces**.

**Euclidian geometry second axiom doesn't hold**

He also constructed **non-euclidian geometry** on the **sphere**, that is sometimes called the **Riemann Geometry** in the narrow sense, and in this case the **fifth postulate** doesn't hold as well as in the construction of **Lobachevsky**, **BUT** also the **second** postulate doesn't hold.

**Second postulate**: All straight lines can be **continued indefenitely**. On the **sphere** all straight lines have **finite** lengths.

**Entire zoo of different geometries emerged**

![](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-4b031c57b73c1af3e74eb4c1e950611d78bfe2a7%2Fzoo_geometries.png?alt=media)

Towards the end of the **1800s** these geometries became **non-unified** fields. Mathematicians were **debating** which geometry is the **right** one and **WHAT** actually defines the geometry.

#### WHAT actually defines geometry? The Erlangen Programme

> Given a \[homogeneous] manifold and a transformation group acting \[transitively] on it, to investigate those properties of figures on that manifold which are invariant under transformations of that group. **F. Klein 1872**

**Erlangen Programme**

Klein proposed a very **radical** approach of **treating** geometry as the **study of invariants** and **symmetries**. These are the **properties** that remain **unchanged** under some class of **transformation**. This approach immediately created clarity by showing that **different** geometries could be defined by an appropriated choice of **symmetry**.

![Erlangen Programme](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-a566b867ceb3f9733f5cb06648512073099c9448%2FErlangen_Programme.png?alt=media)

**Example**: Euclidian geometry are **rigid motions**. These are **translations, reflections and rotations** that **preserves** properties such as **angles, distances, areas, parallelism of lines and their intersections**.

**The language of group theory as the language to formalize the notion of symmetry** Group theory was also a shiny mathematical field that was born in the **19th** century and the term group was first used by **E. Galois** (1832).

#### Influence to other fields

These ideas of symmetry on geometry were very profound and it also spilled into other fields, in particular in **physics**.

> Every \[**differentiable**] symmetry of the action of a physical system \[with conservative forces] has a corresponding conservation law. **E. Noether 1918**

E. Noether, she was Klein's colleagues in Gothengen, she **proved** that every differentiable symmetry of the action of a physical system **has a corresponding conservation law**. By all means this was a really **stunning result** because beforehand you had to do **very detailed** materials **experimental** observations to discover fundamental laws such as **conservation of energy**. So you would measure the energy in many experiments and you will see that up some small **errors** the energy remains the same. It was an **empirical result** not coming from anywhere. **Noether's theorem** established is that the conservation of energy emerges from translational symmetry of time. So it's rather **intuitive** idea that the results of an experiment would be the same if you did it yesterday, today or if you do it tomorrow.

#### Gauge Invariance

**H. Weyl** cited his poetic definition of symmetry. He used these ideas to develop the concept of what he called **Gauge Invariance**. It was a principle from which **electromagnetism** could be **derived**. He also speculated that he tried to unify with gravitation

#### Standard model in particle physics

After several decades **non-abelian gauge theory**, it was finally possible in a theory that was developed by **Yang** & **Mills** in **1954** to provide a **unified framework** that describes all the fundamental forces of nature, with the exception of gravity. This is what is called the standard model in the particle physics and it unifies the description of electromagnetism, weak interactions and strong interactions. All of this using the **language** of **group theory** and **gauge invariance**.

> Unification of electromagnetic and weak forces (modelled with the groups U(1) × SU(2)) and the strong force (based on the group SU(3)) **C. N. Yang** & **R. L. Mills 1954**

> It is only slightly overstating the case to say that Physics is the study of **symmetry**. **P. Anderson 1972**

**Interesting resource on the topic of symmetry in physics** (Roger Penrose) [Fearful Symmetry: The Search for Beauty in Modern Physics](https://www.amazon.com/Fearful-Symmetry-Physics-Princeton-Science-ebook/dp/B0147EPTRI) If you were to distill these thousand plus pages into one word, it would be **symmetry**. He devotes a lot of time to **group theory** to the concepts of **symmetry** and how they are fundamental in physics.

## What does the historical background all have to do with Deep Learning?

The current state of affairs in the field of Deep Learning reminds a lot of the situation of **geometry** in the **19th** century. In the past decade Deep Learning has brought a true revolution in the data science world. It made **possible** many tasks that previously maybe **20 years ago** would be considered nearly **science fiction**. Whether it's **computer vision** that powers **autonomous driving**, **speech recognition** behind every single mobile phone today, **natural language translation** sometimes **trained without supervision**, **playing** intelligent **games** like **go** or doing **science** tasks as solving the **protein folding** problem.

So, on the other hand we **nowadays** have this **zoo** of different **neuronal network architectures** for different types of **data** but very **few unifying principles**. As a consequence it's **difficult** to **understand** the **relations** between different methods and this **inevitably leads** to the **reinvention** and **rebranding** of the same concepts.

![20th Century Zoo of Neuronal Network Architectures](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-a51b9503cd5ab6be6e3b1107984165daad0a2921%2Fzoo_architectures.png?alt=media)

The **same ideas** are presented and published several times in **different communities** under **different names**. Sometimes it also brings unpleasant and bitter fights over **priority**, so this is really an **unhealthy** situation.

### The Erlangen Programme of ML: Geometric Deep Learning

We need some form of **unification** and we want to do it in the spirit of the **Erlangen Programme**. This is what we call **Geometric Deep Learning** and it serves **2 purposes**:

1. **Pedagogical perspective**: It provides a **common** mathematical **framework** to study the most successful neuronal architectures that are currently used ubiquitously in the field of Deep Learning or Deep **representation** Learning.
2. **Constructive procedure to incorporate prior knowledge into neural networks**: Build **future** architectures in a **principled** way.

Question: **What will be after Deep Learning** or maybe beyond learning by **backpropagation**?

Answer: The hope is that these principles will **outlast** their particular implementations.

Question: **When did this theory born?**

![First appearance](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-ff25b2a33cffe1322b1293bd14fd127e5da3e942%2Fbeyond_euclidian_data.png?alt=media)

Answer: The term [Geometric Deep Learning](https://geometricdeeplearning.com) was popularised in a **2017** paper in the **IEEE** *Single Processing* magazine. Co-Authorised by [Michael M. Bronstein](https://twitter.com/mmbronstein?lang=en) and [Joan Bruna](https://arxiv.org/search/cs?searchtype=author\&query=Bruna%2C+J).

Most recently the theory has been extended, to the length of approximately a **book**, and refined with the help of new contributors such as [Taco Cohen](https://tacocohen.wordpress.com) and [Petar Veličković](https://petar-v.com). The new paper is available **online**.

## Motivation

**Where** actually **symmetry** and **how** exactly it is **manifested** in Machine Learning and **why** it is important?

If we consider the **simplest** setting of machine learning, **supervised learning**, this is essentially a **glorified function estimation** problem. **Problem description**: We're given some **unknown** function and we **observe** its **output** on what is called a **training set**. What we try to **do** is to **find** a **function** that **fits** well the **training data** while keeping **generalization** properties. This function **comes** from some **class of functions**, some **hypothesis class**, and this way we try to **predict** outputs on previously **unseen inputs**, the so called the **test set**.

### Learn to discriminate dogs from cats via Supervised Learning

The typical example that is given is **image classification** where you have a **set of dog** and **cat set** images and it's a **binary classification**. This means that given any possible input image, assumed that it contains a cat or a dog, the function would classify them into class of cats or dogs.

![Supervised ML = Function Approximation](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-019ab3840d0b2d7940f661ab41ecc4dfe4112749%2Fsupervised_learning.png?alt=media)

Question: **What happened over the past decade?**

**2 trends** coincided:

1. **Availability of large and high quality data sets**: For example data sets such as [Imagenet](https://www.image-net.org) that has millions of **labeled** images.
2. **Sufficient computational resources**: The continuous improvements and development of **GPUs**, that were thought and originated as **g**raphics hardware, are especially well suitable for general purpose computations that you encounter in deep learning and AI in general (**array programming**).

These two trends have led to the design of **rich function classes** that have the capacity, at least in **theory**, to **interpolate** such large datasets.

### Perceptron

**Neural Networks are a suitable choice to represent functions**.

![Perceptrons, Rosenblatt 1957](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-cf76983e7270d584b0ce0349c19996ee4f9ddbae%2Fperceptron.png?alt=media)

Neural networks of course are **not new at all**. They are at least **70 years** old, so the first works are from the **50s** and with a very **simple** choice of **architecture**, the so called **perceptron**. This type of architecture is probably the earliest and **simplest** neural network.

### Universal Approximation

We can show that if we **connect** just **2 layers** of such networks, **perceptron**, what is called **multi-layer perceptron** or a perceptron with one **hidden layer**, it produces a **dense** class of functions. In other words, we can **approximate any continuous function** or even broader class of functions to **any** desired **accuracy**. We call this **property** universal approximation. It is a very **general** architecture than can **represent** practically **anything**.

![Cybenko 1989; Hornik 1991; Barron 1993; Leshno et al 1993; Maiorov 1999; Pinkus 1999](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-9e8363ed4bd701640fcb49797cb7214c4e067653%2Funiversal_approximation.png?alt=media)

#### Kolmogorov–Arnold representation theorem: Model as a form of Artificial Intelligence

At the second [International Congress of Mathematicians](http://en.wikipedia.org/wiki/International_Congress_of_Mathematicians) in Paris 1900, [Hilbert](http://en.wikipedia.org/wiki/David_Hilbert) presented ten of his [23 problems](http://en.wikipedia.org/wiki/Hilbert's_problems), including the [13th problem](http://en.wikipedia.org/wiki/Hilbert's_thirteenth_problem) about equations of degree seven. He considered the following equation,

$$
x^7+ax^3+bx^2+cx+1=0,
$$

and asked whether its solution $$x(a,b,c)$$, seen as a function of the three parameters a, b and c, can be written as the **composition** of functions of only two variables.

**Hilbert's 13th problem statement**: *Solve 7th degree equation using algebraic (variant: continuous) functions of two parameters*.

**Kolmogorov-Arnold representation theorem**: The model assumes that an output parameter $$z$$ depends on vector input parameters $$x$$. Kolmogorov-Arnold formal model requires the continuity, but we relax this condition and assume that small increments in the inputs are causing small differences in the output. (\_Ref \[4])

$$
z = F(x\_1,x\_2,...,x\_n) \ z : + \Delta z = F(x\_1 : + \Delta x\_1,x\_2 : + \Delta x\_2,...,x\_n : + \Delta x\_n \ z = \sum\_{k=1}^{2n+1}\Phi\_{k}\Bigg\[ \sum\_{j=1}^{n} f\_{k,j}(x\_{j})\Bigg]
$$

The model itself is a set of unspecified functions.

Let $$f : \[0,1]^{d} \mapsto \mathbb{R}$$ be continuous. There exists **univariate** continuous functions $$g\_{q}, \Psi\_{p,q}$$, such that $$f(x\_{1},...,x\_{d}) = \sum\_{q=0}^{2d}g\_{q} \Big ( \Psi\_{p,q}(x\_{p})\Big)$$ Nowadays there are better representations than the one exposed here. There are **3** main reasons why there are no explicit formulas for $$g\_{q}, \Psi\_{p,q}$$.

1. The proofs of these theorems are **non-constructive**.
2. The **outer function**, $$g\_{q}$$ highly depends on $$f$$ so we can't just choose one.
3. The **inner function** $$\Psi\_{p,q}$$ is continuous.

***NOTE***: THE THEOREM ONLY STATES EXISTENCE OF THESE FUNCTIONS.

This form has **similarities** with a **2-layered** neural network but this is highly debated.

**e.g:** The function,

$$
f\_{1}(x\_1,x\_2) = x\_{1}^{2} : + x\_{2}
$$

can be decomposed as in the theorem since it's a summation of continuous univariate functions. Let

$$
g\_{1}(x) = x :, \Psi\_{1,1}(x\_1) = x\_{1}^2 :, \Psi\_{2,1}(x\_2) = x\_2
$$

and for all the other functions to be set with $$g\_q = \Psi\_{p,q} =0$$.

### The Curse of Dimensionality

![](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-e1ade808d39cee25b95148b0d28fdf8b26fa43e5%2Ffunction_curse.png?alt=media)

This is a very well studied problem in **approximation theory** and in **low dimensions** we have a lot of results that tell us exactly how the **error** will behave. An example of these behaviours could be **how** the error would occur if we **sample** our data in a certain way. It has been studied very extensively over the **past century** or even more but the situation appears to be **absolutely** different in **high dimensions**. Even if we pick a very nice class of functions of the so called **Lipschitz continuous functions**.

![Lipschitz functions: superposition of gaussian blobs put in quadrants of a unit cube](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-7e88cdd7870644311c7d9c081687959fdc1e1199%2Fex_lipschitz_continuous_functions.png?alt=media)

What we will find out very quickly that as the **dimensionality** of this **space**, the unit cube, grows then the **number of samples** grow **exponentially**. This phenomenon is colloquially known as the curse of dimensionality. Modern machine learning methods need to operate with data not in two or three dimensions but in **thousands** or even **millions** dimensions. **Images** can serve as an illustrative example of very high dimensional input space.

![Exponential growth explosion](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-b6a9f1e2779e95d2d8280ba4c8f5bb534debacd8%2Fcomplexity_explosion.png?alt=media)

The curse of dimensionality is simply an inevitable God's curse accompanying every machine learning problem. Therefore the previous exposed naive approach to learning is completely impossible to achieve.

This is probably best seen in **computer vision** applications like **image classification**. Since images are well known as high dimensional input space, even for example tiny images from [MNIST data set](http://yann.lecun.com/exdb/mnist/), they are almost thousand dimension. But if we look deeper at this problem, intuitively we see that there is a lot of **structure**.

![MNIST data set example: Number three](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-d40a5a7cc6c2f48098fa0c3665278b89fb6ebc29%2Fimage_three.png?alt=media)

If we treat the input image as a **d-dimensional vector**, to for example the perceptron, we would **destroy** the **input structure**. In this case we are referring to the **local spatial connectivity** between pixels in the image. As a result now if we take the image and **shift** it by just **one pixel**, this vectored input will be very **different** and the neuronal network will need to be shown **a lot of examples** (*data augmentation*) in order to **learn** that **shifted** inputs must be classified in the same way as if they hadn't been applied any **spatial transformation**. **Conclusion**: MUST LEARN SHIFT INVARIANCE FROM DATA

![MNIST data set example: Number three shifted by one single pixel](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-d2cae6cce29f56747604cf037706a6fae6824b4b%2Fthree_shift.png?alt=media)

Question: **How do we learn spatial transformations applied to the inputs?** Answer: Using a standard practice called **data augmentation**. If we don't know how to model specific environments or certain way of telling that the shifted versions of these digits **must** be the **same**, we just **add** such examples to the **training set** in order to **increase** the diversity of known input space. This can be enormously **wasteful**.

### Learning Structure

Question: **If there exists structure and shared information in the input space, how can we exploit that to achieve better learning and understanding?**

![Hubel, Wiesel 1962. LeCun et al. 1989](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-b131551f1bf6060a3a88fc5e6c7c4904d5964421%2FCNN_invention.png?alt=media)

Answer: In the field of **computer vision**, the solution came from a classic work in neuroscience. In particular after the winning work by **Hubel** and **Wiesel** who studied the **organization of the visual cortex**. They showed that brain **neurons** are organised into what they called **local receptive field**. This was enough inspiration to influence a new type of **architectures** with **local shared weights**. Remarkable citation to the **neocognitron** of Fukushima (1980). In his paper from 1980 were already contained many elements of modern deep learning such as for example the **activation function** that he used was already a **ReLU** (**Re**ctified **L**inear **U**nit). This culminated in **C**onvolutional **N**eural **N**etworks (**CNN**) by the pioneering work of **Yann LeCun** (1989). The concept of **weight sharing** across the image effectively partially solved the cursed of dimensionality.

### Beyond Grids

![Graph representation of a caffeine molecule](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-b3c90adf2350c8bd460c731f9ad448d227f5cd36%2Fcafeine_molecule.png?alt=media)

Chemical compounds like this caffeine **molecule** can be represented using a **graph**, where the **nodes** act as the atoms and **edges** as chemical compounds. If we were to apply neural network to this input, for example to **predict** some chemical properties like its binding energy or some other property, we can again **parse** it into a **vector** and give it to the neural network as its input. But now we can see that we can **rearrange** the **nodes** in any way basically. We can effectively **permute** the nodes of our graph because in graphs and like images we don't have a particular preferential or default **order** scheme for the nodes.

Molecular graphs appear to be just one example of data with this **irregular non-Euclidean structure** on which we would like to apply deep learning. Another prominent examples are **social networks**. They are tremendously gigantic with hundreds of millions of nodes and billions of edges among their nodes. We can also find different kinds of networks in **biology** such as **interaction networks**, **manifolds** in computer graphics and many other examples of data which could benefit from a more structured and principled mechanism of learning.

### Symmetry Priors

Fortunately we do have **additional structure** that comes from the **geometry** of the input **signal**. We call this structure **symmetry prior**. It's a general and powerful principle that gives us hope dealing with the curse of dimensionality. In the our previous example of the image classification, the input image is **not** just a **d-dimensional** vector, it's a **signal** defined on some **geometric domain**, denote here by $$\Omega$$. In this example the input space is defined by a **two-dimensional grid** and we denote the **space of signals** by $$\mathcal{X}(\Omega)$$.

![](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-e844ea2d8f32cbf8fe7cd67552c88de0f4eead02%2Fsymmetry_prior_1.png?alt=media)

The structure of the domain is captured by a **symmetry group** that is denoted by $$\boldsymbol{\mathfrak{G}}$$, in this case it's the group of **two-dimensional translations** that acts on the points of $$\Omega$$. For now, and without any further detailed explanation about groups, we need to understand that we have the **domain** and we have a **group** that **acts** on it, so it describes its **structure (symmetry)**. The action of the group on points on the domain is manifested on signals defined on this domain to what is called the **group representation**, denoted by $$ho$$. In the case of this example, pursuing an effort of further clarification, the representation is a **matrix** that acts on the dimensional vectors. It's a $$d \times d$$ matrix that in this example is simply a **translation** or **shift operator**.

![](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-3bfd4a021fc3889147bd9ee9effa3078d35d4f46%2Fsymmetry_prior_2.png?alt=media)

#### Invariant functions: Image Classification

The geometric structure of the domain that underlies the input signal **affects** the **functions** that we define on the **signals** and try to **learn**. Let the function here be denoted by $$f$$. We can then have functions that are **unaffected** by the **action** of the group, what we call **invariant functions**.

For example, in the image classification problem **no** matter where for example the **cat** may be **located** in the image, we still want to say that it's a cat what the image contains. This is an example of **shift invariance**.

![Invariant functions: Shift invariance](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-b931d614f2582ab513272fbceb31b265fdd47f78%2Finvariant_functions.png?alt=media)

#### Equivariant functions: Image Segmentation

We can also have cases where the function has the **same** input and output **structure**, as it occurs in the **image segmentation** problems, the output will be a pixel wise label mask. In these cases we want the **output** to be **transformed** in exactly the same way as the **input**. We name this class of functions as **equivariant functions**. The following example shows an example of shift equivariance in the scenario of an image segmentation problem.

![Equivariant functions: Shift equivariance](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-55d11153b50bf3e312500568c55ee1e649c373a4%2Fequivariant_functions.png?alt=media)

#### Scale Separation Prior

In some cases we can construct a **multi-scale hierarchy** of domains. For example, continuing with the image classification example of the number three, **coarsing** or stretching the **grid** we now have another version of the grid denoted by $$\Omega'$$. This stretching assimilates **nearby** points on the domain and produces also a hierarchy of **signals spaces** that are related by what is called the **coarse graining operator**, denoted by $$P$$. On this coarse grid we can define a **new coarse scale function**, denoted by $$f'$$, and we can say that our function is **locally stable** if it can be **approximated** as the **composition** of this corresponding operation and the coarse scale function.

![Scale Separation Prior](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-96de8a36080b85e1cace9f57ee608414c60234d4%2Fscale_prior.png?alt=media)

While the original function might **depend** on **long range interactions** on the domain, these **spatial locally stable functions** can possible separate the interaction across **scales**. In our discussed example on the image classification, this corresponds to **applying** a classifier at a **lower resolution** image.

### Geometric Deep Learning Blueprint

These exposed principles can probably **recognize** the majority of popular neural **architectures**. We can apply a **sequence of equivariant** layers that **preserves** the **structure** of the domain, possible followed by an **invariant global polling** layer that **aggregates** everything into a **single** output. In some cases we might also need to have a **hierarchy of domains** by some **coarse procedure** that takes the form of **local polling**.

![Geometric Deep Learning general framework](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-e42d91b09e74151d1cdede369098964de5fab388%2Fgdl_blueprint.png?alt=media)

As this framework acts broadly across the structure of the input domain, it can be applied to any kind of geometric data.

![The “5G” of Geometric Deep Learning](https://2089288439-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6TYDq8btP6sh83NP6rvv%2Fuploads%2Fgit-blob-ff2ce414e81d98d459be0f7512d7c866112f94a7%2F5G.png?alt=media)

The implementation of these principles, in the form of inductive biases, leads to some of the most **popular** architectures that exist today in deep representation learning (**CNN**, **G**raph **N**eural **N**etworks, **Transformers**,**L**ong **S**hort **T**erm **M**emory).

**Conclusion**: Every popular neural network architecture can be **derived** from fundamental principles of **symmetry**, same way as all physics can be derived from respective symmetries. We may refer to this as the **deep learning-physics symmetry correspondence**.

## References

1. [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://arxiv.org/abs/2104.13478)
2. [Michael Bronstein's Youtube channel](https://www.youtube.com/c/MichaelBronsteinGDL/videos)
3. [Kolmogorov-Arnold representation as a system of Generalized Additive Models](https://www.youtube.com/watch?v=eS_k6L638k0\&t=28s)
4. [Why Do Neural Networks Need An Activation Function?](https://blog.ntrlab.com/why-do-neural-networks-need-an-activation-function/)