Artificial Neural Networks and Deep Learning

The following are last-minute news you should be aware of ;-)

06/03/2021: Grades for the 2021/2021 homeworks are available here!
24/11/2021: Lecture on 09/12/2021 moved to 16/12/2021
24/11/2021: Change of classroom on the 15/12 from T2.1 to B4.3
12/11/2021: Here is the link to the First Homework!!! (read the registration rules, they have been updated)
12/11/2021: First homework is coming out! Register to submit your solutions!
07/11/2021: We have restructured the notebooks from the labs, please check the new organization and material
13/10/2021: Final grade for 2020/2021 year are here
30/09/2021: Video and slides updated + notebooks published
24/09/2021: Removed last year practicals, this year they will be different!!! Python crash course moved at the end of the material
22/09/2021: Added colab crash course on Python 3 in the material session
18/09/2021: We skip exercising on 22/09 and added exercising on 15/12 I updated the schedule!
16/09/2021: Updated material on the first lectures + added reference to python tutorials in the material section
15/09/2021: Lectures start today!
14/09/2021: Website under maintenance ... come back later

Course Aim & Organization

Neural networks are mature, flexible, and powerful non-linear data-driven models that have successfully been applied to solve complex tasks in science and engineering. The advent of the deep learning paradigm, i.e., the use of (neural) network to simultaneously learn an optimal data representation and the corresponding model, has further boosted neural networks and the data-driven paradigm.

Nowadays, deep neural network can outperform traditional hand-crafted algorithms, achieving human performance in solving many complex tasks, such as natural language processing, text modeling, gene expression modeling, and image recognition. The course provides a broad introduction to neural networks (NN), starting from the traditional feedforward (FFNN) and recurrent (RNN) neural networks, till the most successful deep-learning models such as convolutional neural networks (CNN) and long short-term memories (LSTM).

The course major goal is to provide students with the theoretical background and the practical skills to understand and use NN, and at the same time become familiar and with Deep Learning for solving complex engineering problems.

Teachers

The course is composed of a blending of lectures and exercises by the course teachers and a teaching assistant.

• Matteo Matteucci: the course teacher and this is his webex room
• Giacomo Boracchi: the course co-teacher and this is his webex room
• Francesco Lattari: the course teaching assistant and this is his webex room
• Eugenio Lomurno: the course teaching assistant and this is his webex room

Course Program and Syllabus

This goal is pursued in the course by:

• Presenting major theoretical results underpinning NN (e.g., universal approx, vanishing/exploding gradient, etc.)
• Describing the most important algorithms for NN training (e.g., backpropagation, adaptive gradient algorithms, etc.)
• Illustrating the best practices on how to successfully train and use these models (e.g., dropout, data augmentation, etc.)
• Providing an overview of the most successful Deep Learning architectures (e.g., CNNs, sparse and dense autoencoder, LSTMs for sequence to sequence learning, etc.)
• Providing an overview of the most successful applications with particular emphasis on models for solving visual recognition tasks.

We have compiled a detailed syllabus of the course student can use to double check their preparation against before the exam.

• [2020/2021] Course Syllabus: a detailed list of topics covered by the course and which students are expected to know when approaching the exam

Detailed course schedule

A detailed schedule of the course can be found here; topics are just indicative while days and teachers are correct up to some last minute change (I will notify you by email). Please note that not all days we have lectures!!

Note: Lecture timetable interpretation
* On Wednesday, in T.2.1, Team 1, starts at 15:15, ends at 17:00
* On Wednesday, in T.2.1, Team 2, starts at 17:30, ends at 19:15
* On Thursday, in teacher webex room, starts at 16:30, ends at 19:15

Note: Teams division is based on your Codice Persona (and should minimize overlap)
* Team 1: odd Codice Persona
* Team 2: even Codice Persona

For a google calendar you might look here!

Date	Day	Time	Room	Teacher	Topic
15/09/2021	Wednesday	15:15 - 17:00	T.2.1 (Team1 )	Matteo Matteucci	Course Introduction + Deep Learning Intro
15/09/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
16/09/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	From Perceptrons to Feed Forward Neural Networks
22/09/2021	Wednesday	--	--	--	-- No Lecture --
22/09/2021	Wednesday	--	--
23/09/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Feed forward neural networks and Backpropagation
29/09/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Francesco Lattari	KERAS: Numpy, Tensorflow and FNN
29/09/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
30/09/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Error Functions Design
06/10/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Matteo Matteucci	Overfitting, cross-validation, and Early Stopping
06/10/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
07/10/2021	Thursday	---	---	---	No Lectures (Graduation)
13/10/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Francesco Lattari	KERAS: FFNN and Overfitting
13/10/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
14/10/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Training tricks: activation functions, network initialization, and other stuff...
20/10/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Giacomo Boracchi	The Image Classification Problem
20/10/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
21/10/2021	Thursday	16:30 - 19:15	Virtual Room	Giacomo Boracchi	Convolutional Neural Networks
27/10/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Francesco Lattari	KERAS: Convolutional Neural Networks
27/10/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
28/10/2021	Thursday	16:30 - 19:15	Virtual Room	Giacomo Boracchi	Training with data scarcity
03/11/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Francesco Lattari	KERAS: Convolutional Neural Networks
03/11/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
04/11/2021	Thursday	16:30 - 19:15	Virtual Room	Giacomo Boracchi	Famous CNN architectures
10/11/2021	Wednesday	---	---	---	-- No Lecture (Prove in Itinere) --
11/11/2021	Thursday	16:30 - 19:15	Virtual Room	Giacomo Boracchi	Fully Convolutional CNN, CNN for image segmentation (part 1)(part 2)
17/11/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Giacomo Boracchi	GANs
17/11/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
18/11/2021	Thursday	16:30 - 19:15	Virtual Room	Giacomo Boracchi	CNN for localization and detection
24/11/2021	Wednesday	15:15 - 17:00	T.2.1 (Team 1)	Francesco Lattari	KERAS: reconstruction and segmentation
24/11/2021	Wednesday	17:30 - 19:15	T.2.1 (Team 2)
25/11/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Recurrent neural networks + LSTM
01/12/2021	Wednesday	15:15 - 17:00	E.Gatti - ed.20 (Team 1)	Francesco Lattari	KERAS: learning with time series
01/12/2021	Wednesday	17:30 - 19:15	E.Gatti - ed.20 (Team 2)
02/12/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Sequence to sequence learning and Word Embedding
08/12/2021	Wednesday	---	---	---	-- No Lecture (Holiday) --
09/12/2021	Thursday	---	---	---	-- No Lecture ---
15/12/2021	Wednesday	15:15 - 17:00	B.4.3 (Team 1)	Francesco Lattari	KERAS: learning with text
15/12/2021	Wednesday	17:30 - 19:15	B4.3 (Team 2)
16/12/2021	Thursday	16:30 - 19:15	Virtual Room	Matteo Matteucci	Attention Mechanism and Transformer

Course Evaluation

Course evaluation is composed of two parts:

• A written examination covering the whole program graded up to 20/30
• 2 home projects in the form of a "Kaggle style" challenge practicing the topics of the course graded up to 5/30 each

The final score will sum the grade of the written exam and the grade of the home projects. Home projects are not compulsory and they are issued only once a year.

Teaching Material (the textbook)

Lectures will be based on material from different sources, teachers will provide their slides to students as soon they are available. As a general reference you can check the following text, but keep in mind that teachers will not follow it strictly

• Deep Learning. Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 2016.

Regarding the python programming language, we will provide you the basics about numpy and python scripting in case you want some introductory material you can check here

• Python tutorials: these are the official python tutorials, we suggest 3.An Informal Introduction to Python (Numbers, Strings, Lists), 4.More Control Flow Tools (if, for, range, functions), 5.Data Structures (More on lists, Dictionaries), 9.Classes.

Course Slides

Slides from the lectures by Matteo Matteucci

• [2021/2022] Course Introduction: introductory slides of the course with useful information about the course syllabus, grading, and the course logistics.
• [2021/2022] Machine Learning vs Deep Learning: introduction to machine learning paradigms and definition of deep learning with examples
• [2021/2022] From Perceptrons to Feed Forward Neural Networks: the original Perceptron model, Hebbian learning, feed-forward architecture, backpropagation and gradient descent, error functions and maximum likelihood estimation
• [2021/2022] Neural Networks Training: dealing with overfitting (weight decay, early stopping, dropout), vanishing gradient (ReLU and friends), batch normalization
• [2021/2022] Recurrent Neural Networks: learning with sequences, Recurrent Neural Networks, vanishing gradient, Long Short-Term Memories (LSTM), seq2seq model.
• [2021/2022] Word Embedding: neural autoencoders, language models, word embedding, word2vec, glove.
• [2021/2022] Beyond Sequence 2 Sequence Learning: Neural Turing Machines, attention mechanisms, the Transformer.

Slides from the lectures by Giacomo Boracchi are available in his webpage; for your convenience I am giving pointers to the slide here for you (in case you note discrepancies please notify me)

• [2021/2022] The Image Classification Problem
• [2021/2022] Convolutional Neural Networks
• [2021/2022] CNN Parameters and Training with Data Scarcity
• [2021/2022] Fully Convolutional CNN and CNN for Semantic Segmentation
• [2021/2022] CNN for Localization, CNN Explanations and Famous Architectures
• [2021/2022] Autoencoders and Generative Adversarial Networks
• [2021/2022] Object Detction Networks and Metric Learning

Slides from the practicals by Francesco Lattari and Eugenio Lomurno

• Python and convolutions: a Colab notebook with a crash course on Python 3 to prepare for the practicals and extra material about convolutions!
• Feed Forward Neural Networks: NumPy Arrays, TensorFlow2 Tensors, and Feed Forward Neural Network
• Dealing with Overfitting: Hold Out, Early Stopping and Cross-Validation
• Convolutional Neural Networks: 2D Convolutions and their meaning, Convolutional Neural Networks with Keras
• Augmentation and Fine Tuning: Data Augmentation, Transfer Learning, and Fine Tuning
• Autoencoders and Segmentation: Exercises on Autoencoders and Multiclass Segmentation
• Dealing with Time Series: Recurrent Neural Networks, Long Short-Term Memories, and other models.

External Sources

• Deriving Back Propagation on simple RNN/LSTM: a tutorial step by step to derive the backpropagation formulas
• A Recipe for Training Neural Networks: a neat post on Andrej Karpathy blog about do's and don'ts in training neural networks