Difference between revisions of "Information Retrieval and Data Mining"
From Chrome
| (36 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
The following are last minute news you should be aware of ;-) | The following are last minute news you should be aware of ;-) | ||
| + | 06/10/2016: Grades from the 30/09/2016 exam can be found at this [[Media:Grades_160930_IRDM.pdf|link]] | ||
| + | 16/09/2016: Grades from the 12/09/2016 exam can be found at this [[Media:Grades_160912_IRDM.pdf|link]] | ||
| + | 02/08/2016: Updated the past exams list with exams from the 2015/2016 year | ||
| + | 31/07/2016: Grades from the 08/07/2016 exam can be found at this [[Media:Grades_160708_IRDM.pdf|link]] | ||
| + | 14/03/2016: Grades from the 22/02/2016 exam can be found at this [[Media:Grades_160222_IRDM.pdf|link]] | ||
| + | 17/02/2016: Grades from the 05/02/2016 exam can be found at this [[Media:Grades_160205_IRDM.pdf|link]] | ||
| + | 18/01/2015: Schedule revised due to a cut & paste error :-( | ||
| + | 18/01/2015: Schedule revised to avoid the overlap with Operational Research Course | ||
| + | 15/12/2015: Schedule revised until January (Note: on 18/01/2016 two extra hours are borrowed from PAMI) | ||
| + | 09/12/2015: IRDM Exams for the Winter Calls will be on: 05/02/2016 and 22/02/2016 | ||
| + | 04/12/2015: Added slides on Decision Tree Pruning and Classification Rules | ||
| + | 24/11/2015: Moved from 21/12 to 14/12 a lecture by Luca Bondi | ||
| + | 10/11/2015: Moved from 14/12 to 30/11 a lecture by Luca Bondi | ||
| + | 25/10/2015: Added slides on data representation and decision trees | ||
11/10/2015: Added link to Teaching Assistant website to get his material | 11/10/2015: Added link to Teaching Assistant website to get his material | ||
09/10/2015: First edition of IRDM website is out, stay tuned! | 09/10/2015: First edition of IRDM website is out, stay tuned! | ||
| Line 61: | Line 75: | ||
|19/10/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Basic models for Information Retrieval | |19/10/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Basic models for Information Retrieval | ||
|- | |- | ||
| − | |23/10/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |23/10/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Introduction to probability |
|- | |- | ||
|26/10/2015 || Monday || -- || -- || -- || No Lecture Today | |26/10/2015 || Monday || -- || -- || -- || No Lecture Today | ||
|- | |- | ||
| − | |30/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |30/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Entropy and information gain |
|- | |- | ||
|02/11/2015 || Monday || -- || -- || -- || No Lecture Today | |02/11/2015 || Monday || -- || -- || -- || No Lecture Today | ||
|- | |- | ||
| − | |06/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |06/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Decision Trees |
|- | |- | ||
| − | |09/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || | + | |09/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Evaluation of IR systems and Text processing |
|- | |- | ||
| − | |13/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |13/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Decision Trees |
|- | |- | ||
| − | |16/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || | + | |16/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Math background and Advanced IR models |
|- | |- | ||
|20/11/2015 || Friday || -- || -- || -- || No Lecture Today | |20/11/2015 || Friday || -- || -- || -- || No Lecture Today | ||
|- | |- | ||
| − | |23/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || | + | |23/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Advanced IR models, Inverted indexing |
|- | |- | ||
| − | |27/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |27/11/2015 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Classification rules |
|- | |- | ||
| − | |30/11/2015 || Monday || - | + | |30/11/2015 || Monday || 15:15 - 18:15 || V.S8-B || Luca Bondi || Multidimentional indexing |
|- | |- | ||
| − | |04/12/2015 || Friday || 08:15 - 12:15 || V.S8-B || Matteo Matteucci || | + | |04/12/2015 || Friday || 08:15 - 12:15 || V.S8-B || Matteo Matteucci || Sequential covering algorithm |
|- | |- | ||
|07/12/2015 || Monday || - || - || - || No Lecture Today | |07/12/2015 || Monday || - || - || - || No Lecture Today | ||
|- | |- | ||
| − | |11/12/2014 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |11/12/2014 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || Rule pruning by Chi-Square test |
|- | |- | ||
| − | |14/12/2014 || Monday || 15:15 - | + | |14/12/2014 || Monday || 15:15 - 17:15 || V.S8-B || Luca Bondi || Rank aggregation |
|- | |- | ||
| − | |18/12/2015 || Friday || | + | |18/12/2015 || Friday || - || - || - || No Lecture Today |
|- | |- | ||
| − | |21/12/2014 || Monday || | + | |21/12/2014 || Monday || - || - || - || No Lecture Today |
|- | |- | ||
| − | |08/01/2016 || Friday || | + | |08/01/2016 || Friday || - || - || - || No Lecture Today |
|- | |- | ||
| − | |11/01/2016 || Monday || 15:15 - 18:15 || V.S8-B || Matteo Matteucci || | + | |11/01/2016 || Monday || 15:15 - 18:15 || V.S8-B || Matteo Matteucci || Frequent pattern mining and association rules |
|- | |- | ||
| − | |15/01/2016 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | + | |15/01/2016 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || A-priori algorithm and sequential patterns mining |
| + | |- | ||
| + | |18/01/2016 || Monday || 15:15 - 17:15 || V.S8-B || Matteo Matteucci || Markov chains | ||
| + | |- | ||
| + | |22/01/2016 || Friday || 08:15 - 10:15 || V.S8-B || Matteo Matteucci || | ||
|- | |- | ||
|} | |} | ||
| Line 117: | Line 135: | ||
Teacher Lectures: | Teacher Lectures: | ||
* [[Media:IRDM2015-00-Intro.pdf | [2015] Course introduction]]: introductory slides of the course with useful information about the grading, and the course logistics. | * [[Media:IRDM2015-00-Intro.pdf | [2015] Course introduction]]: introductory slides of the course with useful information about the grading, and the course logistics. | ||
| − | * [[Media:IRDM2015-01-DataMining.pdf | [ | + | * [[Media:IRDM2015-01-DataMining.pdf | [2015] Data Mining]]: Data Mining introduction, historical perspective and related topics, the data mining process. |
| + | * [[Media:IRDM2015-02-DataRepresentation.pdf | [2015] Data Representation]]: Data representation types and issues, file formats and public datasets. | ||
| + | * [[Media:IRDM2015-03-DecisionTrees.pdf | [2015] Decision Trees]]: Recap on probability, condiational entropy and information gain, decision trees. | ||
| + | * [[Media:IRDM2015-04-DecisionTreesPruning.pdf | [2015] Decision Trees Pruning]]: Decision trees and overfitting, subtree rising, and subtree replacement. | ||
| + | * [[Media:IRDM2015-05-ClassificationRules.pdf | [2015] Classification Rules]]: Classification rules, direct and indirect methods, decision tree pruning classification rules pruning. | ||
| + | * [[Media:IRDM2015-06-AssociationRulesBasics.pdf | [2015] Learning Association Rules]]: Association rules, A-Priori algorithms, sequence mining and GSP algorithm. | ||
| + | * [[Media:IRDM2015-07-MarkovModels.pdf | [2015] Markov Models]]: Markov Chains and PageRank | ||
| + | * [[Media:IRDM2015-08-LinkAnalysis.pdf | [2015] Link Analysis]]: Link Analysis, centrality, PageRank | ||
| − | Teaching Assistand Lectures can be found directly on [http://home.deib.polimi.it/lbondi/irdm-201516/index. | + | |
| + | |||
| + | Teaching Assistand Lectures can be found directly on [http://home.deib.polimi.it/lbondi/irdm-201516/index.html Luca Bondi website]. | ||
===Additional Resources=== | ===Additional Resources=== | ||
| − | Papers | + | Papers, links, and stuff useful to integrate the textbook |
| − | + | * [[Media:chisquared.png | [2015] Chi-Square]]: Chi-Square table to be used for the exercises | |
<!-- | <!-- | ||
| Line 137: | Line 164: | ||
Since 2015/2016 the course has changed teacher, this might have an impact on the exams format as well. Some examples from the past year can be found here, please expect differences: | Since 2015/2016 the course has changed teacher, this might have an impact on the exams format as well. Some examples from the past year can be found here, please expect differences: | ||
| − | * [[Media: | + | * [[Media:2016_07_08_IRDM.pdf |08/07/2016 Exam]] |
| − | * [[Media: | + | * [[Media:2016_02_22_IRDM.pdf |22/02/2016 Exam]] |
| − | * [[Media: | + | * [[Media:2016_02_05_IRDM.pdf |05/02/2016 Exam]] |
| − | + | ||
* [[Media:2015_09_29_IRDM.pdf |29/09/2015 Exam]] | * [[Media:2015_09_29_IRDM.pdf |29/09/2015 Exam]] | ||
| + | * [[Media:2015_09_15_IRDM.pdf |15/09/2015 Exam]] | ||
| + | * [[Media:2015_07_08_IRDM.pdf |08/07/2015 Exam]] | ||
| + | * [[Media:2015_02_25_IRDM.pdf |25/02/2015 Exam]] | ||
| + | * [[Media:2015_02_11_IRDM.pdf |11/02/2015 Exam]] | ||
===Online Resources=== | ===Online Resources=== | ||
Latest revision as of 08:20, 6 October 2016
The following are last minute news you should be aware of ;-)
06/10/2016: Grades from the 30/09/2016 exam can be found at this link 16/09/2016: Grades from the 12/09/2016 exam can be found at this link 02/08/2016: Updated the past exams list with exams from the 2015/2016 year 31/07/2016: Grades from the 08/07/2016 exam can be found at this link 14/03/2016: Grades from the 22/02/2016 exam can be found at this link 17/02/2016: Grades from the 05/02/2016 exam can be found at this link 18/01/2015: Schedule revised due to a cut & paste error :-( 18/01/2015: Schedule revised to avoid the overlap with Operational Research Course 15/12/2015: Schedule revised until January (Note: on 18/01/2016 two extra hours are borrowed from PAMI) 09/12/2015: IRDM Exams for the Winter Calls will be on: 05/02/2016 and 22/02/2016 04/12/2015: Added slides on Decision Tree Pruning and Classification Rules 24/11/2015: Moved from 21/12 to 14/12 a lecture by Luca Bondi 10/11/2015: Moved from 14/12 to 30/11 a lecture by Luca Bondi 25/10/2015: Added slides on data representation and decision trees 11/10/2015: Added link to Teaching Assistant website to get his material 09/10/2015: First edition of IRDM website is out, stay tuned!
Contents
Course Aim & Organization
The course covers tools and systems adopted to handle big data, e.g., large collections of textual data. In the first part, the course focuses on the analysis of information embedded in large collections, using tools that range from decision trees, classification rules, association rules, graph-based link analysis. The second part of the course covers the efficient retrieval of information, discussing the algorithms and data structures adopted to enable answering keyword based queries, as well as indexing methods to enable fast search.
Teachers
The course is composed by a blending of lectures and exercises by the course teacher and a teaching assistant.
- Matteo Matteucci: the course teacher
- Luca Bondi: the teaching assistant
Course Program
The course outline is:
- Data mining
- The Data Mining process
- Decision Trees and Decision Rules
- Rule Induction Methods
- Association Rules
- Frequent Itemset Analysis
- Web information retrieval
- Web modelling and crawling
- Graph-based retrieval models (PageRank, HITS)
- Text-based information retrieval
- IR models (Boolean models, vector space models, probabilistic models)
- Evaluation of IR systems
- Text processing
- Advanced IR models (Latent Semantic Indexing)
- Indexing
- Inverted indexing
- Multidimensional indexing
- Rank aggregation
Detailed course schedule
A detailed schedule of the course can be found here; topics are just indicative while days and teachers are correct up to some last minute change (I will notify you by email). Please note that not all days we have lectures!!
Note: Lecture timetable interpretation * On Mondays, in V.08, starts at 15:30 (quarto d'ora accademico), ends at 18:15 * On Fridays, in V.08, starts at 08:30 (quarto d'ora accademico), ends at 10:15
| Date | Day | Time | Room | Teacher | Topic |
| 05/10/2015 | Monday | 15:15 - 18:15 | V08 | Matteo Matteucci | Course Introduction |
| 09/10/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | The Data Mining Process |
| 12/10/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Introduction to Information Retrieval |
| 16/10/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Data representation and coding |
| 19/10/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Basic models for Information Retrieval |
| 23/10/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Introduction to probability |
| 26/10/2015 | Monday | -- | -- | -- | No Lecture Today |
| 30/11/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Entropy and information gain |
| 02/11/2015 | Monday | -- | -- | -- | No Lecture Today |
| 06/11/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Decision Trees |
| 09/11/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Evaluation of IR systems and Text processing |
| 13/11/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Decision Trees |
| 16/11/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Math background and Advanced IR models |
| 20/11/2015 | Friday | -- | -- | -- | No Lecture Today |
| 23/11/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Advanced IR models, Inverted indexing |
| 27/11/2015 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Classification rules |
| 30/11/2015 | Monday | 15:15 - 18:15 | V.S8-B | Luca Bondi | Multidimentional indexing |
| 04/12/2015 | Friday | 08:15 - 12:15 | V.S8-B | Matteo Matteucci | Sequential covering algorithm |
| 07/12/2015 | Monday | - | - | - | No Lecture Today |
| 11/12/2014 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | Rule pruning by Chi-Square test |
| 14/12/2014 | Monday | 15:15 - 17:15 | V.S8-B | Luca Bondi | Rank aggregation |
| 18/12/2015 | Friday | - | - | - | No Lecture Today |
| 21/12/2014 | Monday | - | - | - | No Lecture Today |
| 08/01/2016 | Friday | - | - | - | No Lecture Today |
| 11/01/2016 | Monday | 15:15 - 18:15 | V.S8-B | Matteo Matteucci | Frequent pattern mining and association rules |
| 15/01/2016 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci | A-priori algorithm and sequential patterns mining |
| 18/01/2016 | Monday | 15:15 - 17:15 | V.S8-B | Matteo Matteucci | Markov chains |
| 22/01/2016 | Friday | 08:15 - 10:15 | V.S8-B | Matteo Matteucci |
Course Evaluation
Course evaluation is through a written exam covering the whole program
Teaching Material
Teachers Slides
In the following you can find the lecture slides used by the teacher and the teaching assistants during classes.
Teacher Lectures:
- [2015] Course introduction: introductory slides of the course with useful information about the grading, and the course logistics.
- [2015] Data Mining: Data Mining introduction, historical perspective and related topics, the data mining process.
- [2015] Data Representation: Data representation types and issues, file formats and public datasets.
- [2015] Decision Trees: Recap on probability, condiational entropy and information gain, decision trees.
- [2015] Decision Trees Pruning: Decision trees and overfitting, subtree rising, and subtree replacement.
- [2015] Classification Rules: Classification rules, direct and indirect methods, decision tree pruning classification rules pruning.
- [2015] Learning Association Rules: Association rules, A-Priori algorithms, sequence mining and GSP algorithm.
- [2015] Markov Models: Markov Chains and PageRank
- [2015] Link Analysis: Link Analysis, centrality, PageRank
Teaching Assistand Lectures can be found directly on Luca Bondi website.
Additional Resources
Papers, links, and stuff useful to integrate the textbook
- [2015] Chi-Square: Chi-Square table to be used for the exercises
Past Exams and Sample Questions
Since 2015/2016 the course has changed teacher, this might have an impact on the exams format as well. Some examples from the past year can be found here, please expect differences:
Online Resources
The following are links to online sources which might be useful to complement the material above
* TBC
![[2015] Chi-Square](/images/f/f2/Chisquared.png){kind=link}