CSCI-B 659 Topics in Artificial Intelligence

LING-L 665 Applying Machine Learning Techniques in Computational Linguistics - Neural Networks, Deep Learning for CL/NLP

This is the course page for Topics in Artificial Intelligence by Damir Cavar.

– August 2023 –

Course Material Spring 2023, 2024 (also taught in 2020)

Focusing on:

Natural Language Processing
Data Driven Neural methods (Deep Learning and advanced Word embedding models)
Advanced knowledge-driven methods for Computational Semantics, Reasoning

The new course (Spring 2023) extends the previous discussions and focuses on newest approaches to Natural Language Processing (NLP), Knowledge Graphs and Computational Semantics, using data-driven Deep Learning and knowledge-driven hybrid machine learning approaches.

Course Description

Introduction to major algorithms in Machine Learning (ML) and Natural Language Processing (NLP) as well as applications of these techniques to a wide range of NLP/CL topics. The course includes an advanced introduction to NLP/CL and is focused on advanced NLP algorithms including Deep Learning and Large Language Models. Also considered are applications of ML algorithms to NLP/CL problems.

Introduction

This is a graduate course that focuses on machine learning techniques that are used in Computational Linguistics, Natural Language Processing, Cognitive Modeling.

Machine learning problems in language processing including Large Language Models (LLMs) are rather non-typical for machine learning because natural language includes a significant level of exceptions, parallel and cross-linguistic-level dependencies, or complex connections to other cognitive systems and forms of knowledge. The course will provide an overview of the most important machine learning algorithms, but it will mostly focus on how to apply current popular machine learning to language problems such as parsing, co-reference resolution, morphological analysis, semantic disambiguation and coreference analysis, word sense disambiguation, entity and relation extraction, and linking to general knowledge representations. We will put a strong focus on data-driven deep learning and knowledge-driven graph-based methods for NLP and computational semantics (reasoning, knowledge linking, multi-modal aspects of language processing).

Among others, we will cover word vector representations (embedding models), window-based neural networks, recurrent neural networks, long-short-term-memory models and various forms of recursive, convolutional, or graph neural networks as applied in NLP and Computational Semantics.

We will experiment with and discuss the limitations and problems of popular Deep Learning approaches as applied in the domain of NLP, Computational Semantics, Reasoning, and other aspects of Knowledge processing, and consider new approaches or models addressing Natural Language Understanding.

The course is a series of lectures and hands-on programming exercises.

The course is using material provided by:

Stanford University: CS224n: Natural Language Processing with Deep Learning
The Deep Learning textbook (MIT Press) by Ian Goodfellow, Yoshua Bengio and Aaron Courville
Dan Jurafsky and James H. Martin. Speech and Language Processing (3rd ed. draft)
University of Oxford: Deep Learning for Natural Language Processing: 2016-2017 by Phil Blunsom

These courses are accompanied by videos, slides, research papers, links to supplemental material and tutorials, and other very valuable information. Please use these resources during our course.

Tentative Class Schedule

This schedule is most likely subject to change. All updates will be published on Canvas and communicated to the students via Canvas announcements.

All reading needs to be done before the class. The homework/conspectus/final projects need to be submitted on Canvas latest on the next Saturday following the class.

Week	Topics	Reading and Homework
1	Introduction
	Introduction	Intro: JM Ch 1
2	Vector Semantics and Embeddings	JM Ch 6
	Vector Semantics and Embeddings	JM Ch 6
3	Neural Networks and Neural Language Models	JM Ch 7
	Neural Networks and Neural Language Models	JM Ch 7
4	Neural Networks and Neural Language Models	JM Ch 7
	Sequence Labeling for Parts of Speech and Named Entities	JM Ch 8
5	RNNs and LSTMs	JM Ch 9
	Transformers and Pretrained Language Models	JM Ch 10
6	Fine-tuning and Masked Language Models	JM Ch 11
	Machine Translation, Attention	Neural Machine Translation by Jointly Learning to Align and Translate
		Attention Is All You Need
7	Language Models	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
		RoBERTa: A Robustly Optimized BERT Pretraining Approach
	GPT-2	Language Models are Unsupervised Multitask Learners (code)
8	GPT-3	Language Models are Few-Shot Learners
	Adv. LMs	A Survey on Efficient Training of Transformers
		BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
9	Transformer Efficiency	FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
		Cramming: Training a Language Model on a Single GPU in One Day
	Transformer Efficiency	Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning
		Training Compute-Optimal Large Language Models
	Transformer Efficiency
10	Alignment	Training Language Models to Follow Instructions with Human Feedback
11	Alignment	Constitutional AI: Harmlessness from AI Feedback
	Foundation Models
12	Reinforcement Learning with Human in the Loop	Asynchronous Methods for Deep Reinforcement Learning
		Proximal Policy Optimization Algorithms
		Fine-Tuning Language Models from Human Preferences
	Reinforcement Learning with Human in the Loop	Learning to Summarize from Human Feedback
		Training Language Models to Follow Instructions with Human Feedback
13	Reinforcement Learning with Human in the Loop
	Graphs and Natural Language Semantics
14	Graph Embeddings and LLMs
	Graph Embeddings and LLMs

Prerequisites and Requirements

I expect that you are able or acquire the skills to follow example code in Python (3.x), and provide your own solutions using either Python, Java, C(++), Julia, or Go. If you have no programming experience, follow the different links here and on the mentioned course sites and learn Python and Numpy.

Recommended:

Learn Python; if you have never programmed before, best is start with and learn Python first; I recommend using Python 3.x and not any older Python that might be pre-installed on your system
You will find numerous textbooks and introductions to Java and C(++) on IU’s online library pages, focus on modern Java (newer than 11, ideally version 13 or better) and on moden C++ (either C++11 or newer)
Install and learn about TensorFlow and Pytorch
Look into the documentation of word and graph embeddings
Refresh your knowledge of Calculus and Linear Algebra
Update your knowledge of Probability Theory
Refresh your knowledge of common Machine Learning approaches.
Familiarize yourself with common Linguistic concepts and theories, in particular lexical properties, syntax, semantics, speech; for basic introductions consult Jurafsky and Martin (2017, draft, 3rd ed.).

Work through all the relevant Jupyter notebooks at: Python tutorials for NLP, ML, AI

Literature

I do not require any textbook, I recommend the following:

Dan Jurafsky and James H. Martin. Speech and Language Processing (3rd ed. draft) (available online)
Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing. (download paper)
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press. (read online, book)

We will read the following papers:

Policies

Academic Integrity

(from the Dean for Academic Standards and Opportunities) As a student at IU, you are expected to adhere to the standards and policies detailed in the Code of Student Rights, Responsibilities, and Conduct (http://www.iu.edu/~code/). When you submit an assignment with your name on it, you are signifying that the work contained therein is all yours, unless otherwise cited or referenced. Any ideas or materials taken from another source for either written or oral use must be fully acknowledged. If you are unsure about the expectations for completing an assignment or taking a test or exam, be sure to seek clarification beforehand. All suspected violations of the Code will be handled according to University policies. Sanctions for academic misconduct may include a failing grade on the assignment, reduction in your final course grade, a failing grade in the course, among other possibilities, and must include a report to the Dean of Students who may impose additional disciplinary sanctions.

Students with Disabilities

Students who need an accommodation based on the impact of a disability should contact me to arrange an appointment as soon as possible to discuss the course format, to anticipate needs, and to explore potential accommodations.

I rely on Disability Services for Students for assistance in verifying the need for accommodations and developing accommodation strategies. Students who have not previously contacted Disability Services are encouraged to do so (812-855-7578; http://www.indiana.edu/~iubdss/).

CAPS

One benefit of a school like IU is that there are many, many resources available to you. School and life can be intense at times, and if your academic responsibilities or other personal concerns are distracting or weighing on you this semester, I encourage you to contact Counseling and Psychological Services (CAPS) (812) 855-5711, http://healthcenter.indiana.edu/counseling/). The people there can be a resource and a source of support, not just in times of crisis, but also when you need an extra ear or a little extra support. I’m happy to be a listening ear, as well, but I have no counseling training and the folks at CAPS do. Note, too, that I am required to report certain things (e.g., reports of sexual assault, suicidal thoughts).

Note Selling

Several commercial services have approached students regarding selling class notes/study guides to their classmates. Selling the instructor’s notes/study guides in this course is not permitted. Violations of this policy will be reported to the Dean of Students as academic misconduct (violation of course rules). Sanctions for academic misconduct may include a failing grade on the assignment for which the notes/study guides are being sold, a reduction in your final course grade, a failing grade in the course, among other possibilities. Additionally, you should know that selling a faculty member’s notes/study guides individually or on behalf of one of these services using IU email, or via Canvas may also constitute a violation of IU information technology and IU intellectual property policies and additional consequences may result.

Sexual Misconduct Policies at IU

As your instructor, one of my responsibilities is to create a positive learning environment for all students. Title IX and IU’s Sexual Misconduct Policy prohibit sexual misconduct in any form, including sexual harassment, sexual assault, stalking, and dating and domestic violence. If you have experienced sexual misconduct, or know someone who has, the University can help. If you are seeking help and would like to speak to someone confidentially, you can make an appointment with:

The Sexual Assault Crisis Services (SACS) at (812) 855-8900 (counseling services)
Confidential Victim Advocates (CVA) at (812) 856-2469 (advocacy and advice services)
IU Health Center at (812) 855-4011 (health and medical services)

It is also important that you know that Title IX and University policy require me to share any information brought to my attention about potential sexual misconduct, with the campus Deputy Title IX Coordinator or IU’s Title IX Coordinator. In that event, those individuals will work to ensure that appropriate measures are taken and resources are made available. Protecting student privacy is of utmost concern, and information will only be shared with those that need to know to ensure the University can respond and assist.

I encourage you to visit stopsexualviolence.iu.edu to learn more.

Disclaimer

This syllabus is subject to change and likely will change. All important changes will be made in writing, with ample time for adjustment.

Old Material

An old syllabus from Spring 2018 is available online.

Old material (including topic presentations, project presentations and reports, code examples) is available on the 2018 course page.