8. Neural Networks

Deep Neural Networks has revolutionalized wide areas of artificial intelligence. Applications like object recognition, face recognition, semantic segmentation, speech recognition, etc. achieve a significantly improved accuracy, since they are implemented as Deep Neural Networks. This is particularly true for NLP applications such as text classification, automatic translation, intent recognition, named entity recognition and so on. The invention of Transformers - a particular type of Deep Neural Network - constitutes another revolution in AI: Large Language Models (LLMs). chatGPT, Llama3, Mistral and other LLMs show outstanding performance in tasks like text generation, text-summarisation, question-answering, chatbots, information retrieval and the like.

In this chapter first the basics of conventional neural networks (Single- and Multi-Layer Perceptron) and Convolutional Neural Networks (CNNs) are repeated in Neural Networks Introduction and Convolutional Neural Networks, respectively. Since these two Neural Network types have already been introduced in the Artificial Intelligence lecture, they are not explained in detail in this lecture. Students who do not feel confident in these topics are advised to repeat the content themselves.

In Recurrent Neural Networks different types of recurrent neural networks (RNNs) are described. Until the advent of Transformers RNNs have been the most relevant neural network type for many NLP tasks, such as sentence classification, Language Modelling (next-word-prediction) or automatic translation.

Implementations of CNNs and LSTMs for document classification are shown in IMDB Movie Review classification.

Currently by far the most important Neural Network type for NLP, in particular for LLMs is the Transformer, which is introduced in Sequence-To-Sequence, Attention, Transformer. Popular Transformer types like BERT and GPT-X are also explained in this section. A chatBot like chatGPT does not only consist of a LLM, like GPT-3, but of a LLM, which is fine-tuned with an Reinforcement Learning with Human Feedback (RLHF)) approach. RLHF in the context of chatGPT is subject of section chatGPT.