A

1

Activation Function

Function applied to the weighted sum of inputs in a neural network, determining whether a neuron "fires" or not. (e.g., ReLU, Sigmoid)

2

Adaptive Learning Rate Optimization (ALRO)

Dynamically adjusting the learning rate during training for better convergence.

3

Artificial General Intelligence (AGI)

This is a type of artificial intelligence that has the ability to understand, learn, and apply knowledge across a wide range of tasks, much like a human. While AGI is still largely theoretical, advancements in LLMs are seen as steps towards this goal.

4

Attention-based Mechanisms

Beyond self-attention, explore encoder-decoder attention, multi-head attention, and masked attention for understanding different relationships within the input data.

5

Attention bias

Modifying the attention mechanism to focus on specific parts of the input, potentially mitigating bias.

6

Attention Mechanism

An attention mechanism in large language models allows them to focus on specific parts of input data while processing information, enhancing their performance on various natural language processing tasks.

B

1

Backpropagation

Algorithm for training neural networks, calculating how errors propagate through the network and adjusting parameters accordingly.

2

Batch Normalization

Technique for stabilizing training by normalizing the activations of each mini-batch during training.

3

Batch Size

Number of training examples processed simultaneously during training.

4

BERT (Bidirectional Encoder Representations from Transformers)

BERT is a pre-trained large language model developed by Google, based on the Transformer architecture, and designed to understand bidirectional contextual relationships in text.

5

BERT Score

A newer metric measuring text generation quality based on precision, recall, and F1 score

6

Bias Detection and Mitigation Technique

Identifying and addressing biases present in training data or the LLM itself.

7

BLEU-4

Variation of BLEU metric focusing on n-grams of size 4, commonly used for text generation quality evaluation.

c

1

Contextual Embeddings

Contextual embeddings in large language models represent words or phrases based on their context in a sentence, capturing nuances and meaning variations

2

Corpus-level Evaluation

Evaluating the LLM on a complete dataset to assess overall performance.

3

Counterfactual Reasoning Evaluation

Assessing the LLM's ability to generate different potential outcomes for a given scenario

4

Cross-Entropy Loss

Common loss function for classification tasks, measuring the difference between the predicted and true probability distribution.

5

Curriculum Learning

Gradually increasing the difficulty of training data as the LLM progresses.

6

Counterfactual Explanations

Analyzing how changing specific inputs to the LLM would affect its outputs, offering insights into its decision-making process.

D

1

Data Augmentation

Artificially increasing the size and diversity of training data through techniques like synonym replacement, back-translation, and paraphrasing.

2

Deep Learning

Deep learning is a subset of machine learning that employs neural networks with multiple layers to learn data representations and perform complex tasks.

3

Diffusion Models

Generative models inspired by physical processes like diffusion, offering advantages in text and image generation.

4

Dropout

Randomly dropping out neurons during training to prevent overfitting and improve generalization

E

1

Early Stopping

Technique to prevent overfitting by stopping training when validation loss starts to increase.

2

Error Analysis

Identifying and analyzing common errors made by the LLM to improve performance

3

Embeddings

Embeddings in large language models are vector representations of words or phrases that capture semantic relationships and are used to process textual data effectively.

4

Emergent Capabilities

These are capabilities of LLMs that were not explicitly trained for but emerge as a result of the model’s ability to generalize from its training data

5

Encoder-Decoder Architecture

A common architecture where separate modules process the input (encoder) and generate the output (decoder).

6

Explainable AI (XAI)

Making the LLM's decision-making process transparent and understandable.

7

Explainability Methods (e.g., LIME, SHAP)

Providing insights into how the LLM arrives at its outputs.

8

Extrinsic Evaluation

Evaluating the LLM's performance on specific tasks like text summarization or question answering.

F

1

Factual language modeling

Training LLMs to be more factual and grounded in real-world knowledge

2

Fairness, Accountability, and Transparency (FAT)

Principles for ensuring that LLMs are fair, accountable, and transparent in their decisions and outputs.

3

Few-shot learning

Ability of the LLM to learn new tasks with only a few examples.

4

Fine-Tuning

Fine-tuning is the process of further training a pre-trained large language model on specific tasks or datasets to adapt it for domain-specific applications.

G

1

Gated Recurrent Units (GRUs)

Variant of recurrent neural networks with gating mechanisms for improved memory and information flow

2

Gradient Clipping

Limiting the magnitude of gradients to avoid exploding gradients and unstable training.

3

Generalizability

Ability of the LLM to perform well on unseen data not encountered during training.

4

Generative Adversarial Networks (GANs)

Two models competing to generate realistic data and distinguish real from generated data, sometimes used in conjunction with LLMs.

5

GPT (Generative Pre-trained Transformer)

GPT is a series of large language models developed by OpenAI, based on the Transformer architecture, designed for various natural language processing tasks.

H

1

Hallucinations

In the context of LLMs, hallucinations refer to instances where the model generates non-factual but seemingly plausible responses. This can occur due to erroneous encoding/decoding by the transformer or divergences in the training data

2

Hugging Face

Hugging Face is a popular platform that provides pre-trained language models and tools to use and fine-tune large language models.

3

Human-Centered Evaluation

Moving beyond metrics to consider ethical, societal, and user-centric aspects of LLM performance and impact

4

Human-in-the-Loop Evaluation

Incorporating human judgment alongside automated metrics for a more comprehensive evaluation.

5

Hyperparameters

Parameters outside the LLM architecture that control the training process (e.g., learning rate, batch size).

I

1

Input Embeddings

Input embeddings in large language models convert textual data into numerical representations, making it compatible with neural networks for further processing.

2

Intrinsic Evaluation

Evaluating the LLM's internal properties like coherence and grammatical correctness.

J

1

Joint Embeddings

Joint embeddings in large language models learn shared representations for multiple data types, such as text and images, enabling multimodal applications.

K

1

Knowledge Augmentation

Incorporating external knowledge sources like knowledge graphs into the training process.

2

Knowledge Distillation

Knowledge distillation is a technique in which a large language model transfers its knowledge to a smaller model to achieve similar performance with reduced computational resources.

L

1

Labeling Bias

Biases present in the labels used to train supervised learning models.

2

Language Modeling

Language modeling is a task in which large language models learn to predict the probability of a word or token given its context within a sequence of text.

3

Latent Ability

This refers to the potential capabilities of LLMs that may not be immediately apparent but can emerge with further exploration and application

4

Layer Normalization

Normalizing the outputs of each layer in the LLM to stabilize training and improve performance.

5

Long Short-Term Memory (LSTM)

Recurrent neural network architecture with special memory cells for handling long-term dependencies in sequential data.

M

1

Masked Language Model (MLM)

A type of language model where some words in the input are masked and the model is trained to predict those masked words

2

Memory Mechanism

Memory mechanisms in large language models enable them to store and retrieve information efficiently, enhancing their ability to handle long-range dependencies.

3

Memory Networks

Neural networks with external memory modules for storing and retrieving information beyond their internal state.

4

Meta-learning

Learning how to learn, improving future learning tasks for the LLM.

5

Mixed Precision Training

Using a combination of different data types (e.g., float16 and float32) for faster and more efficient training.

6

Modularization

Breaking down the LLM into smaller, independent modules for easier interpretability and adaptation.

7

Model Agnostic Explanations (MAE)

Techniques like LIME and SHAP that work across different LLM architectures to provide explanations.

8

Model Explainability (e.g., LIME, SHAP)

Techniques to provide insights into how the LLM arrives at its outputs.

9

Multi-head Attention

Allowing the LLM to attend to different parts of the input simultaneously with multiple "heads."

10

Multimodal Embedding

Representing information from different modalities (text, images, audio) in a unified way for enhanced understanding and generation.

11

Multimodality

Integrating information from various modalities like text, images, and audio to enhance understanding and generation capabilities.

N

1

N-gram

An n-gram is a contiguous sequence of n items from a given text sample. Here, the model assumes that the probability of the next word in a sequence depends only on a fixed-size window of previous words

2

Neural Language Models

Neural language models are based on neural networks, a computing system inspired by biological neural networks. These models make use of continuous representations or embeddings of words to make their predictions

3

Natural Language Processing (NLP)

An interdisciplinary subfield of linguistics, computer science, and artificial intelligence. Its goal is for a computer to be able to understand texts and other media in their natural languages, including their contextual nuances

4

Natural Language Understanding (NLU)

Natural Language Understanding is a subfield of natural language processing that focuses on enabling machines to comprehend and interpret human language.

5

Neural Networks

A computing system inspired by biological neural networks. These models make use of continuous representations or embeddings of words to make their predictions

O

1

OpenAI

OpenAI is an artificial intelligence research organization that has developed various large language models, such as GPT, to advance the field of natural language processing.

P

1

Parameter Sharing

Sharing parameters between different parts of the LLM to improve efficiency and memory usage.

2

Pre-training

Pre-training is the initial phase of training large language models on a vast corpus of text data to capture general language patterns and features.

3

Prompt engineering

Crafting prompts that guide the LLM towards desired outputs, with advanced techniques like prompt chaining and nested prompts

4

Prompt Tuning

Fine-tuning prompts based on the LLM's performance and desired outputs.

Q

1

Quantization

Quantization reduces the precision of numerical representations in large language models to make them more memory-efficient during deployment.

R

1

Recurrent Neural Networks (RNNs

RNNs are a class of neural networks that process sequential data, making them suitable for tasks involving text and time-series data.

2

Representation Learning

Learning representations of data (e.g., words, sentences) that capture their meaning and relationships.

3

Residual Connections

Adding "shortcuts" within the LLM's layers to improve information flow and combat vanishing gradients.

S

1

Second-Order Applications

These are applications of LLMs that go beyond basic text generation and involve more complex tasks. For example, using LLMs to generate complete full-stack applications

2

Self-Attention

Self-attention is a mechanism in large language models where each word attends to all other words in a sentence, capturing contextual dependencies.

3

Self-distillation

A knowledge distillation technique where the LLM learns from its own outputs.

4

Self-Supervised and Semi-Supervised Training

LLMs acquire their abilities by learning statistical relationships from text documents during a computationally intensive self-supervised and semi-supervised training process

T

1

Task-Specific Evaluation

Evaluating the LLM's performance on specific tasks beyond general language understanding metrics.

2

Text Corpus

A text corpus is a language resource consisting of a large and structured set of texts in one or more languages. Text corpus can contain text in one or multiple languages and is often annotated

3

Text-to-Code Generation

This is an application of LLMs where the model generates code based on natural language input. It’s a remarkable demonstration of the capabilities of LLMs

4

Transfer Learning

Transfer learning is the application of knowledge gained from pre-training large language models to improve performance on downstream tasks with limited labeled data.

5

Transformer Architecture

A model architecture that uses self-attention mechanisms and has been widely used in many state-of-the-art models for natural language processing tasks

6

Token Vocabulary

Token vocabulary consists of integers, spanning from zero up to the size of the token vocabulary. New words can always be interpreted as combinations of the tokens and the initial-set uni-grams

7

Tokenization

The process of converting a sequence of text into individual tokens (usually words, sub words, or characters) for processing by a language model

U

1

ULMFiT (Universal Language Model Fine-Tuning)

ULMFiT is a transfer learning approach that fine-tunes a pre-trained language model for various NLP tasks, introduced by Jeremy Howard and Sebastian Ruder.

V

1

Vectorization

Vectorization converts text data into numerical vectors, enabling large language models to process and understand natural language.

2

Vocabulary

The set of all unique tokens in a language model’s training data. The size of the vocabulary can significantly impact a model’s performance and computational requirements

W

1

Word Embeddings

Word embeddings in large language models are dense, low-dimensional representations of words that capture semantic relationships and facilitate downstream tasks.

X

1

XLNet

XLNet is a large language model that extends the Transformer architecture and leverages permutation-based training to handle bidirectional contexts.

Y

1

Yelp Dataset

The Yelp data set is a popular corpus used to train and evaluate large language models for sentiment analysis and other NLP tasks

Z

1

Zero-Shot Learning

Zero-shot learning in large language models refers to their ability to perform tasks they were not explicitly trained for by providing hints or prompts.