--------------------------------------------------------------------------------
/templates/content/ai-basics/index.md:
--------------------------------------------------------------------------------
1 | ## AI Basics
2 |
3 | #### What is AI in 2023?
4 |
5 | Artificial intelligence (AI), machine learning (ML), large language models (LLMs) and related technologies and techniques have crossed the chasm from science fiction and niche research domains into widespread awareness and widely adopted products, services, and systems in 2023.
6 |
7 | Although the history of AI research reaches back to the 1950s, the current widespread awareness of AI can be traced to a recent set of products and services using generative AI such as ChatGPT, Stable Diffusion, and the like. In particular, OpenAI's ChatGPT captured the imagination of many by removing many technical barriers to interacting with AI through “natural language” text conversations. As such, ChatGPT allowed individuals to see the power, promise, and pitfalls of such systems, and many heralded the moment as a new epoch in the history of computing.
8 |
9 | The rapid popularity of ChatGPT and other systems meant new attention and investment in the domain. Almost overnight, new applications and companies emerged that sought to make use of these technologies in new ways and new domains. As a result, a myriad of industries are finding new ways to use this technology for better decision-making, automation, and innovation.
10 |
11 | Alongside this expansion in interest and development is the need for individuals in a variety of roles to quickly ramp up their understanding of AI. However, with the increasing complexity of these models, the substantial amount of new things to learn and the extensive list of new libraries being added every single day, onboarding into the state-of-the-art AI world has become challenging for new engineers. While resources exist, many of these resources (and increasingly so) depend on proprietary technologies. Moreover, the state of knowledge is rapidly changing, and existing resources are quickly out-of-date.
12 |
13 | #### Why are people excited about LLMs?
14 |
15 | Large Language Models (LLMs) are AI models that use deep learning techniques to process and understand natural language text. These models have millions or even billions of parameters that allow them to generate human-like language output, making them ideal for tasks such as language translation, natural-sounding chatbots, document and code generation, and more. LLMs have been trained on massive amounts of data, allowing them to identify patterns in language that were previously difficult for computers to comprehend. This has led to breakthroughs in natural language processing, generated output and improved communication between humans and machines.
16 |
17 | #### Why are people concerned about LLMs?
18 |
19 | LLMs are currently used to power chatbots and a wide variety of content tools that can generate text, images, video, and even code. But the very traits that make LLMs so powerful and useful also present important questions that technologists need to consider when developing and deploying them.
20 |
21 | By mimicking human language and creativity, LLMs have the potential to transform or even automate certain tasks or jobs. The mass harvesting of data to train these systems presents largely unresolved challenges to the principles of copyright, fair use, and fair compensation. The tendency of users to view LLM-powered tools as “officious oracles” can lead humans to make flawed or harmful decisions based on the biases and misinformation these systems can produce. LLMs can generate high quality prose that naturally invites consumers to forget that LLMs offer no guarantees of communicating factual or accurate information.
22 |
23 | At Mozilla, we believe that developers should take these risks seriously and cultivate both an understanding and awareness of how their chosen AI technologies behave and impact the world. By their very nature, open source LLMs offer a greater chance to achieve those goals.
24 |
25 | #### What exactly is an LLM?
26 |
27 | At its heart, a Transformer-based Large Language Model (LLM) is essentially a computer program designed to generate text that resembles human-written content. It leverages machine learning techniques, specifically a type of neural network called a Transformer. At a high-level, a Transformer encodes linguistic patterns in the form of statistical relationships between words, and then uses those patterns to generate text. Transformers encode these semantic patterns by ingesting examples of existing text. Now let’s dig a little deeper.
28 |
29 | **Here's how it works:**
30 |
31 |
32 | {% from "partials/image-cards/image-cards.html" import imageCards %}
33 | {% set orientation = "image-top" %}
34 | {% set slides = [
35 | { "alt": "Tokenization Visualization","title": "Tokenization", "description": "
The LLM starts by breaking down the input text, or 'prompt', into smaller pieces known as tokens.
Now comes the real magic of the Transformer architecture: self-attention mechanisms. These allow the model to weigh the importance of different words when predicting the next word in the sentence. Self attention mechanisms allow the model to create context-aware representations of each word.
Finally, the model generates new tokens, and repeats until the final response is complete.
", "image": "/img/ai/carousels/how-llm-works/output.jpg"}
40 | ]
41 | %}
42 | {{imageCards(slides,orientation)}}
43 |
44 |
45 | All these steps involve complex mathematical operations and transformations. But fundamentally, what the model is doing is learning patterns in the data it was trained on, and using those patterns to generate new text that fits within the same patterns.
46 |
47 | So, in essence, a Transformer-based LLM is a cleverly designed pattern recognition system that uses learned associations between words to generate human-like text. It's like having a scribe who has read billions of bits of content and can produce text on any topic in a style or format that mirrors the content it was trained on.
48 |
49 | #### What are the pros & cons of using an LLM?
50 |
51 | Although LLMs have made it possible for computers to process human language in ways that have been previously difficult, if not impossible, they are not without trade-offs:
52 |
53 | - Pros
54 | - **Improved Token Prediction Accuracy**: LLMs are trained on large amounts of human-readable data (e.g. written text, code, and audio) and rely on state-of-the-art techniques to determine patterns within those data. The size and pattern recognition techniques (e.g. Transformers) improve the accuracy of predictions over previous systems.
55 | - **Efficiency**: With LLMs, tasks such as language translation and chatbots can be automated, potentially freeing up time for humans to focus on more complex tasks.
56 | - **Language Generation**: LLMs can generate human-like language output, making them applicable for tasks such as content creation and copywriting.
57 | - **Code Generation**: LLMs can generate code output, making them a powerful addition to developers' toolkits.
58 | - Cons
59 | - **Computational Power Requirements**: Training an LLM requires significant computational power, which can be expensive and time-consuming.
60 | - **Reproducing problematic training data**: Because LLMs are trained on massive amounts of data, they can perpetuate biases and toxicity (like hate speech) that exist in the training data.
61 | - **Lack of Transparency**: The inner workings of an LLM can be difficult to understand, which makes it challenging to identify how it arrived at a particular decision or prediction.
62 | - **LLM hallucinations**: One of the most interesting and controversial aspects of LLMs is their ability to generate realistic language output that they have never been trained on. This phenomenon is known as LLM hallucination, and it has raised concerns about the potential misuse of these models.
63 |
64 | #### What new behaviors do LLMs unlock?
65 |
66 | Large Language Models (LLMs) have unlocked a plethora of new behaviors that were previously impossible for computers to achieve. For example, LLMs can now generate highly convincing human-like text, which has led to the development of more advanced chatbots and virtual assistants. Additionally, LLMs have revolutionized the field of natural language processing by enabling machines to understand and interpret complex human language in ways that were previously impossible. This has opened up new possibilities for automated language translation, content creation, code generation, and even sentiment analysis. With continued advancements in LLM technology, we can expect to see even more exciting developments in the near future.
67 |
68 | #### When I send a Transformer-based LLM a “prompt”, what happens internally in more technical terms?
69 |
70 | 1. **Tokenization**: This is the first step in processing text data. The tokenizer converts raw text into chunks known as 'tokens'. These tokens can represent words, subwords, or even characters depending on the granularity of the tokenization process. For instance, a word-level tokenizer will convert the sentence "I love coding" into 'I', 'love', 'coding'.
71 | 2. **Embedding**: Once the text is tokenized, these tokens are transformed into dense vectors of fixed size in a high-dimensional space through the embedding layer. These embeddings capture semantic information about the words. For example, in this space, 'king' and 'queen' would be closer to each other than 'king' and 'apple'.
72 | 3. **Positional Encoding**: The encoder processes the input sequence into a context-dependent representation that captures the meaning of the text. It uses a Transformer architecture which allows it to pay varying levels of 'attention' to different parts of the input sequence at each step of the encoding process. The output of the encoder is a series of hidden states.
73 | 4. **Encoder**: The encoded input is passed through several layers (the exact number depends on the model architecture). Each layer consists of self-attention mechanisms and feed-forward neural networks. These layers allow the model to consider all parts of the input when generating each part of the output.
74 | 5. **Decoder (if applicable)**: The decoder generates output text token by token based on the hidden states from the encoder. In some models like GPT-3, the decoder is essentially the entire model. It also uses a Transformer architecture and attention mechanism to focus on the relevant parts of the input.
75 | 6. **Attention Mechanism**: This is a key part of both the encoder and decoder in a Transformer. It helps the model dynamically focus on different parts of the input sequence as it processes the data. It computes a weighted sum of input values (or 'query') based on their relevance (or 'attention scores') to the current 'key' value.
76 | 7. **Output Generation**: Finally, the model generates the output text. This is done one token at a time, with the model deciding on the most likely next token based on the current state. The process continues until the model generates a termination token or reaches a preset maximum length.
77 |
78 | Each step in this process involves complex mathematical operations and transformations. But at a high-level, the goal is to convert text into a form that the model can understand, let the model process and generate a response, and then convert that response back into human-readable text.
79 |
80 |
This section will walk through a very specific approach to select, run and evaluate the results of ML models. We're starting with a small Language Model to illustrate this building block. It's highly recommended to follow along inside Colab so you can run the examples. All examples will run within Colab's free T4 GPU constraints.
This section will walk through a very specific approach to select, run and evaluate the results of ML models. We're starting with a small Language Model to illustrate this building block. It's highly recommended to follow along inside Colab so you can run the examples. All examples will run within Colab's free T4 GPU constraints.
4 |
5 | #### First Steps With Language Models
6 |
7 |
8 |
9 | {% include "content/choosing-ml-models/ai-guide-pick-a-model-test-a-model.html" %}
10 |
11 |
12 |
13 | #### Evaluating ML Model Results
14 |
The AI Guide will be kickstarted by Mozilla's team but powered by community contributions. Here's how you can help!
3 |
Why contribute to AI Guide?
4 |
Mozilla champions an open, accessible Internet where people have the reins. We welcome these amazing new AI breakthroughs.
5 |
However, with substantial corporate dollars being invested into AI R&D, it's unclear for engineers new to the scene which paths are the ones that lead to sustainable open software. We've seen this story before.
6 |
Mozilla's efforts in AI are more than just technical - they're a call to action and unity across the currently fragmented open source AI Community.
7 |
Our AI Guide is a living, breathing resource, rooted in collaboration and community input for experts and newcomers alike, and we invite you to build alongside us.
18 |
--------------------------------------------------------------------------------
/templates/content/contributions/index.md:
--------------------------------------------------------------------------------
1 | ## Contribution Guide
2 | The AI Guide will be kickstarted by Mozilla's team but powered by community contributions. Here's how you can help!
3 |
4 | ### Why contribute to AI Guide?
5 | Mozilla champions an open, accessible Internet where people have the reins. We welcome these amazing new AI breakthroughs.
6 |
7 | However, with substantial corporate dollars being invested into AI R&D, it's unclear for engineers new to the scene which paths are the ones that lead to sustainable open software. We've seen this story before.
8 |
9 | Mozilla's efforts in AI are more than just technical - they're a call to action and unity across the currently fragmented open source AI Community.
10 |
11 | Our AI Guide is a living, breathing resource, rooted in collaboration and community input for experts and newcomers alike, and we invite you to build alongside us.
12 |
13 | [Join our community →](https://discord.gg/3egbzTKhdk)
14 |
15 | ### How do I contribute?
16 | Where the Guide needs your help:
17 | * [Submit content changes, typos or errors](https://github.com/mozilla/ai-guide/issues/new?assignees=&labels=topics&projects=&template=topic-coverage.yml) to existing pages
18 | * [Suggest missing or new content](https://github.com/mozilla/ai-guide/issues/new?assignees=&labels=topics&projects=&template=topic-coverage.yml) for the Guide
19 | * Fork the repo and create [new Python notebooks](https://github.com/mozilla/ai-guide), submit a Github PR for review
20 | * [Provide feedback](https://github.com/mozilla/ai-guide/issues/new?assignees=&labels=general&projects=&template=general-feedback.yml) on audience, tone and other content suggestions
21 | * [Share other thoughts](https://forms.gle/eYJ2s6avtCBXMUQH9) on the guide and our approach
22 |
--------------------------------------------------------------------------------
/templates/content/embeddings-and-rag/_sidebar.html:
--------------------------------------------------------------------------------
1 |
--------------------------------------------------------------------------------
/templates/content/embeddings-and-rag/index-content.html:
--------------------------------------------------------------------------------
1 |
Read these two sections first. They will help you understand the current lay of the land, along with what all this new jargon means so that you're not completely lost.
Large language and image/video generation models are opening up new ways to create, plan and execute the things we do at work and home every day.
21 |
So far, onboarding into the state-of-the-art within AI has been challenging for engineers of any level who are new to the AI space. Current onboarding options require piecing-together various solutions in disjointed ways. Most of these solutions are also not default open source, and there are vast amounts of new terminology, entities and libraries to learn.
22 |
As of this moment, the historical pattern of engineers starting and ending projects with primarily open source components is at risk. Foundation models are very expensive to train from scratch, and open source versions have a hard time competing with well resourced for-profit companies. Mozilla is planting a flag in the ground in this space and creating a new hub for open-source AI projects, kickstarting a vibrant open-source community, with clear shared goals and improved open-source visibility for this important new sector.
23 |
Why Mozilla?
24 |
Mozilla has been developing and shipping AI through its products and initiatives for many years. From Common Voice to Firefox Translate and beyond, we see in AI the potential to empower humans, make technology more accessible, and improve lives.
25 |
But like many others, we also see the risk of real-world harm. We don’t have to cast our gazes to some far-flung science fiction future to envision it: people are already experiencing real-world impacts from this technology, right here and now.
26 |
Much will depend on how this technology is developed and deployed, but it will also depend on who guides it. We believe that the open source community has a critical role to play in ensuring that AI develops in a way that is responsible, trustworthy, equitable, and safe. We hope that this guide contributes to mobilizing the open source community to join us in this important work.
35 |
--------------------------------------------------------------------------------
/templates/content/introduction/index.md:
--------------------------------------------------------------------------------
1 | ## Introduction
2 |
3 | #### If you're new to AI
4 | Read these two sections first. They will help you understand the current lay of the land, along with what all this new jargon means so that you're not completely lost.
5 |
9 |
10 | #### Getting started with code
11 | This is the first part of the "Choosing ML Models" section.
12 |
13 |
14 | {% include "content/choosing-ml-models/ai-guide-pick-a-model-test-a-model.html" %}
15 |
16 |
19 |
20 |
21 | #### Why this guide?
22 | Large language and image/video generation models are opening up new ways to create, plan and execute the things we do at work and home every day.
23 |
24 | So far, onboarding into the state-of-the-art within AI has been challenging for engineers of any level who are new to the AI space. Current onboarding options require piecing-together various solutions in disjointed ways. Most of these solutions are also not default open source, and there are vast amounts of new terminology, entities and libraries to learn.
25 |
26 | As of this moment, the historical pattern of engineers starting and ending projects with primarily open source components is at risk. Foundation models are very expensive to train from scratch, and open source versions have a hard time competing with well resourced for-profit companies. Mozilla is planting a flag in the ground in this space and creating a new hub for open-source AI projects, kickstarting a vibrant open-source community, with clear shared goals and improved open-source visibility for this important new sector.
27 |
28 | #### Why Mozilla?
29 | Mozilla has been developing and shipping AI through its products and initiatives for many years. From Common Voice to Firefox Translate and beyond, we see in AI the potential to empower humans, make technology more accessible, and improve lives.
30 |
31 | But like many others, we also see the risk of real-world harm. We don’t have to cast our gazes to some far-flung science fiction future to envision it: people are already experiencing real-world impacts from this technology, right here and now.
32 |
33 | Much will depend on how this technology is developed and deployed, but it will also depend on who guides it. We believe that the open source community has a critical role to play in ensuring that AI develops in a way that is responsible, trustworthy, equitable, and safe. We hope that this guide contributes to mobilizing the open source community to join us in this important work.
34 |
35 |
43 |
--------------------------------------------------------------------------------
/templates/content/llms-101/_params-carousel.html:
--------------------------------------------------------------------------------
1 | {% set slides = [
2 | { "alt": "LLM Model Parameter Visualization",title": "Model", "description": "Different models are trained on different sets of data, which influences their response.", "image": "/img/ai/carousels/llm-01-model.jpg"},
3 | { "alt": "Max Length Parameter Visualization",title": "Max Length", "description": "Setting a limit to the number of tokens.", "image": "/img/ai/carousels/llm-02-maxlength.jpg"},
4 | { "alt": "Temperature Parameter Visualization",title": "Temperature", "description": "Degree of randomness. Setting a high temperature may lead to more creative responses. Setting a low temperature will result in responses that are more literal.", "image": "/img/ai/carousels/llm-03-temp.jpg"},
5 | { "alt": "Top P Parameter Visualization",title": "Top P (Probability)", "description": "Include only the top tokens whose percentage likelihood adds up to the specified “Top P” (e.g. 15%).", "image": "/img/ai/carousels/llm-04-top-p.jpg"},
6 | { "alt": "Frequency Parameter Visualization",title": "Frequency", "description": "The higher this is set, the more that repetition of tokens present in the context will be penalized in suggestions.", "image": "/img/ai/carousels/llm-05-frequency.jpg"}
7 | ]
8 | %}
9 |
10 |
11 | {% for slide in slides %}
12 |
13 |
14 |
15 |
16 |
17 |
{{ slide.title }}
18 |
{{ slide.description }}
19 |
20 |
21 | {% endfor %}
22 |
23 |
24 | {% for slide in slides %}
25 | {{loop.index}}
26 | {% endfor %}
27 |
What's the difference between a "language model" and a "large language model"?
3 |
A "Large Language Model" (LLM) is a type of "Language Model" (LM) with more parameters, which allows it to generate or understand text better. The term 'large' refers to the number of parameters the model has been trained on. Usually, a LLM provides higher quality results than smaller LMs due to its ability to capture more complex patterns in the data.
4 |
What do these numbers mean in the names of models?
5 |
For example: "Vicuna-13B". The name of the model is Vicuna, and it has 13 billion parameters.
6 |
What is a parameter?
7 |
A parameter is a value that the model learns during training. These values are adjusted through a process called backpropagation, which involves calculating the error between the model's predictions and the actual output and adjusting the parameters to minimize this error. The number of parameters in an LLM is typically very large, often numbering in the millions or even billions. These parameters capture the relationships between different words and phrases in language, allowing the model to generate human-like output and make accurate predictions. Without these parameters, a language model would not be able to perform natural language processing tasks at a high level of accuracy.
8 |
What does “training” an ML model mean?
9 |
Training a model involves exposing it to large amounts of data so that it can learn patterns and make accurate predictions. During training, the parameters of the model are adjusted based on the input data and desired output. This process can take a significant amount of time and computational resources, but it is essential for achieving high levels of accuracy in natural language processing tasks.
10 |
How does a typical training run work?
11 |
During a typical training run, the language model is fed with a large amount of text data and corresponding targets. Generally, the targets are “masked” words within a training sequence that the model needs to predict. The model then uses its prediction errors as compared against these inputs to repeatedly update its parameters through an optimization algorithm like stochastic gradient descent (SGD) or Adam.
12 |
During training, the network makes predictions on a batch of input data and calculates the loss, which is a measure of how well the predictions match the actual targets. The optimizer then adjusts the weights of the network using backpropagation to minimize the loss. This process is repeated iteratively for a fixed number of epochs or until a convergence criterion is met. The process of updating the model's parameters continues iteratively until the model reaches a satisfactory level of accuracy on the training set.
13 |
It's worth noting that training a language model (especially a LLM) can be a resource-intensive process, requiring significant computational power and time.
14 |
What is backpropagation?
15 |
Backpropagation is a process used to adjust the parameters of a model during training. It involves calculating the error between the model's predictions and the actual output and adjusting the parameters to minimize this error.
16 |
The process starts by making a forward pass through the neural network, where input data is fed into the model, and output predictions are generated. The difference between the predicted output and the actual targets is then calculated using a loss function. This loss value is then backpropagated through the network, starting from the last layer and moving backward towards the first layer.
17 |
As it moves backward, each layer updates its weights based on how much it contributed to the final loss value. This process continues until all layers have updated their weights, resulting in a new set of parameters that hopefully improve the model's performance on future inputs.
18 |
What does “fine-tuning” a language model mean?
19 |
Fine-tuning a language model involves taking a pre-trained language model and providing additional training using task-specific data sets moderated by humans. This process typically requires less data than training a model from scratch and can be done relatively quickly. Fine-tuning has become increasingly popular in recent years as more pre-trained models have become available.
20 |
“RLHF” approach
21 |
The "Reinforcement Learning From Human Feedback" approach involves incorporating human feedback into machine learning models to improve their accuracy and performance. In the context of natural language processing, this might involve having humans review text generated by a model and provide feedback on its quality, which can then be used to fine-tune the model. This approach is valuable because it can be used to reduce bias and errors and make usage of AI more trustworthy.
22 |
What does "inference" mean?
23 |
Inference refers to the process of using a trained machine learning model to make predictions or decisions based on new input data. In other words, it's the application of a trained model to real-world data in order to obtain useful insights or take action based on those insights. When performing inference with a language model, the model takes in new text as input and generates output text based on what it has learned during training and fine-tuning. Inference is a critical step in the machine learning workflow, as it allows models to be used for practical applications such as chatbots, language translation, and sentiment analysis.
24 |
Is inference computationally expensive?
25 |
YES, especially for larger models with more parameters. To address this, some models use techniques such as Beam search or sampling to generate output text more efficiently. Additionally, some cloud providers offer pre-trained language models that can be accessed via APIs for a fee, which can help reduce the computational burden of running them locally.
26 |
What is a vector?
27 |
In the context of natural language processing, a vector is a list of numbers used to represent a word or phrase in a way that captures its relationship to other words or phrases within a model. A key useful feature of these vectors is that similar things cluster together.
28 |
Each vector in a model consists of the same list length, with every position in the list representing a semantically interesting feature discovered about the word or phrase through statistical analysis. While it's typically difficult to explain what these individual features might mean to a human, quantifying them allows vectors to be mathematically compared as coordinates in a high-dimensional space. In such a space, distance can approximate overall similarity in meaning between the words or phrases from which the vectors were derived.
29 |
Vectors are typically derived using techniques such as word embeddings, which maps each word to a high-dimensional vector based on its co-occurrence with other words in a large corpus of text.
30 |
What is Beam search?
31 |
Beam search is a search algorithm used to generate output sequences from a model during inference. It works by maintaining a set of the top k most likely sequences at each step of the generation process, where k is known as the Beam width. The algorithm then continues generating new tokens for each sequence in the set until all sequences have reached an end-of-sequence token or a maximum length has been reached. At each step, the set of possible sequences is pruned based on their likelihood according to the model's predictions, resulting in a final set of top-k output sequences.
Frequency is a parameter used in language models to control how often a token can be repeated in the generated output. It works by penalizing the model for repeating tokens that have already been used, preventing the model from generating repetitive or nonsensical text. The higher this is set, the more that repetition of tokens present in the context will be penalized in suggestions.
Temperature is a technique used in language models to control the level of randomness and creativity in the generated output during inference. It works by scaling the predicted probability distribution over possible tokens at each step by a temperature parameter, which controls how much the probabilities are 'softened' or spread out.
Lower temperatures result in more conservative and predictable output, while higher temperatures lead to more diverse and creative output. However, setting the temperature too high can also lead to nonsensical or ungrammatical sentences. Finding the optimal temperature for a given task or application often requires experimentation and fine-tuning.
Sampling is another algorithm used to generate output sequences from a model during inference. Unlike Beam search, which generates only the top-k most likely sequences at each step, sampling generates output tokens probabilistically based on the model's predicted probability distribution over all possible tokens at that step. This can lead to more diverse and creative output compared to Beam search, but it can also result in less coherent or grammatical sentences if not properly controlled through techniques such as temperature scaling or nucleus sampling.
50 |
What is 'top_k' sampling?
51 |
Top-k sampling is a method used in language generation where, instead of considering all possible next words in the vocabulary, the model only considers the top 'k' most probable next words.
52 |
This technique helps to focus the model on likely continuations and reduces the chances of generating irrelevant or nonsensical text. It strikes a balance between creativity and coherence by limiting the pool of next word choices, but not so much that the output becomes deterministic.
53 |
What is 'top_p' sampling?
54 |
55 | {% set slide = { "alt": "Top P Parameter Visualization","title": "Top P (Probability)", "description": "
Top-p sampling, also known as nucleus sampling, is a strategy where the model considers only the smallest set of top words whose cumulative probability exceeds a threshold 'p'.
Unlike top-k which considers a fixed number of words, top-p adapts based on the distribution of probabilities for the next word. This makes it more dynamic and flexible. It helps create diverse and sensible text by allowing less probable words to be selected when the most probable ones don't add up to 'p'.
69 |
--------------------------------------------------------------------------------
/templates/content/llms-101/index.md:
--------------------------------------------------------------------------------
1 | ## Language Models 101
2 |
3 | #### What's the difference between a "language model" and a "large language model"?
4 |
5 | A "Large Language Model" (LLM) is a type of "Language Model" (LM) with more parameters, which allows it to generate or understand text better. The term 'large' refers to the number of parameters the model has been trained on. Usually, a LLM provides higher quality results than smaller LMs due to its ability to capture more complex patterns in the data.
6 |
7 | #### What do these numbers mean in the names of models?
8 |
9 | For example: "Vicuna-13B". The name of the model is Vicuna, and it has 13 billion parameters.
10 |
11 | #### What is a parameter?
12 |
13 | A parameter is a value that the model learns during training. These values are adjusted through a process called backpropagation, which involves calculating the error between the model's predictions and the actual output and adjusting the parameters to minimize this error. The number of parameters in an LLM is typically very large, often numbering in the millions or even billions. These parameters capture the relationships between different words and phrases in language, allowing the model to generate human-like output and make accurate predictions. Without these parameters, a language model would not be able to perform natural language processing tasks at a high level of accuracy.
14 |
15 | #### What does “training” an ML model mean?
16 |
17 | Training a model involves exposing it to large amounts of data so that it can learn patterns and make accurate predictions. During training, the parameters of the model are adjusted based on the input data and desired output. This process can take a significant amount of time and computational resources, but it is essential for achieving high levels of accuracy in natural language processing tasks.
18 |
19 | #### How does a typical training run work?
20 |
21 | During a typical training run, the language model is fed with a large amount of text data and corresponding targets. Generally, the targets are “masked” words within a training sequence that the model needs to predict. The model then uses its prediction errors as compared against these inputs to repeatedly update its parameters through an optimization algorithm like stochastic gradient descent (SGD) or Adam.
22 |
23 | During training, the network makes predictions on a batch of input data and calculates the loss, which is a measure of how well the predictions match the actual targets. The optimizer then adjusts the weights of the network using backpropagation to minimize the loss. This process is repeated iteratively for a fixed number of epochs or until a convergence criterion is met. The process of updating the model's parameters continues iteratively until the model reaches a satisfactory level of accuracy on the training set.
24 |
25 | It's worth noting that training a language model (especially a LLM) can be a resource-intensive process, requiring significant computational power and time.
26 |
27 | #### What is backpropagation?
28 |
29 | Backpropagation is a process used to adjust the parameters of a model during training. It involves calculating the error between the model's predictions and the actual output and adjusting the parameters to minimize this error.
30 |
31 | The process starts by making a forward pass through the neural network, where input data is fed into the model, and output predictions are generated. The difference between the predicted output and the actual targets is then calculated using a loss function. This loss value is then backpropagated through the network, starting from the last layer and moving backward towards the first layer.
32 |
33 | As it moves backward, each layer updates its weights based on how much it contributed to the final loss value. This process continues until all layers have updated their weights, resulting in a new set of parameters that hopefully improve the model's performance on future inputs.
34 |
35 | #### What does “fine-tuning” a language model mean?
36 |
37 | Fine-tuning a language model involves taking a pre-trained language model and providing additional training using task-specific data sets moderated by humans. This process typically requires less data than training a model from scratch and can be done relatively quickly. Fine-tuning has become increasingly popular in recent years as more pre-trained models have become available.
38 |
39 | #### “RLHF” approach
40 | The "Reinforcement Learning From Human Feedback" approach involves incorporating human feedback into machine learning models to improve their accuracy and performance. In the context of natural language processing, this might involve having humans review text generated by a model and provide feedback on its quality, which can then be used to fine-tune the model. This approach is valuable because it can be used to reduce bias and errors and make usage of AI more trustworthy.
41 |
42 | #### What does "inference" mean?
43 |
44 | Inference refers to the process of using a trained machine learning model to make predictions or decisions based on new input data. In other words, it's the application of a trained model to real-world data in order to obtain useful insights or take action based on those insights. When performing inference with a language model, the model takes in new text as input and generates output text based on what it has learned during training and fine-tuning. Inference is a critical step in the machine learning workflow, as it allows models to be used for practical applications such as chatbots, language translation, and sentiment analysis.
45 |
46 | #### Is inference computationally expensive?
47 |
48 | YES, especially for larger models with more parameters. To address this, some models use techniques such as Beam search or sampling to generate output text more efficiently. Additionally, some cloud providers offer pre-trained language models that can be accessed via APIs for a fee, which can help reduce the computational burden of running them locally.
49 |
50 | #### What is a vector?
51 |
52 | In the context of natural language processing, a vector is a list of numbers used to represent a word or phrase in a way that captures its relationship to other words or phrases within a model. A key useful feature of these vectors is that similar things cluster together.
53 |
54 | Each vector in a model consists of the same list length, with every position in the list representing a semantically interesting feature discovered about the word or phrase through statistical analysis. While it's typically difficult to explain what these individual features might mean to a human, quantifying them allows vectors to be mathematically compared as coordinates in a high-dimensional space. In such a space, distance can approximate overall similarity in meaning between the words or phrases from which the vectors were derived.
55 |
56 | Vectors are typically derived using techniques such as word embeddings, which maps each word to a high-dimensional vector based on its co-occurrence with other words in a large corpus of text.
57 |
58 | #### What is Beam search?
59 |
60 | Beam search is a search algorithm used to generate output sequences from a model during inference. It works by maintaining a set of the top k most likely sequences at each step of the generation process, where k is known as the Beam width. The algorithm then continues generating new tokens for each sequence in the set until all sequences have reached an end-of-sequence token or a maximum length has been reached. At each step, the set of possible sequences is pruned based on their likelihood according to the model's predictions, resulting in a final set of top-k output sequences.
61 |
62 | #### What is frequency?
63 |
64 |
65 | {% set slide = { "alt": "Frequency Parameter Visualization","title": "Frequency", "description": "
Frequency is a parameter used in language models to control how often a token can be repeated in the generated output. It works by penalizing the model for repeating tokens that have already been used, preventing the model from generating repetitive or nonsensical text. The higher this is set, the more that repetition of tokens present in the context will be penalized in suggestions.
Temperature is a technique used in language models to control the level of randomness and creativity in the generated output during inference. It works by scaling the predicted probability distribution over possible tokens at each step by a temperature parameter, which controls how much the probabilities are 'softened' or spread out.
Lower temperatures result in more conservative and predictable output, while higher temperatures lead to more diverse and creative output. However, setting the temperature too high can also lead to nonsensical or ungrammatical sentences. Finding the optimal temperature for a given task or application often requires experimentation and fine-tuning.
", "image": "/img/ai/carousels/parameters/llm-03-temp.jpg"}%}
75 | {% from "partials/image-cards/image-card.html" import imageCard %}
76 | {% set orientation = "image-single" %}
77 | {{ imageCard(slide, orientation,false) }}
78 |
79 |
80 | #### What is sampling?
81 |
82 | Sampling is another algorithm used to generate output sequences from a model during inference. Unlike Beam search, which generates only the top-k most likely sequences at each step, sampling generates output tokens probabilistically based on the model's predicted probability distribution over all possible tokens at that step. This can lead to more diverse and creative output compared to Beam search, but it can also result in less coherent or grammatical sentences if not properly controlled through techniques such as temperature scaling or nucleus sampling.
83 |
84 | #### What is 'top_k' sampling?
85 |
86 | Top-k sampling is a method used in language generation where, instead of considering all possible next words in the vocabulary, the model only considers the top 'k' most probable next words.
87 |
88 | This technique helps to focus the model on likely continuations and reduces the chances of generating irrelevant or nonsensical text. It strikes a balance between creativity and coherence by limiting the pool of next word choices, but not so much that the output becomes deterministic.
89 |
90 | #### What is 'top_p' sampling?
91 |
92 |
93 | {% set slide = { "alt": "Top P Parameter Visualization","title": "Top P (Probability)", "description": "
Top-p sampling, also known as nucleus sampling, is a strategy where the model considers only the smallest set of top words whose cumulative probability exceeds a threshold 'p'.
Unlike top-k which considers a fixed number of words, top-p adapts based on the distribution of probabilities for the next word. This makes it more dynamic and flexible. It helps create diverse and sensible text by allowing less probable words to be selected when the most probable ones don't add up to 'p'.
--------------------------------------------------------------------------------
/templates/content/notable-projects/index.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | ## Notable Projects
4 | Submit a project you love to ai-guide@mozilla.com.
5 |
6 | #### Complete LLM Solutions
7 | These projects offer a full-stack chatbot solution, meaning that there is at least an inference run-time and a UI for interacting with it. Actual feature sets may vary rather wildly beyond that.
8 |
9 | ##### [GPT4All](https://gpt4all.io)
10 | License: MIT | Built with: gpt4all, llama.cpp
11 |
12 | A one-click installer (for Windows, MacOS, and Ubuntu) for a locally-running chatbot with a native UI client. Probably the easiest to use and best all-around open source chatbot app. Supports a wide variety of model architectures (including LLaMA, LLaMA 2, GPT-J, MPT, and Falcon), makes it easy to download weights files, supports retrieval augmented generation (i.e. asking questions across local private documents), offers both CPU and GPU inference, and is highly configurable. It’s part of the overall GPT4All project from Nomic.ai.
13 |
14 | ##### [privateGPT](https://github.com/imartinez/privateGPT)
15 | License: Apache 2.0 |
16 | Built with: gpt4all, llama.cpp, langchain, sentence-transformers, chroma
17 |
18 | privateGPT is a chatbot project focused on retrieval augmented generation. It runs from the command line, easily ingests a wide variety of local document formats, and supports a variety of model architecture (by building on top of the gpt4all project).
19 |
20 | ##### [Ollama](https://github.com/jmorganca/ollama)
21 | License: MIT |
22 | Built with: llama.cpp, and a bunch of original Go code
23 |
24 | Ollama provides a one-click chatbot installer for MacOS, with version for Windows and Linux expected in the future. It runs from the command line and lacks a built-in native GUI interface (although one is available). It also lacks many of the features offered by GPT4All. But it’s extremely easy to get working, and it runs quite well on consumer-grade Apple Silicon-based MacBooks. Ultimately, it’s similar to running llama.cpp yourself, just easier.
25 |
26 | ##### [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
27 | License: AGPL 3.0
28 |
29 | This project is probably the most popular open source Web-based chatbot front-end, and comes paired with most of the major inference runtimes. It provides most of the user-facing functionality found in the ChatGPT client, although with a different UI metaphor. This project has quite a lot of users and quite a number of forks, so it’s one to keep an eye on.
30 |
31 |
32 | #### Inference runtimes
33 | These projects provide tools for running LLMs for text inference activities. They roughly bifurcate on a “LLaMA vs. Hugging Face Transformers” axis. Some are command line tools, whereas others are meant to be used in a server capacity.
34 |
35 |
36 | ##### [llama.cpp](https://github.com/ggerganov/llama.cpp)
37 | License: MIT |
38 | Built with: GGML, a model quantization framework, recently replaced by GGUF
39 |
40 | llama.cpp is perhaps the most noteworthy and impactful project in open source LLM tooling to date. It enables users to do inference generation with LLaMA-architecture models in a way that is performant on consumer-grade hardware. It accomplishes this by quantizing model weights files, reducing their size and thus their memory and computational requirements. This, along with other tricks, makes it possible to run these models just with a CPU. Originally optimized for Apple Silicon, llama.cpp has grown to support Intel, as well as GPUs from Nvidia and Apple. As the underpinning of many other OSS LLM projects, this project remains highly influential. As evidence, the project maintainer recently made a major breaking change (from GGML- to GGUF-format weights) and most developers opted to come along for the ride.
41 |
42 |
43 | ##### [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
44 | License: MIT |
45 | Built with: llama.cpp
46 |
47 | This project wraps llama.cpp to provide Python bindings. Perhaps even more significantly, it exposes an API endpoint that mimics the OpenAI API signature. This neat trick makes it relatively easy to take any software that integrates w/ the OpenAI API and “swap it out” for a locally-running open source LLM.
48 |
49 |
50 | ##### [transformers](https://github.com/huggingface/transformers)
51 | License: Apache 2.0 |
52 | Built with: Hugging Face Transformers
53 |
54 | Hugging Face’s implementation of Transformers has proven to be a critical enabling technology for open source machine learning developers. It makes it easy to, often with just a few lines of code, execute a huge variety of models. HF’s site features literally thousands of models, many of which are useful in cases beyond the content of generative chatbots. With the transformers library, they are all within easy reach. However, transformers is a GPU-centric library; it does not offer the kind of CPU-based computational frugality of something like llama.cpp.
55 |
56 | ##### [llm.rs](https://github.com/rustformers/llm)
57 | License: Apache 2.0 + MIT |
58 |
59 | Essentially a Rust implementation of llama.cpp, this young project leverages Rust’s unique advantages and already has a small ecosystem growing up around it.
60 |
61 | ##### [Text-generation-inference](https://github.com/huggingface/text-generation-inference)
62 | License: Apache 2.0 |
63 | Built with: transformers
64 |
65 | This project is the open source version of the software that powers Hugging Face’s Inference Endpoints service. As such, it’s a production-ready inference-as-a-service API that can be used to run any transformers-compatible model.
66 |
67 | ##### [VLLM](https://github.com/vllm-project/vllm)
68 | License: Apache 2.0 |
69 | Built with: transformers
70 |
71 | This recent project essentially offers a higher-performance take on text-generation-inference. That is, it provides an inference engine that is considerably faster than regular old transformers. It can also act as a server by exposing an OpenAI-compatible API (much like llama-cpp-python does). There’s been a fair amount of buzz about this project lately because it offers a relatively plug-and-play means of improving performance within the Hugging Face ecosystem.
72 |
73 |
74 | ##### [llm](https://github.com/simonw/llm)
75 | License: Apache 2.0
76 |
77 | This is a command line tool that makes it easy to experiment with a variety of open source LLMS and OpenAI tools, both via the same tool.
78 |
79 |
80 | ##### [MLC](https://github.com/mlc-ai/mlc-llm)
81 | License: Apache 2.0
82 |
83 | MLC is a relatively recent development that could merit ongoing attention. This project aims to enable “native deployment” of LLMs; this means running LLMs in a manner optimized to a wide variety of hardware platforms, including desktop/laptops, phones, and environments that lack GPUs. It’s still in its early days, but it’s already seeing some uses, including as a plug-in for llm.
84 |
85 |
86 | ##### [gpt4all-backend](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-backend) and [gpt4all-api](https://github.com/nomic-ai/gpt4all/tree/main/gpt4all-api)
87 | License: MIT
88 |
89 | Note that the GPT4All one-click installer is based on underlying libraries that can be used to power an inference server. It can even expose an OpenAI-compatible API endpoint. A number of OSS projects embed these libraries as a way to easily get multi-architecture LLM support.
90 |
91 |
92 |
93 | #### Retrieval augmented generation
94 | These projects support retrieval augmented generation use cases for chatbots, wherein custom external documents can be used to augment a chatbot’s responses. This often requires the use of similarity searches across vector embedding databases.
95 |
96 |
97 | ##### [FAISS](https://github.com/facebookresearch/faiss)
98 | License: MIT
99 |
100 | This project comes from Facebook and provides an easy-to-use mechanism for storing vector embeddings and then searching across them. It has proven popular with many open source developers and is rather commonly found in tutorials explaining how to do retrieval augmented generation.
101 |
102 |
103 | ##### [Chroma](https://github.com/chroma-core/chroma)
104 | License: Apache 2.0
105 |
106 | Probably the most popular open source vector database, Chroma is used by a wide variety of opens source and commercial AI products. It has a robust feature set, is easy to use and deploy, and integrates with other popular tools (like LangChain; see below).
107 |
108 |
109 | ##### [Weviate](https://github.com/weaviate/weaviate)
110 | License: BSD 3-Clause
111 |
112 | Weviate is an alternative to Chroma that is written primarily in Go (vs. Python for Chroma) and may offer better performance while boasting similar integrations.
113 |
114 |
115 | ##### [pgvector](https://github.com/pgvector/pgvector)
116 | License: PostgreSQL license?
117 |
118 | This project enables you to use the mature and beloved PostgreSQL database as a vector embeddings solution. This is included in the list not because we have much experience working with it, but simply because it represents an interesting alternative solution for developers who prefer to stick with tools they already know and have deployed in production.
119 |
120 |
121 | ##### [llama_index](https://github.com/jerryjliu/llama_index)
122 | Licensed: MIT |
123 | Built with: langchain
124 |
125 | LlamaIndex offers a robust set of integrations to data sources. This makes it easier to harvest the data that you want to provide to your chatbot, and to keep it updated. This project is built on top of LangChain (see below).
126 |
127 | ##### [LanceDB](https://lancedb.github.io/lancedb/)
128 | Licensed: Apache
129 |
130 | LanceDB is a local high-performance vector database written in Rust. A hidden gem similar to SQLite.
131 |
132 |
133 | #### Orchestration
134 | Orchestration tools help developers compose LLM-based applications by integrating and coordinating multiple tools and models.
135 |
136 |
137 | ##### [LangChain](https://github.com/langchain-ai/langchain)
138 | License: MIT
139 |
140 | LangChain is the leading open source LLM orchestration tool. A number of open source AI projects utilize LangChain under the covers, so its influence has grown steadily over the past year. There is a growing faction of open source developers who believe that LangChain is overly complex and attempts to do too much. But even if one subscribes to that belief, LangChain remains a useful resource as a way to learn and understand a variety of LLM development best practices for common use cases.
141 |
142 |
143 | ##### [simpleaichat](https://github.com/minimaxir/simpleaichat)
144 | License: MIT
145 |
146 | Created by a frustrated LangChain user as a simpler alternative, this project has some early buzz and is worth watching. It doesn’t offer nearly as much functionality as LangChain, but it’s arguably much easier to learn and use.
147 |
148 | ##### [Guardrails AI](https://github.com/guardrails-ai/guardrails)
149 | License: Apache 2.0
150 |
151 | Guardrails adds AI safety and control mechanisms to your LLM, giving you the ability to validate model output formats and content, and to automatically (and invisibly to the end user) re-query the model when its responses don’t meet your expectations or needs.
152 |
153 |
154 | #### Front-ends
155 | These projects provide chatbot-style user interfaces on top of LLM inference engines.
156 |
157 | ##### [chatbot-ui](https://github.com/mckaywrigley/chatbot-ui)
158 | License: MIT
159 |
160 | This project is an excellent carbon copy of the ChatGPT Web interface. It implements the OpenAI API, which means you can easily point it at most of the interference runtimes mentioned earlier in this document.
161 |
162 |
As of the now, the absolute best and easiest way to run open-source LLMs locally is to use Mozilla's new llamafile project.
4 |
llamafiles are executable files that run on six different operating systems (macOS, Windows, Linux, FreeBSD, OpenBSD and NetBSD). They bundle up a single model's weights along with an inference environment into A SINGLE FILE (so amazing!).
5 |
You can choose between a "server" version (API plus web interface) or a "command-line" version (similar to Meta's original LLaMa interface on CLI). To learn more about this project, here's the README and a long post about it from Simon Willison.
6 |
For our work, we will be spending some time with a small 7B parameter model, Mistral_7B, which reportedly shows stellar performance for a 7B model.
7 |
Installing Mistral_7B
8 |
Let's grab the llamafile first from the creator of llamafile Justine Tunney's Huggingface. We'll use the command line model to showcase some features.
llamafiles are a combination of Justine's cosmopolitan (native single-file executables on any platform), combined with the community's amazing work on llama.cpp, a C++ version of Meta's LLaMa that can run usably on CPUs instead of GPUs created by ggerganov.
11 |
Mistral-7B is a model created by French startup Mistral AI, with open weights and sources. Since it's based on the LLaMa architecture, we are able to run inference on it locally using llama.cpp, which then enables a llamafile.
12 |
The model we're downloading is the instruct-tuned version. This model is tuned to respond by following a system prompt with instructions.
13 |
To run, it's as simple as running these commands in your terminal:
This makes the llamafile executable, then kicks off the model with an interactive prompt. You'll see that the model is loaded into memory using a 'memory map' - you'll need roughly 5GB of RAM to enable usable inference. A GPU is not mandatory since llama.cpp is designed to utilize all available CPUs to maximize performance.
21 |
Interactive mode enables you to chat with the bot like you normally do with an LLM chat interface, but it's quite barebones. Using the server llamafile instead adds an option to use a self-hosted web UI with more features and OpenAI-compatible API at https://localhost:8080
22 |
Other models
23 |
Here are other interesting llamafiles:
24 |
25 |
LLaVa v1.5 7B: a compelling new multi-modal (takes image input) model
36 |
--------------------------------------------------------------------------------
/templates/content/running-llms-locally/index.md:
--------------------------------------------------------------------------------
1 | ## Running LLMs Locally
2 |
3 | #### What is a llamafile?
4 |
5 | As of the now, the absolute best and easiest way to run open-source LLMs locally is to use Mozilla's new [llamafile](https://github.com/Mozilla-Ocho/llamafile) project.
6 |
7 | llamafiles are executable files that run on six different operating systems (macOS, Windows, Linux, FreeBSD, OpenBSD and NetBSD). They bundle up a single model's weights along with an inference environment into **A SINGLE FILE** (so amazing!).
8 |
9 | You can choose between a "server" version (API plus web interface) or a "command-line" version (similar to Meta's original LLaMa interface on CLI). To learn more about this project, here's the [README](https://github.com/Mozilla-Ocho/llamafile) and a [long post about it from Simon Willison](https://simonwillison.net/2023/Nov/29/llamafile/).
10 |
11 | For our work, we will be spending some time with a small 7B parameter model, _Mistral\_7B_, which reportedly shows stellar performance for a 7B model.
12 |
13 | #### Installing Mistral_7B
14 |
15 | Let's grab the llamafile first from the creator of llamafile [Justine Tunney](https://justine.lol)'s [Huggingface](https://huggingface.co/jartine/mistral-7b.llamafile). We'll use the command line model to showcase some features.
16 |
17 | Download the model by [tapping this link](https://huggingface.co/jartine/mistral-7b.llamafile/resolve/main/mistral-7b-instruct-v0.1-Q4_K_M-main.llamafile?download=true).
18 |
19 | llamafiles are a combination of Justine's [cosmopolitan](https://justine.lol/cosmopolitan/index.html) (native single-file executables on any platform), combined with the community's amazing work on [llama.cpp](https://github.com/ggerganov/llama.cpp), a C++ version of Meta's LLaMa that can run usably on CPUs instead of GPUs created by [ggerganov](https://github.com/ggerganov/llama.cpp).
20 |
21 | Mistral-7B is a model created by French startup [Mistral AI](https://mistral.ai/company/), with open weights and sources. Since it's based on the LLaMa architecture, we are able to run inference on it locally using llama.cpp, which then enables a llamafile.
22 |
23 | The model we're downloading is the `instruct`-tuned version. This model is tuned to respond by following a system prompt with instructions.
24 |
25 | To run, it's as simple as running these commands in your terminal:
26 | ```
27 | mv mistral-7b-instruct-v0.1-Q4_K_M-main.llama mistral-7b-instruct.llamafile
28 |
29 | chmod +x mistral-7b-instruct.llamafile
30 |
31 | ./mistral-7b-instruct.llamafile --interactive-first
32 | ```
33 |
34 | This makes the llamafile executable, then kicks off the model with an interactive prompt. You'll see that the model is loaded into memory using a 'memory map' - you'll need roughly 5GB of RAM to enable usable inference. A GPU is not mandatory since llama.cpp is designed to utilize all available CPUs to maximize performance.
35 |
36 | Interactive mode enables you to chat with the bot like you normally do with an LLM chat interface, but it's quite barebones. Using the `server` llamafile instead adds an option to use a self-hosted web UI with more features and OpenAI-compatible API at [https://localhost:8080](https://localhost:8080)
37 |
38 |
39 | #### Other models
40 |
41 | Here are other interesting llamafiles:
42 | - [LLaVa v1.5 7B](https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4-server.llamafile?download=true): a compelling new multi-modal (takes image input) model
43 | - [WizardCoder 7B](https://huggingface.co/jartine/wizardcoder-13b-python/blob/main/wizardcoder-python-13b-server.llamafile): a code-generation model
44 |
45 | Our next section will help us compare open models for our particular use-case.
46 |
47 |
14 | 