You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pgml-cms/docs/guides/chatbots/README.md
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,7 @@ Here is an example flowing from:
30
30
31
31
text -> tokens -> LLM -> probability distribution -> predicted token -> text
32
32
33
-
<figure><imgsrc="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrvfCoPdoQeoovZiqNG90%2Fuploads%2FPzJzmVS3uNhbvseiJbgi%2FScreenshot%20from%202023-12-13%2013-19-33.png?alt=media&token=11d57b2a-6aa3-4374-b26c-afc6f531d2f3"alt=""><figcaption><p>The flow of inputs through an LLM. In this case the inputs are "What is Baldur's Gate 3?" and the output token "14" maps to the word "I"</p></figcaption></figure>
33
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_Limitations-Diagram.svg"alt=""><figcaption><p>The flow of inputs through an LLM. In this case the inputs are "What is Baldur's Gate 3?" and the output token "14" maps to the word "I"</p></figcaption></figure>
34
34
35
35
{% hint style="info" %}
36
36
We have simplified the tokenization process. Words do not always map directly to tokens. For instance, the word "Baldur's" may actually map to multiple tokens. For more information on tokenization checkout [HuggingFace's summary](https://huggingface.co/docs/transformers/tokenizer\_summary).
@@ -108,11 +108,11 @@ What does an `embedding` look like? `Embeddings` are just vectors (for our use c
<figure><imgsrc="../../.gitbook/assets/embedding_king.png"alt=""><figcaption><p>The flow of word -> token -> embedding</p></figcaption></figure>
111
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_King-Diagram.svg"alt=""><figcaption><p>The flow of word -> token -> embedding</p></figcaption></figure>
112
112
113
113
`Embeddings` aren't limited to words, we have models that can embed entire sentences.
114
114
115
-
<figure><imgsrc="../../.gitbook/assets/embeddings_tokens.png"alt=""><figcaption><p>The flow of sentence -> tokens -> embedding</p></figcaption></figure>
115
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_Tokens-Diagram.svg"alt=""><figcaption><p>The flow of sentence -> tokens -> embedding</p></figcaption></figure>
116
116
117
117
Why do we care about `embeddings`? `Embeddings` have a very interesting property. Words and sentences that have close [semantic similarity](https://en.wikipedia.org/wiki/Semantic\_similarity) sit closer to one another in vector space than words and sentences that do not have close semantic similarity.
118
118
@@ -157,7 +157,7 @@ print(context)
157
157
158
158
There is a lot going on with this, let's check out this diagram and step through it.
159
159
160
-
<figure><img src="../../.gitbook/assets/chatbot_flow.png"alt=""><figcaption><p>The flow of taking a document, splitting it into chunks, embedding those chunks, and then retrieving a chunk based off of a users query</p></figcaption></figure>
160
+
<figure><img src="../../.gitbook/assets/Chatbots_Flow-Diagram.svg"alt=""><figcaption><p>The flow of taking a document, splitting it into chunks, embedding those chunks, and then retrieving a chunk based off of a users query</p></figcaption></figure>
161
161
162
162
Step 1: We take the document and split it into chunks. Chunks are typically a paragraph or two in size. There are many ways to split documents into chunks, for more information check out [this guide](https://www.pinecone.io/learn/chunking-strategies/).
0 commit comments