📚 AI 工程与智能体教程站

English

简体中文

English

简体中文

Chapter 2 · Tokens and Token Embeddings ​

Chapter goals: Understand tokens and embeddings as indispensable components for building LLMs; hands-on compare different tokenizers; and experience applications ranging from word embeddings to song recommendations.

2.1 Downloading and Running an LLM ​

The first step is to load the model onto the GPU for faster inference. Note that we load the model and tokenizer separately and keep them independent so they can be explored separately later.

python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=False,
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

Below, run a full generation with an apology-email prompt:

python
prompt = "Write an email apologizing to Sarah for the tragic gardening mishap. Explain how it happened.<|assistant|>"

# Tokenize the input prompt
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

# Generate the text
generation_output = model.generate(
  input_ids=input_ids,
  max_new_tokens=20
)

# Print the output
print(tokenizer.decode(generation_output[0]))

Examine the tokenized input_ids:

python
print(input_ids)

Decode each token ID one by one and observe how the text is split:

python
for id in input_ids[0]:
   print(tokenizer.decode(id))

Inspect the raw structure of the generation result:

python
generation_output

An interesting experiment: decoding a single token and a token combination are not always equivalent—

python
print(tokenizer.decode(3323))
print(tokenizer.decode(622))
print(tokenizer.decode([3323, 622]))
print(tokenizer.decode(29901))

2.2 [Optional] Installing Dependencies in Colab ​

If you are viewing this notebook on Google Colab, you need to uncomment and run the code block below to install the dependencies for this chapter:

💡 NOTE: Running the examples requires a GPU (select T4 on Colab).

python
# %%capture
# !pip install --upgrade transformers==4.41.2 sentence-transformers==3.0.1 gensim==4.3.2 scikit-learn==1.5.0 accelerate==0.31.0 peft==0.11.1 scipy==1.10.1 numpy==1.26.4

2.3 Comparing Tokenizers Trained on Different LLMs ​

Different models use different tokenizers, and the splitting granularity varies enormously. Define a visualization function that prints each token with a colored background:

python
from transformers import AutoModelForCausalLM, AutoTokenizer

colors_list = [
    '102;194;165', '252;141;98', '141;160;203',
    '231;138;195', '166;216;84', '255;217;47'
]

def show_tokens(sentence, tokenizer_name):
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
    token_ids = tokenizer(sentence).input_ids
    for idx, t in enumerate(token_ids):
        print(
            f'\x1b[0;30;48;2;{colors_list[idx % len(colors_list)]}m' +
            tokenizer.decode(t) +
            '\x1b[0m',
            end=' '
        )

Prepare a test text covering case variations, emoji, Chinese, punctuation, arithmetic, and whitespace:

python
text = """
English and CAPITALIZATION
🎵 Bird
show_tokens False None elif == >= else: two tabs:"    " Three tabs: "       "
12.0*50=600
"""

Compare the tokenization results across different model tokenizers—bert-base-uncased (case-insensitive):

python
show_tokens(text, "bert-base-uncased")

bert-base-cased (case-sensitive):

python
show_tokens(text, "bert-base-cased")

gpt2:

python
show_tokens(text, "gpt2")

google/flan-t5-small:

python
show_tokens(text, "google/flan-t5-small")

The tokenizer used by GPT-4 (the official implementation is tiktoken; here is the same tokenizer ported to the HF platform):

python
# The official is `tiktoken` but this the same tokenizer on the HF platform
show_tokens(text, "Xenova/gpt-4")

StarCoder2 (requires requesting access first):

python
# You need to request access before being able to use this tokenizer
show_tokens(text, "bigcode/starcoder2-15b")

facebook/galactica-1.3b:

python
show_tokens(text, "facebook/galactica-1.3b")

Phi-3's tokenizer:

python
show_tokens(text, "microsoft/Phi-3-mini-4k-instruct")

2.4 Contextual Word Embeddings from Language Models (BERT-Style Models as an Example) ​

Embeddings produced by language models are contextual—the same word has different vectors in different sentences. Here we demonstrate with DeBERTa:

python
from transformers import AutoModel, AutoTokenizer

# Load a tokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base")

# Load a language model
model = AutoModel.from_pretrained("microsoft/deberta-v3-xsmall")

# Tokenize the sentence
tokens = tokenizer('Hello world', return_tensors='pt')

# Process the tokens
output = model(**tokens)[0]

Inspect the shape of the output tensor:

python
output.shape

Break down the input to see each token:

python
for token in tokens['input_ids'][0]:
    print(tokenizer.decode(token))

The complete output tensor (one contextual vector per token):

python
output

2.5 Text Embeddings (for Sentences and Full Documents) ​

To generate a fixed-length vector for an entire sentence or document, sentence-transformers is the most commonly used tool:

python
from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer('sentence-transformers/all-mpnet-base-v2')

# Convert text to text embeddings
vector = model.encode("Best movie ever!")

The dimensionality of the resulting vector:

python
vector.shape

2.6 Word Embeddings Beyond LLMs ​

Before LLMs appeared, static word embeddings such as Word2Vec and GloVe were already widely used. Load GloVe vectors with gensim:

python
import gensim.downloader as api

# Download embeddings (66MB, glove, trained on wikipedia, vector size: 50)
# Other options include "word2vec-google-news-300"
# More options at https://github.com/RaRe-Technologies/gensim-data
model = api.load("glove-wiki-gigaword-50")

Query words most similar to king (the classic king − man + woman ≈ queen relationship):

python
model.most_similar([model['king']], topn=11)

2.7 Song Recommendation Using Embeddings ​

Finally, a complete application: train Word2Vec on playlist data to recommend similar songs. First download and parse the dataset:

python
import pandas as pd
from urllib import request

# Get the playlist dataset file
data = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/train.txt')

# Parse the playlist dataset file. Skip the first two lines as
# they only contain metadata
lines = data.read().decode("utf-8").split('\n')[2:]

# Remove playlists with only one song
playlists = [s.rstrip().split() for s in lines if len(s.split()) > 1]

# Load song metadata
songs_file = request.urlopen('https://storage.googleapis.com/maps-premium/dataset/yes_complete/song_hash.txt')
songs_file = songs_file.read().decode("utf-8").split('\n')
songs = [s.rstrip().split('\t') for s in songs_file]
songs_df = pd.DataFrame(data=songs, columns = ['id', 'title', 'artist'])
songs_df = songs_df.set_index('id')

Look at the contents of the first two playlists:

python
print( 'Playlist #1:\n ', playlists[0], '\n')
print( 'Playlist #2:\n ', playlists[1])

Each playlist is treated as a "sentence", and the song IDs within it are like "words"; train Word2Vec on this basis:

python
from gensim.models import Word2Vec

# Train our Word2Vec model
model = Word2Vec(
    playlists, vector_size=32, window=20, negative=50, min_count=1, workers=4
)

Ask the model for songs similar to song #2172:

python
song_id = 2172

# Ask the model for songs similar to song #2172
model.wv.most_similar(positive=str(song_id))

Look up the metadata for this song:

python
print(songs_df.iloc[2172])

Wrap a recommendation function to output a list with song titles:

python
import numpy as np

def print_recommendations(song_id):
    similar_songs = np.array(
        model.wv.most_similar(positive=str(song_id),topn=5)
    )[:,0]
    return  songs_df.iloc[similar_songs]

# Extract recommendations
print_recommendations(2172)

Try a few different seed songs:

python
print_recommendations(2172)
python
print_recommendations(842)

2.8 Chapter Summary ​

  • Text is first tokenized into tokens and then mapped to token IDs; tokenizer(prompt).input_ids yields the ID sequence, and tokenizer.decode reverses the process;
  • Decoding a single token and decoding a combination are not always equivalent: decode([3323, 622]) may differ from decoding each individually and concatenating;
  • Different models' tokenizers have vastly different splitting granularities (bert-base-uncased, gpt2, Xenova/gpt-4, Phi-3, etc.), which directly affects sequence length and computational cost;
  • BERT-style models output contextual word embeddings; sentence-transformers outputs sentence-level fixed-length vectors;
  • Static word embeddings (GloVe/Word2Vec), though not LLM products, can still support practical applications such as the "king ≈ queen" analogy and song recommendation.

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What phenomenon does the experiment tokenizer.decode([3323, 622]) in this chapter illustrate?

2. Comparing the tokenization results of bert-base-uncased and bert-base-cased, what is the main difference?

3. What does the encode method of the SentenceTransformer model all-mpnet-base-v2 output?

4. In this chapter's song recommendation application, what do the "sentence" and "word" in Word2Vec correspond to respectively?

🛠️ Hands-on Practice ​

  1. Use show_tokens to compare how "bert-base-cased" and "Xenova/gpt-4" tokenize the same Chinese+emoji test text (the text from Section 2.3); count each tokenizer's token count and explain the source of the difference.
  2. Building on Section 2.5, use all-mpnet-base-v2 to encode both "Best movie ever!" and "Worst film I have seen.", then use NumPy to compute the cosine similarity between the two vectors.
  3. In the song recommendation from Section 2.7, change the song_id in model.wv.most_similar(positive=str(song_id), topn=5) to any ID you like; print songs_df.iloc[song_id] and evaluate whether the recommendations are reasonable.