📚 编程进阶教程站

第 8 章 · 语义搜索 ​

本章目标：掌握语义搜索的完整链路——文本分块、嵌入、向量索引、BM25 关键词检索、重排序，并最终组装出 RAG（检索增强生成）流水线。

8.1 准备工作 ​

本章对应的官方笔记本是 chapter08/Chapter 8 - Semantic Search.ipynb。

如果你在 Google Colab（或任何其他云环境）中查看笔记本，需要取消注释并运行以下代码块来安装本章依赖：

python
# %%capture
# !pip install langchain==0.2.5 faiss-cpu==1.8.0 cohere==5.5.8 langchain-community==0.2.5 rank_bm25==0.2.2 sentence-transformers==3.0.1
# !pip install llama-cpp-python==0.2.78  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

## IMPORTANT: Make sure to restart the session after installing the packages above.

💡 NOTE：运行本章示例需要 GPU。在 Google Colab 中，进入 Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4。

8.2 Dense Retrieval 稠密检索示例 ​

8.2.1 获取文本档案并分块 ​

首先准备 Cohere 客户端（需要自行申请 API Key）：

python
import cohere

# Paste your API key here. Remember to not share publicly
api_key = ''

# Create and retrieve a Cohere API key from os.cohere.ai
co = cohere.Client(api_key)

准备一段关于电影《星际穿越》（Interstellar）的文字档案，并把它切分成句子块：

python
text = """
Interstellar is a 2014 epic science fiction film co-written, directed, and produced by Christopher Nolan.
It stars Matthew McConaughey, Anne Hathaway, Jessica Chastain, Bill Irwin, Ellen Burstyn, Matt Damon, and Michael Caine.
Set in a dystopian future where humanity is struggling to survive, the film follows a group of astronauts who travel through a wormhole near Saturn in search of a new home for mankind.

Brothers Christopher and Jonathan Nolan wrote the screenplay, which had its origins in a script Jonathan developed in 2007.
Caltech theoretical physicist and 2017 Nobel laureate in Physics[4] Kip Thorne was an executive producer, acted as a scientific consultant, and wrote a tie-in book, The Science of Interstellar.
Cinematographer Hoyte van Hoytema shot it on 35 mm movie film in the Panavision anamorphic format and IMAX 70 mm.
Principal photography began in late 2013 and took place in Alberta, Iceland, and Los Angeles.
Interstellar uses extensive practical and miniature effects and the company Double Negative created additional digital effects.

Interstellar premiered on October 26, 2014, in Los Angeles.
In the United States, it was first released on film stock, expanding to venues using digital projectors.
The film had a worldwide gross over $677 million (and $773 million with subsequent re-releases), making it the tenth-highest grossing film of 2014.
It received acclaim for its performances, direction, screenplay, musical score, visual effects, ambition, themes, and emotional weight.
It has also received praise from many astronomers for its scientific accuracy and portrayal of theoretical astrophysics. Since its premiere, Interstellar gained a cult following,[5] and now is regarded by many sci-fi experts as one of the best science-fiction films of all time.
Interstellar was nominated for five awards at the 87th Academy Awards, winning Best Visual Effects, and received numerous other accolades"""

# Split into a list of sentences
texts = text.split('.')

# Clean up to remove empty spaces and new lines
texts = [t.strip(' \n') for t in texts]

8.2.2 文本块向量化 ​

调用 co.embed 把每个句子块转换为嵌入向量（注意文档用 search_document 类型）：

python
import numpy as np

# Get the embeddings
response = co.embed(
  texts=texts,
  input_type="search_document",
).embeddings

embeds = np.array(response)
print(embeds.shape)

8.2.3 构建检索索引 ​

用 FAISS 建一个 L2 距离的平面向量索引：

python
import faiss

dim = embeds.shape[1]
index = faiss.IndexFlatL2(dim)
index.add(np.float32(embeds))

8.2.4 检索索引 ​

封装一个完整的检索函数：查询向量化 → 最近邻搜索 → 格式化输出：

python
import pandas as pd

def search(query, number_of_results=3):

  # 1. Get the query's embedding
  query_embed = co.embed(texts=[query],
                input_type="search_query",).embeddings[0]

  # 2. Retrieve the nearest neighbors
  distances , similar_item_ids = index.search(np.float32([query_embed]), number_of_results)

  # 3. Format the results
  texts_np = np.array(texts) # Convert texts list to numpy for easier indexing
  results = pd.DataFrame(data={'texts': texts_np[similar_item_ids[0]],
                              'distance': distances[0]})

  # 4. Print and return the results
  print(f"Query:'{query}'\nNearest neighbors:")
  return results

测试查询「电影的科学有多精确」：

python
query = "how precise was the science"
results = search(query)
results

8.2.5 BM25 关键词检索 ​

稠密检索之外，传统的基于词频的 BM25 词法检索仍是重要基线。先定义分词器：

python
from rank_bm25 import BM25Okapi
from sklearn.feature_extraction import _stop_words
import string

def bm25_tokenizer(text):
    tokenized_doc = []
    for token in text.lower().split():
        token = token.strip(string.punctuation)

        if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:
            tokenized_doc.append(token)
    return tokenized_doc

对语料全量分词并构建 BM25 索引：

python
from tqdm import tqdm

tokenized_corpus = []
for passage in tqdm(texts):
    tokenized_corpus.append(bm25_tokenizer(passage))

bm25 = BM25Okapi(tokenized_corpus)

实现关键词检索函数：

python
def keyword_search(query, top_k=3, num_candidates=15):
    print("Input question:", query)

    ##### BM25 search (lexical search) #####
    bm25_scores = bm25.get_scores(bm25_tokenizer(query))
    top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]
    bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]
    bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)

    print(f"Top-3 lexical search (BM25) hits")
    for hit in bm25_hits[0:top_k]:
        print("\t{:.3f}\t{}".format(hit['score'], texts[hit['corpus_id']].replace("\n", " ")))
python
keyword_search(query = "how precise was the science")

8.2.6 稠密检索的注意事项 ​

稠密检索并非万能——换一个与语料措辞毫无重叠的问题（月球的质量），看看会发生什么：

python
query = "What is the mass of the moon?"
results = search(query)
results

8.3 Reranking 重排序示例 ​

重排序（Reranking）是提升检索质量的关键手段：先用宽松的召回拿到候选集，再用更强大的模型精排。直接调用 Cohere 的 Rerank API：

python
query = "how precise was the science"
results = co.rerank(query=query, documents=texts, top_n=3, return_documents=True)
results.results

查看每条结果的相关性得分：

python
for idx, result in enumerate(results.results):
    print(idx, result.relevance_score , result.document.text)

把 BM25 关键词检索与重排序组合成混合检索方案：

python
def keyword_and_reranking_search(query, top_k=3, num_candidates=10):
    print("Input question:", query)

    ##### BM25 search (lexical search) #####
    bm25_scores = bm25.get_scores(bm25_tokenizer(query))
    top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]
    bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]
    bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)

    print(f"Top-3 lexical search (BM25) hits")
    for hit in bm25_hits[0:top_k]:
        print("\t{:.3f}\t{}".format(hit['score'], texts[hit['corpus_id']].replace("\n", " ")))

    #Add re-ranking
    docs = [texts[hit['corpus_id']] for hit in bm25_hits]

    print(f"\nTop-3 hits by rank-API ({len(bm25_hits)} BM25 hits re-ranked)")
    results = co.rerank(query=query, documents=docs, top_n=top_k, return_documents=True)
    for hit in results.results:
        print("\t{:.3f}\t{}".format(hit.relevance_score, hit.document.text.replace("\n", " ")))
python
keyword_and_reranking_search(query = "how precise was the science")

8.4 Retrieval-Augmented Generation 检索增强生成 ​

8.4.1 示例：使用 LLM API 的接地生成 ​

RAG 的完整流程：检索相关文档 → 把文档交给 LLM 生成有据可依的回答。

python
query = "income generated"

# 1- Retrieval
# We'll use embedding search. But ideally we'd do hybrid
results = search(query)

# 2- Grounded Generation
docs_dict = [{'text': text} for text in results['texts']]
response = co.chat(
    message = query,
    documents=docs_dict
)

print(response.text)

响应对象中还包含引用信息：

python
response
python
response.citations

8.4.2 示例：使用本地模型的 RAG ​

不依赖云端 API，也可以用本地模型搭建完整的 RAG 流水线。

加载生成模型——下载 Phi-3 的量化 GGUF 权重：

python
!wget https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf
python
from langchain import LlamaCpp

# Make sure the model path is correct for your system!
llm = LlamaCpp(
    model_path="Phi-3-mini-4k-instruct-q4.gguf",
    n_gpu_layers=-1,
    max_tokens=500,
    n_ctx=2048,
    seed=42,
    verbose=False
)

加载嵌入模型：

python
from langchain.embeddings.huggingface import HuggingFaceEmbeddings

# Embedding Model for converting text to numerical representations
embedding_model = HuggingFaceEmbeddings(
    model_name='BAAI/bge-small-en-v1.5'
)

准备向量数据库：

python
from langchain.vectorstores import FAISS

# Create a local vector database
db = FAISS.from_texts(texts, embedding_model)

编写 RAG 提示并把所有组件串成 RetrievalQA 流水线：

python
from langchain import PromptTemplate
from langchain.chains import RetrievalQA


# Create a prompt template
template = """<|user|>
Relevant information:
{context}

Provide a concise answer the following question using the relevant information provided above:
{question}<|end|>
<|assistant|>"""
prompt = PromptTemplate(
    template=template,
    input_variables=["context", "question"]
)

# RAG Pipeline
rag = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type='stuff',
    retriever=db.as_retriever(),
    chain_type_kwargs={
        "prompt": prompt
    },
    verbose=True
)
python
rag.invoke('Income generated')

8.5 本章小结 ​

  • 语义搜索四步走：文本分块 → 嵌入向量化 → FAISS 向量索引 → 最近邻检索；文档与查询分别使用 search_document 与 search_query 两种嵌入类型；
  • BM25 是基于词频的经典词法检索基线，与稠密检索互补——当查询与语料措辞毫无重叠时稠密检索可能失灵；
  • 重排序采用「宽召回 + 精排」策略：BM25 先取候选集，再由 Rerank API 按相关性重新打分排序；
  • RAG = 检索 + 接地生成：云端可用 co.chat 直接传入 documents 获得带引用的回答，本地则用 LangChain 的 FAISS 向量库 + RetrievalQA 流水线组合 LlamaCpp 与 bge-small 嵌入模型。

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. 调用 co.embed 时，文档与查询分别应使用哪种 input_type？

2. 查询 "What is the mass of the moon?" 在稠密检索中效果不佳，说明了什么？

3. BM25 + Reranking 混合方案的流程是？

4. 本地 RAG 流水线（8.4.2 节）中各组件的正确组合是？

🛠️ 动手实践 ​

  1. 把 8.2.1 节的分块策略从「按句号切分」改为「按段落切分」（以 \n\n 分隔），重建嵌入与 FAISS 索引，对比查询 "how precise was the science" 的返回结果差异。
  2. 用 8.2.6 节的失败查询 "What is the mass of the moon?" 分别跑一遍稠密检索、BM25 关键词检索和 8.3 节的混合重排序检索，整理三种方法 Top-3 结果的对比表格并解释差异。
  3. 参考 8.4.2 节的本地 RAG 流水线，把嵌入模型换成 'sentence-transformers/all-MiniLM-L6-v2'，重建向量库后分别调用 rag.invoke('Income generated') 和 rag.invoke('Who directed Interstellar?')，观察回答质量是否变化。