Chapter 9 · Self-Consistency and Generated Knowledge

Chapter goals: Understand the principle and implementation of Self-Consistency multi-sample voting, master generated knowledge prompting, and learn to balance cost and accuracy.

9.1 Self-Consistency: Multi-Path Reasoning Voting

Self-Consistency is a simple but powerful post-processing technique. The core idea: instead of relying on a single greedy decode, sample multiple reasoning paths and vote for the most consistent answer.

Traditional methods take only the single highest-probability answer:

Prompt: "2 + 3 × 4 = ?"
Model output: "14"  ← Wrong, because addition was done before multiplication

Self-Consistency samples multiple answers and votes:

Sampled 5 times: ["14", "14", "14", "14", "14"]  → Vote result: "14" (majority)
Sampled 5 times: ["5", "14", "14", "5", "14"]    → Vote result: "14" (3 vs 2)
Sampled 5 times: ["5", "5", "5", "5", "14"]      → Vote result: "5" (majority, correct answer)

9.2 Implementing Self-Consistency

import random
from collections import Counter

def self_consistency(prompt, model, n_samples=10, temperature=0.7):
    """
    Self-Consistency implementation
    
    Args:
        prompt: input prompt
        model: LLM model interface
        n_samples: number of samples (more = more accurate but costlier)
        temperature: sampling temperature, controls diversity
    """
    answers = []
    
    for _ in range(n_samples):
        # Generate reasoning chain using CoT
        cot_prompt = f"{prompt}\nLet's think step by step."
        response = model.generate(cot_prompt, temperature=temperature)
        
        # Extract the final answer (assuming it's at the end)
        answer = extract_answer(response)
        answers.append(answer)
    
    # Vote for the most common answer
    most_common = Counter(answers).most_common(1)[0][0]
    
    return {
        'answer': most_common,
        'votes': dict(Counter(answers)),
        'confidence': Counter(answers).most_common(1)[0][1] / n_samples
    }

# Usage example
result = self_consistency(
    prompt="Xiao Ming has 5 apples, eats 2, then buys 3 more. How many does he have now?",
    model=gpt_4,
    n_samples=10
)
print(f"Answer: {result['answer']}, Confidence: {result['confidence']:.2%}")

9.3 Generated Knowledge Prompting

Generated knowledge prompting reduces hallucination and improves accuracy by having the model first generate relevant knowledge, then answer based on that knowledge.

Workflow

Original question → Knowledge generation → Knowledge + question → Final answer

Example Implementation

def generated_knowledge_prompt(question, model):
    """
    Generated knowledge prompting: generate relevant knowledge first, then answer
    """
    # Step 1: Generate relevant knowledge
    knowledge_prompt = f"""Generate relevant knowledge for answering the question:
Question: {question}
Knowledge:
"""
    generated_knowledge = model.complete(knowledge_prompt)
    
    # Step 2: Answer based on generated knowledge
    answer_prompt = f"""Based on the following knowledge, answer the question:

Knowledge:
{generated_knowledge}

Question: {question}
Answer:
"""
    final_answer = model.complete(answer_prompt)
    
    return {
        'question': question,
        'generated_knowledge': generated_knowledge,
        'final_answer': final_answer
    }

Use Cases

• Factual Q&A: generating background knowledge reduces model hallucination
• Complex reasoning: decompose the problem into knowledge + reasoning in two steps
• Domain-specific: generate domain knowledge first, then apply

9.4 Cost-Accuracy Tradeoff

Both Self-Consistency and generated knowledge prompting increase computational cost, so tradeoffs are needed:

Method	Extra Cost	Accuracy Gain	Applicable Scenario
Single CoT	None	+10–20%	Simple reasoning
Self-Consistency (n=5)	×5	+15–30%	High-accuracy needs
Self-Consistency (n=10)	×10	+20–40%	Critical decisions
Generated knowledge	×2	+10–25%	Factual Q&A

Optimization Strategy

def adaptive_consistency(question, model, max_samples=10):
    """
    Adaptive Self-Consistency: dynamically decide sample count based on confidence
    """
    answers = []
    
    for i in range(1, max_samples + 1):
        response = model.generate(f"{question}\nLet's think step by step.")
        answers.append(extract_answer(response))
        
        # Check if consensus is reached
        if i >= 5:  # minimum 5 samples
            vote_counts = Counter(answers)
            top_answer, top_count = vote_counts.most_common(1)[0]
            
            # If one answer exceeds 80% share, stop early
            if top_count / i >= 0.8:
                print(f"Consensus reached, stopping after {i} samples")
                return top_answer, i
    
    # Return final vote result
    return Counter(answers).most_common(1)[0][0], max_samples

9.5 Practical Advice

1. Start Simple: begin with single CoT; consider Self-Consistency only if results are good enough
2. Tune n_samples: start from 5, adjust to 10–20 based on效果
3. Use Temperature: set temperature=0.7–0.9 to increase sampling diversity
4. Extract Carefully: answer extraction must be reliable to avoid parsing errors
5. Monitor Cost: record token consumption and latency per call

Chapter Summary

• Self-Consistency: multi-sample voting mechanism, improves complex reasoning accuracy
• Generated knowledge prompting: generate knowledge first, then answer, reducing hallucination
• Cost tradeoff: larger n_samples = more accurate but costlier; use adaptive strategies to optimize
• Practice points: start simple, tune temperature, be careful with answer extraction

🧪 随堂测验

点击你认为正确的选项。答错时会展示正确答案与原因解析。

1. What is the core idea of Self-Consistency?

2. What is the first step of generated knowledge prompting?

3. What is the typical early-stop condition for adaptive Self-Consistency?

4. Which scenario is Self-Consistency best suited for?

🛠️ Hands-on Practice

1. Implement the Self-Consistency function and compare accuracy differences for the same question at n=1, 5, 10.
2. Design a generated knowledge prompting application and compare "direct answer" vs. "generate knowledge then answer" on factual Q&A.
3. Implement adaptive Self-Consistency: stop sampling early when confidence reaches 80%, and analyze the cost savings.

Next chapter: Chain-of-Thought