Understanding Large Language Models

Large Language Models (LLMs) are the core technology powering modern AI agents. Understanding how they work, their capabilities, and their limitations is essential for effective AI agent development.

How LLMs Work

At a conceptual level, LLMs function through these key mechanisms:

1. Pattern Recognition: LLMs identify statistical patterns in language from massive training datasets
2. Contextual Understanding: They interpret input text based on surrounding context
3. Prediction: They predict the most likely next tokens based on learned patterns
4. Generation: They assemble these predictions into coherent responses

Key LLM Capabilities

Modern LLMs can perform a wide range of tasks that make them valuable for AI agents:

Capability	Description	Agent Applications
Text Generation	Creating coherent, contextually relevant text	Content creation, responses, summaries
Reasoning	Working through problems step-by-step	Decision making, planning, troubleshooting
Instruction Following	Executing specific directions	Task automation, workflow execution
Information Extraction	Identifying key data points from text	Data processing, form filling, analysis
Transformation	Converting between formats and styles	Translation, summarisation, simplification
Tool Use	Generating structured outputs for external tools	API calls, database queries, function execution

The Token Economy

Understanding tokens is crucial for both effective and cost-efficient use of LLMs. Tokens are the fundamental units that LLMs process, and they directly impact both performance and cost.

What Are Tokens?

Tokens are the basic units of text that LLMs process. They're not exactly words or characters, but pieces of text that the model treats as single units.

Token Counting Rules of Thumb

• In English, 1 token ≈ 4 characters
• 1 token ≈ ¾ of a word
• 100 tokens ≈ 75 words
• 1,500 tokens ≈ 1 page of text

Context Windows

The context window is the maximum number of tokens an LLM can process at once, including both the input prompt and the generated output.

Model	Context Window (tokens)	Approximate Text Equivalent
GPT-3.5 Turbo	16,385	~12 pages
GPT-4 Turbo	128,000	~85 pages
Claude 3 Opus	200,000	~133 pages
Mistral Large	32,768	~22 pages
Llama 3 70B	8,192	~5 pages

Token Economy Optimisation Strategies

Implementing a Token-Efficient Agent

import tiktoken

class TokenEfficientAgent:
    def __init__(self, model_name="gpt-3.5-turbo", max_history_tokens=4000):
        self.conversation_history = []
        self.max_history_tokens = max_history_tokens
        self.token_count = 0
        try:
            self.encoding = tiktoken.encoding_for_model(model_name)
        except KeyError:
            print(f"Warning: Model {model_name} not found. Using cl100k_base encoding.")
            self.encoding = tiktoken.get_encoding("cl100k_base")
    
    def _count_tokens(self, text):
        """Count tokens using the specified model's encoder."""
        return len(self.encoding.encode(text))

    def add_message(self, role, content):
        """Add a message to the conversation history with accurate token counting."""
        # Accurate token count for the message content
        message_tokens = self._count_tokens(content)
        # Add overhead tokens per message (approximation, varies by model)
        overhead_tokens_per_message = 4 
        total_tokens = message_tokens + overhead_tokens_per_message
        
        message = {"role": role, "content": content, "tokens": total_tokens}
        self.conversation_history.append(message)
        self.token_count += total_tokens
        
        # Prune history if needed
        self._prune_history()
    
    def _prune_history(self):
        """Ensure conversation history stays within token limits."""
        # Account for tokens needed for the response
        buffer_tokens = 1000 # Reserve tokens for the next model response
        target_tokens = self.max_history_tokens - buffer_tokens

        if self.token_count <= target_tokens:
            return
            
        # Always keep the system message if present
        system_message = None
        if self.conversation_history and self.conversation_history[0]["role"] == "system":
            system_message = self.conversation_history.pop(0)
            self.token_count -= system_message["tokens"]
        
        # Remove oldest messages (excluding system message) until under the limit
        while self.token_count > target_tokens and len(self.conversation_history) > 0:
            # Remove from the beginning (oldest conversation turns)
            removed = self.conversation_history.pop(0)
            self.token_count -= removed["tokens"]
        
        # Re-add system message if it existed
        if system_message:
            self.conversation_history.insert(0, system_message)
            self.token_count += system_message["tokens"]
            # If adding system message back puts us over, remove another message
            if self.token_count > target_tokens and len(self.conversation_history) > 1:
                 removed = self.conversation_history.pop(1) # Remove message after system prompt
                 self.token_count -= removed["tokens"]
    
    def get_messages_for_api(self):
        """Get messages formatted for API call."""
        return [{"role": m["role"], "content": m["content"]} for m in self.conversation_history]

Copy

Temperature and Sampling Controls

Temperature and related parameters control the randomness and creativity of LLM outputs. Understanding these controls is essential for tailoring responses to different tasks.

Temperature Explained

Temperature controls the randomness of the model's token selection. It's typically set between 0.0 and 1.0 (or sometimes higher), with higher values producing more diverse and creative outputs.

Other Sampling Parameters

• Top-p (Nucleus Sampling): Considers only the most probable tokens that make up a certain cumulative probability mass (e.g., top_p=0.9 considers tokens covering 90% probability). Often used instead of or alongside temperature.
• Top-k: Considers only the top k most probable tokens. Less common now than top-p.
• Frequency Penalty: Decreases the likelihood of repeating tokens already generated.
• Presence Penalty: Decreases the likelihood of repeating tokens already present in the prompt.

Major LLM Providers and Models

Several major players offer powerful LLMs via APIs. Here's an overview of the landscape in 2025:

1. OpenAI

• Key Models: GPT-4 Turbo, GPT-3.5 Turbo
• Strengths: Strong general reasoning, instruction following, code generation
• Considerations: Can be expensive, closed-source

2. Anthropic

• Key Models: Claude 3 (Opus, Sonnet, Haiku)
• Strengths: Strong performance on benchmarks, large context window, focus on safety
• Considerations: Newer player, API access might be less widespread

3. Google

• Key Models: Gemini (Pro, Ultra)
• Strengths: Multimodal capabilities, integration with Google ecosystem
• Considerations: API offerings still evolving, performance varies

4. Mistral AI

• Key Models: Mistral Large, Mistral Small, Mixtral (Open Weights)
• Strengths: High performance open-weight models, competitive API offerings
• Considerations: European focus, rapidly evolving

5. Meta (via Hugging Face, etc.)

• Key Models: Llama 3 (70B, 8B)
• Strengths: Leading open-weight models, widely available for self-hosting
• Considerations: Typically require self-hosting or use via third-party APIs

Limitations and Challenges of LLMs

While powerful, LLMs have inherent limitations that agent developers must understand and mitigate:

Next Steps: Mastering Prompt Engineering

Understanding LLM fundamentals is the first step. The next crucial skill is prompt engineering—the art and science of crafting effective inputs to guide LLMs towards desired outputs.

In the next section, we delve into Prompt Engineering Essentials, providing techniques to unlock the full potential of LLMs within your AI agents.

Continue to Prompt Engineering →