How AI Actually Works: No Buzzwords, Just the Mechanics

Neural networks, transformers, hallucinations, energy costs — here is what is actually happening when you talk to an AI.

Neural Networks Are Not Magic

Let's start with what a neural network actually is: a mathematical function with millions or billions of adjustable parameters that takes input (text, images, audio) and produces output. The "neural" metaphor is loose — these networks were inspired by biological neurons but bear little resemblance to actual brain architecture. What they share with brains is that they learn from examples rather than being explicitly programmed with rules.

Training is the process of adjusting those parameters. You show the network millions of examples — text from the internet, labeled images, human conversations — and adjust the parameters to make the network's outputs more closely match the desired outputs. The adjustment algorithm is called backpropagation, developed by Rumelhart, Hinton, and Williams in 1986. The network doesn't understand the examples; it finds statistical patterns in them.

What "Large Language Model" Actually Means

A large language model (LLM) is a neural network trained to predict the next word (technically, the next "token") in a sequence. GPT-4, Claude, LLaMA — all of these are, at base, sophisticated next-token predictors. The "large" refers to parameter count: GPT-3 had 175 billion parameters; current frontier models are estimated in the trillions.

The transformer architecture, introduced by Vaswani and colleagues at Google in a 2017 paper titled "Attention Is All You Need," is the foundation of almost all current LLMs. The key innovation: attention mechanisms that allow the model to weigh relationships between any two positions in the input sequence, regardless of distance. When processing the word "it" in a sentence, attention allows the model to look back at every earlier word and weight which ones "it" likely refers to.

Inference vs. Training

Training and inference are fundamentally different operations. Training adjusts the model's parameters and requires massive computational resources — GPT-4's training was estimated to cost over $100 million in compute. Inference is running the trained model to generate outputs, which is much cheaper but still substantial at scale. When you send a message to an AI assistant, you're running inference on a model whose parameters were frozen at the end of training.

This distinction matters for understanding limitations. The model doesn't learn from your conversation (unless it's explicitly fine-tuned on user interactions). It doesn't "know" what you told it last week. Its knowledge is frozen at its training cutoff date.

Hallucinations: Why AI Confidently Makes Things Up

Hallucinations — the model producing fluent, confident, factually wrong output — are not bugs in the implementation but features of the architecture. An LLM is trained to produce plausible text, not accurate text. When asked about something it wasn't trained on, or something at the edges of its training distribution, it generates the statistically most likely continuation — which may be plausible-sounding nonsense.

The problem is compounded by the model having no internal flag for "I don't know this." It produces text regardless. Retrieval-augmented generation (RAG) — connecting the model to searchable knowledge bases — helps by grounding responses in retrieved facts, but doesn't eliminate hallucinations entirely.

The Energy Cost Nobody Talks About

Training GPT-3 was estimated to emit roughly 552 metric tons of CO2 equivalent — comparable to driving 120 cars for a year. A single ChatGPT query uses roughly 10 times the energy of a Google search. A 2023 Goldman Sachs report estimated that by 2030, AI data centers could account for 8% of US electricity demand. This is not an argument against AI development, but it is a cost that belongs in any honest accounting.

What AI Can and Cannot Do

Current AI excels at: pattern matching in text and images, generating fluent text in a given style, translation, code generation, summarization, and many classification tasks. It struggles with: reliable factual recall (hallucinations), precise arithmetic, long-horizon planning, genuine causal reasoning, and tasks requiring knowledge of recent events past training cutoff. The "reasoning" models show improvement on multi-step problems but remain unreliable on novel situations requiring true understanding rather than pattern matching.

FAQ

Q: Does an AI like ChatGPT actually understand language, or is it just pattern matching?

A: This is genuinely contested. Technically, LLMs perform very sophisticated pattern matching over statistical regularities in language. Whether this constitutes "understanding" depends on your definition of understanding — a question philosophers haven't resolved. LLMs pass many traditional benchmarks for comprehension, demonstrate apparent reasoning, and fail in ways that suggest they lack the grounded, embodied model of the world that human language understanding involves. The honest answer: we don't fully know what's happening inside these models.

Q: How do AI "reasoning" models like o1 or DeepSeek-R1 differ from standard language models?

A: Reasoning models use chain-of-thought prompting baked into training — they're trained to produce intermediate reasoning steps before giving final answers, often using reinforcement learning from verifiable outcomes. This improves performance on problems with checkable correct answers (math, coding, logic puzzles) significantly. The improvement on open-ended reasoning where there's no clear ground truth is less dramatic. The underlying architecture is the same transformer; the difference is in training methodology and inference behavior.

Curious minds belong together. Find people who ask the same questions on Good People Map.