Key takeaways
- A word gets its meaning from the sentence context and the patterns learned from training data.
- Attention mechanisms help LLMs connect the relevant words in a context.
- Chain-of-thought prompting gives the model planning context before its final answer.
When we think about how language works, it is clear that a word’s meaning does not exist in isolation. Words like “chair” or “dog” do not mean anything by themselves. They are only letters we grouped together to represent concepts we recognize.
The meaning of words comes from two key factors: the other words in the sentence and the other words in every sentence you have ever heard or said. This dual source of meaning explains why large language models (LLMs) perform so well when powered by (1) attention mechanisms and (2) extensive training on millions, or even trillions, of sentences.
Isn’t it ironic that a word’s meaning comes from everything except the word itself?
These two key factors reflect how humans understand language. We draw meaning from (1) the context we receive and (2) our past experiences.
It is common to think that autoregressive LLMs are only advanced autocomplete systems. In reality, these LLMs do much more than predict the next word from a simple statistical probability, as basic autocomplete systems do. LLMs do not merely imitate understanding. They achieve it in their own way.
Note: Autoregressive models generate text one word at a time, predicting each word from the preceding context.
Unlike LLMs, we think and plan before we speak. We do not think autoregressively. We do not wait until we say one word before we start thinking about the next. We build ideas and concepts, then the words and sentences follow. This leads to one of the most fascinating capabilities of LLMs, which we can observe through chain-of-thought (CoT) prompting.
CoT lets these models imitate this planning behavior. They can plan their answers, moving away from the simplistic “word by word” predictions typical of autoregressive models. This capability brings LLMs even closer to a real form of intelligence, depending on how you define intelligence.
Instead of generating words directly, one at a time, they develop a plan and add it to their context, a little like we think before speaking. This is why CoT, especially when combined with a few examples through few-shot prompting, as we covered in a recent edition, makes models so powerful. It reflects our thought process while imitating our ability to adapt to different situations.
🙏 We want to thank Geoffrey Hinton for his excellent presentation at the Ai4 event we attended, which inspired this article.
FAQ
Is an LLM only an advanced autocomplete system?
No. It does predict the next token, but it uses contextual representations and patterns learned at scale to produce answers that are far more structured than basic statistical completion.
Where does a word's meaning come from for a language model?
It comes from the other words in the context and the relationships learned from the millions or billions of sentences used during training.
What does attention do in an LLM?
Attention helps the model give more weight to the parts of the context that matter for understanding a word or generating what comes next.
What is chain-of-thought prompting?
It is a way to have the model produce intermediate steps before its answer, giving it an explicit plan inside its own context.
Does chain-of-thought make a model truly intelligent?
It often improves planning and apparent reasoning, but the answer still depends on the prompt, the learned data, and the model's limitations.