What Is a Token? (And Why AI Has a Memory Limit)

You're in the middle of a long conversation with ChatGPT. You've gone back and forth maybe a dozen times. Then you ask a follow-up question — and it gives you an answer that contradicts something it said thirty minutes ago.

Or worse: it acts like you never told it something important.

This isn't a glitch. It's how the technology actually works. And once you understand it, you'll use AI tools completely differently.

AI doesn't read words — it reads pieces of words

When you type something into an AI tool, it doesn't process your text the way you'd read a sentence. It breaks everything down into small chunks called tokens.

A token is roughly three or four characters. Sometimes it's a whole short word. Sometimes it's just a syllable or a piece of a longer word.

Here's a rough idea of what that looks like:

"chatbot" → 2 tokens (chat + bot)
"I love using AI tools" → about 5 tokens
"unbelievable" → about 3 tokens

You don't need to count tokens yourself. The point is: everything you type — and everything the AI responds with — gets broken into these small pieces and measured.

Why does this matter? Because AI has a limit.

Every AI model has what's called a context window — a maximum number of tokens it can hold in "working memory" at one time.

Think of it like a whiteboard. You can write a lot on a whiteboard. But eventually it fills up. Once it's full, something has to get erased to make room for new information.

For most AI tools, that context window is big — often hundreds of thousands of tokens, which is more than enough for most conversations. But it's not infinite.

When a conversation gets long — lots of back-and-forth, large documents you've pasted in, detailed instructions you've given — the whiteboard fills up. The AI starts to lose earlier parts of the conversation. It doesn't warn you. It just... forgets.

What "forgetting" looks like in practice

You might see this as:

The AI repeating something it already told you, as if it forgot it said it
Contradicting itself between early and late parts of a conversation
Ignoring instructions you gave at the start of the chat
Giving you a generic answer when it had been giving you personalized ones

None of this means the AI is broken. It means the whiteboard got full.

A session isn't forever

Here's another thing most people don't realize: each new conversation starts with a blank slate.

When you close a chat and start a new one, the AI has no memory of what you discussed before. The context window is wiped clean. It doesn't know your name, your preferences, your project, or anything else you told it last week.

Some tools are starting to add memory features — where the AI can remember a few key things across sessions. But even those have limits, and they're opt-in. By default, assume every new chat is a fresh start.

The practical takeaway

You now know two things that most people don't:

Long conversations get less reliable over time. As the chat fills up, earlier context gets pushed out. The AI at message 50 is working with less of your original information than the AI at message 5.
New sessions don't remember anything. Every chat starts from zero unless you re-establish context.

This doesn't make AI tools less useful. It just means there's a right and wrong way to work with them — especially for longer or more complex tasks.

The next step is learning how to manage your context on purpose: what to include, what to leave out, and when to start fresh. That's what the next post is about.

Want to get genuinely good at using these tools — not just the basics, but the habits that save real time? That's what the Clearly, AI course is built for. Plans start at $15/mo — see what's included.