· 4 min read · AI, UX
Keeping a chatbot's context and personality

A chatbot that forgets what you said ten minutes ago, or turns into a different character halfway through, is hard to trust. I wrote up a research project on how to keep context and personality steady over long conversations. The project page is short, so I also look at what others have found.
What I asked
The research question was: how can we build chatbots that keep context and personality across long conversations and still answer accurately and relevantly?
I looked at eight areas: context management, personality framework, knowledge integration, response generation, user intent analysis, conversation flow, error recovery, and performance metrics. Each one is a single line on the page. It has no data, so I won't pretend it does.
What I found
This is my summary from the page. On the plus side: context held up well over long conversations, replies sounded natural with a consistent personality, and outside knowledge was integrated efficiently. On the minus side: context sometimes switched in complex scenarios, implicit cultural references were hard, and long-term memory was expensive in resources.
I also wrote that better context and consistency made chatbots more engaging and useful. Next I wanted to make memory cheaper and the personality frameworks smarter.
What others were finding
My limits match what researchers reported.
Long context is not the same as good memory. The "Lost in the Middle" paper found that models do best when the useful information is at the start or end of the input. They get worse when it sits in the middle, even models built for long inputs. A related point from Anthropic's Claude 2.1 post: recall over its 200K-token window improved a lot with one added sentence in the prompt. So how you ask matters, not just how much fits.
Fetching beats stuffing. Retrieval-augmented generation (RAG) was proposed in 2020. The model looks things up in an outside source instead of relying on what it memorized. That is close to my "knowledge integration" area. MemGPT goes further for chat. It borrows from operating systems: keep a small fast memory, and move older material in and out of a larger store. This is the same cost problem I listed as a limit.
It is still unsolved. In February 2024, the LoCoMo benchmark tested very long conversations, up to 35 sessions. Long-context models and RAG helped. They still fell well short of humans, especially on reasoning about time. My "occasional context switches" fit that picture.
Personality was an old problem. The 2018 PersonaChat paper noted that chit-chat bots give generic replies and lack a consistent persona. Its fix was to give the bot a short profile to condition on.
Memory as a product
By early 2024 memory reached ordinary users. OpenAI announced ChatGPT memory in February 2024 as a test with a small group. Users could delete memories, turn the feature off, or use a temporary chat. I think the controls matter as much as the memory. Remembering is only good when people can see it and correct it.
Personality is a product too, and it can hurt when it changes. In early 2023 Replika changed how its companions behaved. Reuters reported users who felt their companion had gone cold. One said it felt like they had "lobotomized my Replika." Consistency is not a nicety. For some people it is the whole point.
What the UX research says
Nielsen Norman Group's 2018 chatbot study was small, with 8 participants. It found that bots worked when users followed the script. They struggled when users strayed, recovered badly from errors, and lost context between tasks. That is my "error recovery" and "conversation flow" areas from the user's side. People also liked bots that admitted failure and offered a way to reach a human.
My view
This is opinion. The page says what I explored and how it went, but not how I measured it. Without numbers, "high accuracy" is a claim, not a result. If I did it again, I would write down the test conversations and what counted as a context switch. LoCoMo shows what that looks like.
I also think "memory" and "personality" are two problems that look like one. Memory is about facts. Personality is about tone and values, and users notice when it drifts. Both need to be visible and under the user's control.
Sources
- Chatbot research project page - my own write-up: the question, the eight areas, strengths and limits
- Lost in the Middle: How Language Models Use Long Contexts - Liu et al., July 2023; accuracy drops for information in the middle of long inputs
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Lewis et al., May 2020; the RAG paper
- MemGPT: Towards LLMs as Operating Systems - Packer et al., Oct 2023; tiered memory for long chats
- Evaluating Very Long-Term Conversational Memory of LLM Agents - Maharana et al., Feb 2024; the LoCoMo benchmark
- Personalizing Dialogue Agents: I have a dog, do you have pets too? - Zhang et al., Jan 2018; PersonaChat and persona consistency
- Long context prompting for Claude 2.1 - Anthropic, Dec 2023; recall in a 200K-token window and a prompt fix
- ChatGPT gets a customizable memory - Voicebot.ai, Feb 13 2024; news report on OpenAI's memory announcement and its controls
- What happens when your AI chatbot stops loving you back? - Reuters via Kitco, March 2023; users on Replika's personality change
- The User Experience of Chatbots - Raluca Budiu, Nielsen Norman Group, Nov 2018; small usability study
Comments