· 4 min read · AI, UX

Keeping a chatbot's context and personality

A close-up of a rotary telephone dial
Photo: Daria Nepriakhina, CC0

A chatbot that forgets what you said ten minutes ago, or turns into a different character halfway through, is hard to trust. I wrote up a research project on how to keep context and personality steady over long conversations. The project page is short, so I also look at what others have found.

What I asked

The research question was: how can we build chatbots that keep context and personality across long conversations and still answer accurately and relevantly?

I looked at eight areas: context management, personality framework, knowledge integration, response generation, user intent analysis, conversation flow, error recovery, and performance metrics. Each one is a single line on the page. It has no data, so I won't pretend it does.

What I found

This is my summary from the page. On the plus side: context held up well over long conversations, replies sounded natural with a consistent personality, and outside knowledge was integrated efficiently. On the minus side: context sometimes switched in complex scenarios, implicit cultural references were hard, and long-term memory was expensive in resources.

I also wrote that better context and consistency made chatbots more engaging and useful. Next I wanted to make memory cheaper and the personality frameworks smarter.

What others were finding

My limits match what researchers reported.

Long context is not the same as good memory. The "Lost in the Middle" paper found that models do best when the useful information is at the start or end of the input. They get worse when it sits in the middle, even models built for long inputs. A related point from Anthropic's Claude 2.1 post: recall over its 200K-token window improved a lot with one added sentence in the prompt. So how you ask matters, not just how much fits.

Fetching beats stuffing. Retrieval-augmented generation (RAG) was proposed in 2020. The model looks things up in an outside source instead of relying on what it memorized. That is close to my "knowledge integration" area. MemGPT goes further for chat. It borrows from operating systems: keep a small fast memory, and move older material in and out of a larger store. This is the same cost problem I listed as a limit.

It is still unsolved. In February 2024, the LoCoMo benchmark tested very long conversations, up to 35 sessions. Long-context models and RAG helped. They still fell well short of humans, especially on reasoning about time. My "occasional context switches" fit that picture.

Personality was an old problem. The 2018 PersonaChat paper noted that chit-chat bots give generic replies and lack a consistent persona. Its fix was to give the bot a short profile to condition on.

Memory as a product

By early 2024 memory reached ordinary users. OpenAI announced ChatGPT memory in February 2024 as a test with a small group. Users could delete memories, turn the feature off, or use a temporary chat. I think the controls matter as much as the memory. Remembering is only good when people can see it and correct it.

Personality is a product too, and it can hurt when it changes. In early 2023 Replika changed how its companions behaved. Reuters reported users who felt their companion had gone cold. One said it felt like they had "lobotomized my Replika." Consistency is not a nicety. For some people it is the whole point.

What the UX research says

Nielsen Norman Group's 2018 chatbot study was small, with 8 participants. It found that bots worked when users followed the script. They struggled when users strayed, recovered badly from errors, and lost context between tasks. That is my "error recovery" and "conversation flow" areas from the user's side. People also liked bots that admitted failure and offered a way to reach a human.

My view

This is opinion. The page says what I explored and how it went, but not how I measured it. Without numbers, "high accuracy" is a claim, not a result. If I did it again, I would write down the test conversations and what counted as a context switch. LoCoMo shows what that looks like.

I also think "memory" and "personality" are two problems that look like one. Memory is about facts. Personality is about tone and values, and users notice when it drifts. Both need to be visible and under the user's control.

Sources

Comments