evals for open-endedness

questions to ask…

are we predicting the user’s behavior better? are we doing what the user wants? are we reasoning about their mental state correctly?

random idea — can we treat this as a loom-like simulation problem?

• user says x
• AI says Y
  • we predict user is going to say (Z1, …, Zn)
• user says x2
  • we see which Z it fit
  • reason about why
  • store that fact F1
• AI says Y2 …
• user says xn
  • we see which Z it fit
  • reason about why
  • store that fact Fn

then you have a distribution of facts that represent a person. what can you do with that? we stored those facts about the user in a vector DB that grew over time… this is an open research problem, too. how do you best manage that user representation as it grows? is there a time decay? how do you retrieve over it? do you train an LLM on it?

recap from my claude session…

---

Here’s a concise to-do list for your experiment:

1. Prepare dataset of user-AI conversations
2. Generate multiple thought predictions for each user message
3. Create XGBoost model to evaluate thought-response pairs
4. For each conversation: a. Generate thoughts b. Evaluate thoughts c. Rank thoughts
5. Aggregate results across conversations
6. Use top-ranked thoughts to fine-tune LLM
7. Implement chosen method for near-real-time updates
8. Compare original and updated LLM performance
9. Iterate and refine process

---

what i’m still unsure of is the best way to…

1. represent the user (aka what is the user model)
2. test the user model beyond its construction (are there held-out conversations to test ToM on?)

2024-07-15

claude suggested i create a vector that’s the weighted average of all the thoughts that have successfully predicted the user’s response

this is interesting

it’d be like a user-theory-of-mind control vector that you could append to the user’s input?

that’s actually very interesting

i’m looking for a simple service to provide, and a ToM control vector would be pretty simple

it still kinda creates this problem where the application doesn’t really control the output

asked claude about this…

this would be really interesting if it works

the four areas of ML for honcho

1. mental state imputation (ToM)
   1. thought prediction
   2. mental state imputation
   3. etc
2. user representation
   1. A collection of facts that have been embedded?
   2. A vector representing the weighted average of all the successful thoughts?
   3. SAE on top of ToM Model?
   4. etc
3. using the user representation
   1. RAG over the vectors of facts?
   2. a thought endpoint? use the user vector to generate a personalized ToM thought?
   3. or have us generate the whole response w/ thought?
   4. etc
4. maintaining the user representation
   1. how do we update the representation of the user?
   2. VoE?
   3. updating the vector?
   4. editing the language model?
   5. seriously depends on what form the user representation take