open-endedness is essential for ASI
- potential metrics for building a system
roadmap to pluralistic alignment
- contains methods for evaluating coherence in an open-ended setting
- honestly should probably just evaluate on all these dimensions as well as product outcomes and see which ones matter
mech interp to interact with information in novel ways
- treating the LLM as a big impressive slab of marble that needs to be carved into a unique sculpture
i think as i write all this out, open-ended systems are really the only ones i’m interested in
- i think there exists opportunity to do open-ended things within closed-ended systems
can honcho improve your enterprise rag question-answering bot?
it would be nice if the devex could be as simple as passing to honcho before you send to your user… but that wouldn’t work because the honcho context could completely change the response… so really honcho should sit in the middle… honcho as a “thought” step to get context before generating your response might be the simplest it can be
i was wondering why we don’t just predict what the user is going to say verbatim… is it worth it to generate this proxy representation? the compute-utopian solution is to just generate a distribution of responses the user could provide and measure the distance between them… run backprop to update your model, and extract features from that model. is that the most effective representation of a user? is that the “carve from stone” approach, where we start with a general-purpose LM and update each per-user instance somehow to bias the way it predicts what they will say? then the “transparency” is found by running mech interp methods over those models to see in which ways they’re biased… this would probably require a deep understanding of a baseline model’s features in order to measure how a user’s changes make it deviate…
there’s probably a simpler approach that serves as a step function on this road to individual alignment. right now we’re at a prompting framework. the “model” of the user is a vector db of “facts” derived about the user.
see honcho application overview
was thinking about how cursor et. al. are effectively open-ended systems… they’re relatively well constrained environment-wise (to your editor) but are helping you achieve… basically anything? they’ve faced integration and context challenges but are relatively useful. but i think that’s the key here… the environment needs to be well defined.