So predicting the next word given all humanity’s knowledge is surely going to max out at slightly less good (we probably can’t get perfect data) than the best human in any specific field. What test does the AI do to be able to understand it is improving? At some point it becomes impossible to know that the output is actually better right?
I dunno if it would max out at slightly less good, I imagine it would max out around the distribution of it's data set, which could be significantly worse than top experts.
At what point is human intelligence going to hold back machine intelligence?
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
I love your thought experiment. May I counter, what purpose does such a machine have to us, that can think beyond our needs? Sorry, but to reference the great Rick and Morty, "your purpose is to pass the butter".
The model will have to convince the human that it's making the right kind of progress. That will necessarily become part of the improvement loop - either implicitly (human trusts RSI) or explicitly (human gatekeeps every major decision).
Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.
Color me skeptical. LLMs seem to make writing code faster, so of course that means that people can iterate on ideas faster, but I have yet to see actual creative output from an LLM that wasn't coached into it or random juxtaposition.
Was the creation of writing as such coached into humans or a random juxtaposition? Could all human inventions just be coached into humans (who coached?) or be random juxtapositions?
People are always using this example because of the "LLMs can't jump paper". Suffice to say - it's not that simple, and special relativity was definitely an incremental improvement to well-studied theory that was being developed by dozens of the leading physicists of the day.
They're doing RL on open problems these days, not just next token prediction.
Citation needed
Imagine you are evaluating what the machine should do when it is improving itself. It does a bunch of work and returns with "I supervaluated the liminal overdecomposition from the previous homological calibulation pass. It shows us that subtransitory mulutination will underspecify the tensor of stermullification. Where do you want to go from here?"
It will be like when you are reading a Wikipedia about a topic you don't understand. You follow the links, and you get more questions with more links. Your whole day is taken up following links, to the point where you forgot the original question.
Except this time, all the words come from the AI's work. You can't refer to an external authority who has already been there and can tell you what to do.
The AI needs you to tell it whether it is more intelligent than it was before, but you don't know, because you can't follow its reasoning any more. It's like an ordinary person trying to hire a math professor, there's just no way to do it.
But whereas a human math prof can evaluate another one, a machine intelligence can't evaluate another one, by construction. Because it's still usefulness to humans that is the evaluation criterion.
For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.
To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.
Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...