Comment on LLMs and Humans are Cognitive Cousins – Sean Carroll podcast w/Dr. Chandra Sripada
TehPers@beehaw.org 2 weeks agoLlm’s that only have one glyph per token can easily count this
[Citation needed]
There’s no difference between counting tokens containing the letter “r” and tokens solely representing the letter “r”. To begin with, this assumes the way we encode words in our brain is by tokenizing each letter, which we pretty much know to be false (we can process words without seeing evry ltr of th wrd and even think about them without it being mentioned).
yeah we train in bursts because it’s computationally cheaper, this hardly means anything important.
No, this is not how neural networks work at all.
We train in bursts because we need a ground truth/scoring function to guide the learning, which means we need a way to guide the model in the right direction. Even with unsupervised learning, the model has a way to determine whether it’s going in the right direction, like with GANs.
Neural networks are also incapable of synthesizing completely new skills from its own weights. Humans have been doing that for the entirety of human history.
nymnympseudonym@piefed.social 2 weeks ago
These are completely orthogonal things.
We train in bursts ("epochs") because empirically it leads to more rapid convergence on an acceptable error minimum.
The fact that a scoring function is needed is almost a tautology. You have to have some differentiable function to say what direction is lower error. Typically you segment your training corpus, and use the scoring function to get the error for one batch at a time to a certain threshold before proceeding to the next batch. Then go to the first batch and push the error bar even lower, and so on.
Kinda like tightening nuts on a car wheel. You don’t fully tighten the first one first; you have to wiggle the system into its final configuration.
TehPers@beehaw.org 2 weeks ago
We’re comparing this to the observable behaviors of humans, though. Humanity engages in unsupervised learning without a scoring function (let alone a differentiable one) and, as a result, can not only train on the fly, but can also find its own ways to self-evaluate its learning. Humans also have hormones which change how the brain works even further.
This is also all disregarding the ability for the human mind to become faster and more accurate on recall when it has more knowledge to search in a field. LLMs can’t recall very effectively, instead becoming less effective the more information it needs to search.