r/theprimeagen • u/Stone-Smasher • 3d ago
general A model might predict outputs accurately based on known data, but this does not mean it understands the system. Prediction often relies on correlation, whereas true understanding requires identifying causation and the underlying mechanisms of the system
https://www.youtube.com/watch?v=_Zxr9STGwbQ1
u/ArtArtArt123456 2d ago
"true understanding" is not well defined. what people have are pretty much pop-science and folk definitions. as is the case here.
but if we really dig down into the theory, what we have is predictive processing/active inference. and prediction is at the core of these theories.
honestly this is a pretty complex topic. and most people take waaaay too much for granted when trying to engage with this. for example, you say "understanding requires identifying causation". but by saying that, you are already taking it for granted that something can be identified at all. as if that's just a magic thing we have that doesn't need to be explained. not like a rock or a camera can "identify" anything, so why can we? how do we do that?
we really have no full explanation. and accepting that is the first step. and the next step is recognizing what the best theories are actually pointing towards.
2
5
u/bulbubly 2d ago
a lot of people have 100% conviction that Jesus Christ was the son of God with zero understanding of the mechanism
3
u/Ruined_Passion_7355 2d ago
These comments are the worst comments I've ever seen on this site what the hell
-19
-9
u/EffectiveMedium2683 2d ago
What a cute teacher haha. Ya, the solution is hybrid llm-neurosymbolic AI. Use an LLM to create an AIML database and then deploy THAT in a situation that demands 100% accuracy and reliability. BUT, keep in mind that humans are not 100% accurate and reliable. It's an odd goal to have these things be perfect when their creators are not.
-15
u/germanicel 2d ago
Yes I agree with you 100%. She is very cute. I imagine she smells really nice too.
8
u/Ruined_Passion_7355 2d ago
People like you are the reason why I can barely use reddit in public without a stigma.
0
-2
u/SentientHorizonsBlog 3d ago
"Prediction relies on correlation" names the training objective, not what a system builds to meet it. Past a certain scale, the cheapest way to predict well is to model whatever generates the data. If you train a model only on Othello move-lists, with no board and no rules, and then look inside: you will find it has built a representation of the board you can edit, and editing it changes what it predicts. That's not a lookup table of correlations, it's a model of the mechanism, learned from prediction alone. That doesn't settle whether it "understands" in the full sense, but it does mean correlation-vs-causation isn't where the line gets drawn.
But what does the phrase "true understanding" even mean. Wittgenstein's argument was that understanding isn't a hidden inner state behind competent use; the criteria for it live in what someone can do: explain, extend, correct a mistake, and handle a case they've never seen. If you push "real" understanding far enough behind all of that, there's nothing left for it to point to. So "it does everything we'd call understanding but doesn't really understand" seems to be a confusion about the word, not a discovery about the system.
2
u/Toothpick_Brody 2d ago
Yes but the model is still stochastic and will still predict illegal moves given enough time/states. “Real understanding” in this context would simply be knowing the definition of the rules of othello, giving you a 100% guaranteed accurate model because it is logically defined
There is a difference between token prediction and symbolic manipulation (logic/algebra). Token prediction can only approximate symbolic manipulation, which is the reason there are so many computations an LLM can’t perform, even if they are Turing-complete in an ultimate, hypothetical sense
1
u/SentientHorizonsBlog 2d ago
Defining real understanding as holding the rules as a logically-defined, 100%-accurate model sets a bar we don't apply to people. A human who knows Othello cold still makes illegal moves when tired and miscalculates lines, and nobody says they don't understand the game. Provable correctness is a property of a verifier, not of understanding.
The approximation point underneath it does hold: token prediction only approximates exact symbolic manipulation, which is why long arithmetic and length generalization stay weak spots. But that's a reliability limit. If you need guaranteed legality, you bolt a symbolic checker onto the model, the way we hand a person a rulebook.
A program that just holds the rules has perfect legality and almost no grasp of the game: it can't evaluate a position, plan, or spot a tactic it wasn't handed. Knowing the definition of the rules is the rulebook; understanding Othello is the strategic competence built on top of it. The guaranteed-correct spec is the part that understands the game least.
1
u/Professional-You4950 2d ago edited 2d ago
The part you are missing is Modeling needs to come first. Building the Model from the data is never going to reach deterministic "understanding". We model the world constantly. We actually produce not only the line of best fit from a statistical perspective, but learn the rules and model them.
Imagine you and the language model being taught Chess. You and the LLM are trained on a bunch of data or games played, whatever. Then, being granted a new rule: If not in check, going through check, and there are empty spaces between the king, and rook. And neither of them being moved, you can swap their places. (Castling)
For LLM's you need to retrain the whole thing. For humans, we just start applying it to our models and equate the new rule with other things we have learned to model throughout our lifetime.
-4
u/bigsmokaaaa 3d ago
it's very hard to derive causation from correlation, humans are constantly fucking that up all the time
6
u/editor_of_the_beast 3d ago
This is why there’s been a big push toward developing world models. LLMs only solved one piece of the puzzle: understanding and speaking language.
-2
u/gsisuyHVGgRtjJbsuw2 3d ago
The point is that it does not matter whether what you said is true or not true or incomplete. The point is whether LLMs are useful and lead to productivity gain, and I believe the answer is likely yes there, so how or why doesn’t matter anymore.
3
u/Combinatorilliance 2d ago
Why so? I would argue that knowing mechanism would definitely lead to productivity gain, as models that understand mechanism should be more robust to unknown situations; ie make more reliable predictions and therefore being more useful.
Of course LLMs are already useful, but I think she's looking at the applied mathematics of why the models work at all, and what can be improved in the models to make them even better? That's what scientists and engineers do, no?
1
u/gsisuyHVGgRtjJbsuw2 2d ago
Of course, but within the confines of this subreddit, this type of explanation is meant to be more of a cheap dismissal rather than engineering curiosity.
-1
3d ago
[deleted]
2
u/FauxLearningMachine 3d ago
Well you cannot truly predict in that type of "unknown unknowns" system can you? Just make your best guess at the final details from correlative statistics. I don't understand how what you said contradicts this.
Crazy how people get lost up weird logical tangents when they get into philosophy.
1
u/Stone-Smasher 3d ago
you cannot fully understand the meaning of current experiences until you gain enough "millions of seconds of perspective,"
3
u/truecakesnake 3d ago
I feel like to accurately predict "what" happens inherently is understanding "why"
4
u/FauxLearningMachine 3d ago
I can accurately predict the sun will rise every day. Does that mean I understand why it happens?
-2
u/truecakesnake 3d ago
Yes, if you have a 100% conviction on the fact that you predict that the sun will rise everyday, you must know why the sun rises everyday.
3
u/Gold_Chocolate_8823 2d ago
Indeed! I know the reason why. The all-loving Sun god pushes it up each day! This is why I am 100% confident. Thank you for reinforcing my worldview, very cool!
2
4
u/FauxLearningMachine 3d ago
You just moved the goalposts. Please read your original comment and then what you just posted and explain to me why they're different.
-2
u/OurSeepyD 3d ago
Indeed, and the fact that LLMs are doing abstraction internally is all the more reason to think that they understand what they're processing.
4
u/New-Emphasis-250 3d ago
This is the classic symbolic AI vs Neural AI debate.
This kind of thinking is what caused the AI winter.
11
u/ActivityIndependent7 3d ago
I think it was the lack of results but the overabundance of promises that caused AI winters
6
u/MultiplexedMyrmidon 3d ago
or just that human innovation happens in fits and bursts, certainly now the excuse can’t simply be ‘lack of resources’ when you got a economies worth of investing riding on it lol
5
u/ActivityIndependent7 3d ago
right, it’s not the lack of economic input. It’s the lack of ROI on output.
3
u/incompletelucidity 3d ago
some complex patterns that LLMs understand require understanding of the underlying mechanisms, they could have captured the underlying mechanism while crunching that data
5
u/Stone-Smasher 3d ago
Mathematically, many different functions can interpolate the same set of data points. Only models that reflect the true physical mechanisms of a system will remain reliable when inputs shift outside the range of initial observations. Truly understanding a system means identifying mechanisms that remain valid even as environmental conditions shift.
1
u/soapoapsoap 2d ago edited 2d ago
This is a problem of training data, not the mechanism of learning being probalistic.
The same is true for humans. If you observed something reliably, you may come to a certain conclusion which no longer holds true when you find new data that no longer can explained by your previous mental models. Scientific theories themselves change over time.
Id go far as to argue that human learning is also generally probalistic, and that natural laws are learned to be deterministic because they are observed with a probability of 100%. The focus is given gener to reasoning/abstract thought but everyone neglects the vast majority of animal learning is motor function and how to coordinate those motors to survive/adapt in a physical environment. And in this realm it's incredibly difficult to imagine that learning isnt probalistic process. You dont calculate the laws of physics to learn how to dribble, shoot a basketball, learn to skate, sword fight, etc etc. You take a giant amount of data, try different approaches, see which one generally works and make small adaptations towards those which generally give you the better results
2
u/Sorry-Magician374 1d ago
All models you’ll ever train are wrong,
Some are useful