Really needs a comparison to the "megaprompt" itself (i.e. "here is a tweet, rate it as ironic or not, considering the following properties; explain your reasoning then output your final answer at the end"). I bet that would get you very far towards the logistic classifier, and would generalize much better out of distribution.
Why not use a text embedder for the unstructured data and concatenate with the structured data?
For example you could freeze most of the layers of the embedder but let the final ones learn. Then you wouldn’t need to do either feature or prompt engineering?
I really wish people would define terms when using math. What is y? What is LLM(x)? Presumably it evaluates to some real number so that it can be fed to the logistic sigmoid function. If it is the logistic function, then why does beta going to infinity matter? It seems to just collapse the output of the sigmoid function to 1 and make the value of LLM(x) meaningless instead of their claim that it recovers the LLM classifier. What is the function I()?
Maybe these are well understood terms in some field? Maybe I'm just lost?
I don't understand the point they are trying to make.
It's very often (always?) the case that something general also solves particular problems.
A sorting algorithm is an implementation of min()
A parser also is a syntax checker.
A route planner is a reachability checker.
A computer algebra system is a basic arithmetic calculator.
A general constraint solver is a Soduku hint maker.
It's true that LLM output can be used as an input to another classifier, this is also true of any classifier. The improvement on top of the straight LLM classification is relatively small, and I would argue that working on the prompt or just including in the prompt for the LLM what features might be useful to consider would likely work even better.
Fundamentally I read this article as: We want to build a simpler, dumbed down clone of Mathematica, so we cobbled together the following pieces... We also needed a way to do arithmetic, so we also include a copy of Mathematica to do basic arithmetic.
I took the point as: don't make the LLM the classifier. Use it to turn messy input into useful features, then let a normal model make the actual decision. That gives you thresholds/calibration you can inspect.
What I'm not sure about is how stable those features are when you switch the underlying LLM or model version.
Much like how you shouldn't ask the LLM to solve the (repeated, logical) problem, but you should instead prompt it to generate code that you can inspect/test/fix/reuse.
In my work on LLM as a judge, I prefer to use LLM decisions as features in a downstream classic ML model for the final decision. It works really well
https://softwaredoug.com/blog/2025/01/21/llm-judge-decision-...
Why does he first ask to label "ironic" or "not", and then answer the feature questions? Wouldn't it be better to reverse the order?
Really needs a comparison to the "megaprompt" itself (i.e. "here is a tweet, rate it as ironic or not, considering the following properties; explain your reasoning then output your final answer at the end"). I bet that would get you very far towards the logistic classifier, and would generalize much better out of distribution.
Why not use a text embedder for the unstructured data and concatenate with the structured data?
For example you could freeze most of the layers of the embedder but let the final ones learn. Then you wouldn’t need to do either feature or prompt engineering?
You can also get LLM to optimize rules for a rules engine iteratively against some dataset.
It’s sort of like memoizing or distilling the knowledge. Works really well for certain type of problems.
Well timed article. :)
I really wish people would define terms when using math. What is y? What is LLM(x)? Presumably it evaluates to some real number so that it can be fed to the logistic sigmoid function. If it is the logistic function, then why does beta going to infinity matter? It seems to just collapse the output of the sigmoid function to 1 and make the value of LLM(x) meaningless instead of their claim that it recovers the LLM classifier. What is the function I()?
Maybe these are well understood terms in some field? Maybe I'm just lost?
I don't understand the point they are trying to make.
It's very often (always?) the case that something general also solves particular problems.
It's true that LLM output can be used as an input to another classifier, this is also true of any classifier. The improvement on top of the straight LLM classification is relatively small, and I would argue that working on the prompt or just including in the prompt for the LLM what features might be useful to consider would likely work even better.Fundamentally I read this article as: We want to build a simpler, dumbed down clone of Mathematica, so we cobbled together the following pieces... We also needed a way to do arithmetic, so we also include a copy of Mathematica to do basic arithmetic.
I took the point as: don't make the LLM the classifier. Use it to turn messy input into useful features, then let a normal model make the actual decision. That gives you thresholds/calibration you can inspect.
What I'm not sure about is how stable those features are when you switch the underlying LLM or model version.
Much like how you shouldn't ask the LLM to solve the (repeated, logical) problem, but you should instead prompt it to generate code that you can inspect/test/fix/reuse.
> Calibration / Threshold Control
The amount of thinking is relatively calibrated. Ask an obvious classification, you get an instant answer. Ask a tricky one, much more thinking.