It's a very interesting concept, and I signed up to try it. However, after seeing the landing page, my first question was:
"Where's the data on accuracy?"
Backtesting is difficult to do correctly with LLMs, but because this is marketed as being for macro investing, I would expect to see a level of rigor and quantitative analysis consistent with that.
The Monte Carlo simulation engine sounds really cool, but is there evidence to indicate that it generates superior results to expert predictions, or to LLMs alone?
I actually think it would be totally fine for your beta version to have low accuracy numbers. After all, this seems to be something in the very early stages. But to have no quantitative analysis of your system's performance definitely makes me uneasy to trust it.
> because this is marketed as being for macro investing, I would expect to see a level of rigor and quantitative analysis consistent with that.
Thanks for bringing this up - while we talk about Soros' forecasts and comparing them against those of an LLM, in the end Soros is not a forecasting tool, it's an analytical framework.
There is a gap between quant modeling and geopolitical analysis that we seek to fill. Specifically, quant models are great at capturing statistical regularities in financial time series but typically treat geopolitical shocks as exogenous noise. Meanwhile, geopolitical analyses in the policy and intelligence communities (with the exception of Bueno de Mesquita [BdM]'s work) provide deep contextual reasoning but rarely produce probabilistic scenario structures or asset-level transmission mappings that can directly inform capital allocation.
We will be shortly publishing a technical preprint laying out the Soros framework in full, but the TL;DR is: we model geopolitical events (or crises in the literature) as partially observed ("fog of war") stochastic games with multiple actors jostling for control over resources. We map out actors across various axes (think of these as actor embeddings), identify key decision points, and enumerate paths across them to estimate scenario probabilities. The scenarios in turn have associated transmission flows and market implications. We will evaluate those as mentioned in the sibling comment. Happy to discuss more.
First, thank you so much for signing up to try out Soros!
You are absolutely right, of course, to ask about accuracy. TL;DR: we don't have any formal calibration data yet.
The reason why is interesting, though, and it strikes at the heart of global macro investing in particular: things change, often, and sometimes dramatically. Basically, geopolitical "events" are really smeared across time (and sometimes space). Each event update can lead to a cascade of new scenarios branching off and older ones dying out, each with implications on capital flow. It's difficult to disentangle, which is why our preference has been to enable the system itself to monitor feeds, but also update its alerts as it deems fit, and re-run the analysis when it feels there's been enough of a change of state (pun not intended).
One markets-focused eval we have been building towards (and apparently you have been thinking of as well) is comparing against LLMs. Our plan is to run simultaneous comparisons against a variety of frontier models, armed with the same information that we provide Soros, but without the structural framework and simulation engine we've built though. Ideally we want to map out the Pareto frontier of model capability vs realized returns, and examine performance over horizons, asset classes, and so on, and have concrete numbers on where Soros pushes the curve outwards.
This is being built :), and we hope to get there in the coming few weeks!
* This brings us to a larger question - why did we build Soros?
First, let's address the elephant in the room: we were inspired by George Soros' theory of reflexivity and how human tendencies affect markets more prominently than expected. Yes, there's a corny backronym [0]. No, this is not a political statement or endorsement of his views.
Coming back to the main point, we (the founding team at Lookback Labs) have both spent a long time at the intersection of financial markets, technology, and machine learning. During that time, one key thing that kept bothering us [1] was simply this: when a geopolitical crisis breaks, an investor's actual problem is not really to find out "what is happening now" — it's more of "which scenario plays out, how likely is each one, and what do I buy, sell, or hedge under each? For how long?"
There are a ton of existing tools and services that seek to answer the first question reasonably well (newsletters such as StratFor, publications such as Foreign Affairs and Foreign Policy, Bloomberg terminals for breaking news, etc.).
None of these answer the other questions particularly deftly. Sure, one can engage with ChatGPT (or Claude if one prefers), and play through multiple scenarios. You will, of course, miss out on the grounded structural model that powers Soros' analysis, along with the simulations that serve up the relative probability estimates.
Also, one of the worst things purely LLM-based ad hoc frameworks do is assume that countries are monolithic decision-making units from a game-theoretic perspective. This is hardly the case - "Iran" doesn't make choices, Mojtaba and the IRGC faction does. "China" doesn't decide, the Politburo Committee does. And so on.
There are of course formal analytical frameworks that dig deeper, studying groups, factions, organizations that are jostling to gain control (Bruce Bueno de Mesquita's Expected Utility Model and selectorate theory [2] is the most academically serious and is a prime inspiration for our system design), but they are extraordinarily hard to operationalize in real time, and produce no market implications.
To sum up, the choices are stark: ask AI and hope for the best, or build out your own systematic framework to organize evidence, assumptions, and implications. We chose the latter path.
Zooming out, our mission at Lookback Labs (https://www.lookbacklabs.com/) is to build "the intelligence layer for AI-native investing"; accordingly, Soros is the first of several agentic systems that we are designing across the systematic and discretionary spaces, that are both usable and useful from the get go, and not merely demo eye candy.
* Some minor details:
(1) We are currently in private beta for Soros and are onboarding selectively.
(2) The static demo is not completely static; you can still chat with the analysis (up to 20 messages a day per IP).
(3) We are still working on pricing: something that captures the value Soros provides.
(4) We want this to work for individual investors as well, not just institutional desks, and would love to price accordingly.
We're curious to hear what the HN community thinks about our approach. AUA!
Feel free to reach out offline if you'd like! We are, sadly enough, on LinkedIn, but are also available via email (anshuman/karen@lookbacklabs.com)
PS: As is probably obvious to the diligent reader :), every token in this post has been lovingly handcrafted by the Lookback Labs team.
[0] Scenario-Oriented Reasoner for Opportunity Synthesis. Lol.
[1] Many things bothered us. Buy us drinks, get stories.
[2] We heartily recommend two of BdM's books: "Predicting Politics" and "The Dictator's Handbook"
Hmmm, good question. I think one interesting incident for us was when we saw scenario probabilities being updated near last Friday EOD for the US-Iran conflict, biased towards further kinetic action by the US around Kharg island (?). This was basically captured from changes in odds for Polymarket events that the system was tracking.
The news came in a few minutes later, post equity market closing.
> So someone monitoring Polymarket could have reached the same conclusion?
Maybe? If they are professionally trading prediction markets, I'm pretty sure that would be the case. Polymarket especially is a great source of insider traded information, as you pointed out.
We do near realtime tracking of most major markets, plus X accounts that Soros identifies as being important. The system also composes search queries per analysis, along with frequency of scanning, and that's run as requested. (We use a mix of Perplexity and other smaller search providers, along with Exa via OpenRouter's integration.)
Other inputs might be direct statements from leaders involved in the conflict, especially Trump. Also maybe bond market and oil market price movements?
Then you would want to generate an alert when you an actionable prediction. You don't want the user to have to prompt the AI. It needs to be running in the background, having been prompted on the scenarios to monitor?
It's a very interesting concept, and I signed up to try it. However, after seeing the landing page, my first question was:
"Where's the data on accuracy?"
Backtesting is difficult to do correctly with LLMs, but because this is marketed as being for macro investing, I would expect to see a level of rigor and quantitative analysis consistent with that.
The Monte Carlo simulation engine sounds really cool, but is there evidence to indicate that it generates superior results to expert predictions, or to LLMs alone?
I actually think it would be totally fine for your beta version to have low accuracy numbers. After all, this seems to be something in the very early stages. But to have no quantitative analysis of your system's performance definitely makes me uneasy to trust it.
> because this is marketed as being for macro investing, I would expect to see a level of rigor and quantitative analysis consistent with that.
Thanks for bringing this up - while we talk about Soros' forecasts and comparing them against those of an LLM, in the end Soros is not a forecasting tool, it's an analytical framework.
There is a gap between quant modeling and geopolitical analysis that we seek to fill. Specifically, quant models are great at capturing statistical regularities in financial time series but typically treat geopolitical shocks as exogenous noise. Meanwhile, geopolitical analyses in the policy and intelligence communities (with the exception of Bueno de Mesquita [BdM]'s work) provide deep contextual reasoning but rarely produce probabilistic scenario structures or asset-level transmission mappings that can directly inform capital allocation.
We will be shortly publishing a technical preprint laying out the Soros framework in full, but the TL;DR is: we model geopolitical events (or crises in the literature) as partially observed ("fog of war") stochastic games with multiple actors jostling for control over resources. We map out actors across various axes (think of these as actor embeddings), identify key decision points, and enumerate paths across them to estimate scenario probabilities. The scenarios in turn have associated transmission flows and market implications. We will evaluate those as mentioned in the sibling comment. Happy to discuss more.
First, thank you so much for signing up to try out Soros!
You are absolutely right, of course, to ask about accuracy. TL;DR: we don't have any formal calibration data yet.
The reason why is interesting, though, and it strikes at the heart of global macro investing in particular: things change, often, and sometimes dramatically. Basically, geopolitical "events" are really smeared across time (and sometimes space). Each event update can lead to a cascade of new scenarios branching off and older ones dying out, each with implications on capital flow. It's difficult to disentangle, which is why our preference has been to enable the system itself to monitor feeds, but also update its alerts as it deems fit, and re-run the analysis when it feels there's been enough of a change of state (pun not intended).
One markets-focused eval we have been building towards (and apparently you have been thinking of as well) is comparing against LLMs. Our plan is to run simultaneous comparisons against a variety of frontier models, armed with the same information that we provide Soros, but without the structural framework and simulation engine we've built though. Ideally we want to map out the Pareto frontier of model capability vs realized returns, and examine performance over horizons, asset classes, and so on, and have concrete numbers on where Soros pushes the curve outwards.
This is being built :), and we hope to get there in the coming few weeks!
* This brings us to a larger question - why did we build Soros?
First, let's address the elephant in the room: we were inspired by George Soros' theory of reflexivity and how human tendencies affect markets more prominently than expected. Yes, there's a corny backronym [0]. No, this is not a political statement or endorsement of his views.
Coming back to the main point, we (the founding team at Lookback Labs) have both spent a long time at the intersection of financial markets, technology, and machine learning. During that time, one key thing that kept bothering us [1] was simply this: when a geopolitical crisis breaks, an investor's actual problem is not really to find out "what is happening now" — it's more of "which scenario plays out, how likely is each one, and what do I buy, sell, or hedge under each? For how long?"
There are a ton of existing tools and services that seek to answer the first question reasonably well (newsletters such as StratFor, publications such as Foreign Affairs and Foreign Policy, Bloomberg terminals for breaking news, etc.).
None of these answer the other questions particularly deftly. Sure, one can engage with ChatGPT (or Claude if one prefers), and play through multiple scenarios. You will, of course, miss out on the grounded structural model that powers Soros' analysis, along with the simulations that serve up the relative probability estimates.
Also, one of the worst things purely LLM-based ad hoc frameworks do is assume that countries are monolithic decision-making units from a game-theoretic perspective. This is hardly the case - "Iran" doesn't make choices, Mojtaba and the IRGC faction does. "China" doesn't decide, the Politburo Committee does. And so on.
There are of course formal analytical frameworks that dig deeper, studying groups, factions, organizations that are jostling to gain control (Bruce Bueno de Mesquita's Expected Utility Model and selectorate theory [2] is the most academically serious and is a prime inspiration for our system design), but they are extraordinarily hard to operationalize in real time, and produce no market implications.
To sum up, the choices are stark: ask AI and hope for the best, or build out your own systematic framework to organize evidence, assumptions, and implications. We chose the latter path.
Zooming out, our mission at Lookback Labs (https://www.lookbacklabs.com/) is to build "the intelligence layer for AI-native investing"; accordingly, Soros is the first of several agentic systems that we are designing across the systematic and discretionary spaces, that are both usable and useful from the get go, and not merely demo eye candy.
* Some minor details:
(1) We are currently in private beta for Soros and are onboarding selectively.
(2) The static demo is not completely static; you can still chat with the analysis (up to 20 messages a day per IP).
(3) We are still working on pricing: something that captures the value Soros provides.
(4) We want this to work for individual investors as well, not just institutional desks, and would love to price accordingly.
We're curious to hear what the HN community thinks about our approach. AUA!
Feel free to reach out offline if you'd like! We are, sadly enough, on LinkedIn, but are also available via email (anshuman/karen@lookbacklabs.com)
PS: As is probably obvious to the diligent reader :), every token in this post has been lovingly handcrafted by the Lookback Labs team.
[0] Scenario-Oriented Reasoner for Opportunity Synthesis. Lol.
[1] Many things bothered us. Buy us drinks, get stories.
[2] We heartily recommend two of BdM's books: "Predicting Politics" and "The Dictator's Handbook"
Ok, do you have any interesting, unusual insights generated by Soros?
Hmmm, good question. I think one interesting incident for us was when we saw scenario probabilities being updated near last Friday EOD for the US-Iran conflict, biased towards further kinetic action by the US around Kharg island (?). This was basically captured from changes in odds for Polymarket events that the system was tracking. The news came in a few minutes later, post equity market closing.
So someone monitoring Polymarket could have reached the same conclusion?
But Soros can process many more inputs than a human analyst?
Polymarket seems like a very good input, because of the probable insider trading. What other inputs do you use?
> So someone monitoring Polymarket could have reached the same conclusion?
Maybe? If they are professionally trading prediction markets, I'm pretty sure that would be the case. Polymarket especially is a great source of insider traded information, as you pointed out.
We do near realtime tracking of most major markets, plus X accounts that Soros identifies as being important. The system also composes search queries per analysis, along with frequency of scanning, and that's run as requested. (We use a mix of Perplexity and other smaller search providers, along with Exa via OpenRouter's integration.)
Hope this helps! Thanks for your questions!
Other inputs might be direct statements from leaders involved in the conflict, especially Trump. Also maybe bond market and oil market price movements?
Then you would want to generate an alert when you an actionable prediction. You don't want the user to have to prompt the AI. It needs to be running in the background, having been prompted on the scenarios to monitor?
Exactly! That's how it works - the static demo is just that, static.
Would love to onboard you for the full thing if you'd like! Just LMK (team@lookbacklabs.com) or add your info on the site
[dead]
[dead]