HN

Need help?
<- Back
Artificial intelligence now beats some of the best human forecasters

Artificial intelligence now beats some of the best human forecasters

ddp26

Comments (75)

  • gertlabs
    We measure skill differentiation between frontier / last-gen LLMs across our environments, and one of our curated coding environments is a closed-system market simulator, containing only other agents and some system participants (a market maker and a liquidity provider via issuance / buybacks) whose behavior is fully defined for all of the agents.This has the least measured skill differentiation of all of our environments, and not because forecasting/markets don't require skill or intelligence. Even the best models are so far from anticipating the behavior of the other agents and understanding the emergent effects that a 2025 model with a naive strategy can often outperform over the timeframes of the simulation simply because some other models in the simulation chose a similar self-reinforcing strategy. This likely happens to some degree in real markets.You can watch these simulations here https://gertlabs.com/spectate?game=market
  • glimshe
    Product idea: a LLM trained separately from mainline LLMs that anticipate market trends by analyzing how mainline LLMs will invest. As retail investors will probably use mainline AI for decisions going forward , one could get an edge."The AI-driven Market Hypothesis"Please let me know where I should pick up my Nobel prize.
  • bagels
    Aren't forecasters already using 'artificial intelligence' for decades in the form of non-llm machine learning models?
  • phyzix5761
    Stock analysts have a success rate of 47% or lower for directional predictions. That's worse than a coin flip. All AI has to do is product fair 50/50 results and it can beat analysts. But you can do it too for the price of a quarter.
  • seanhunter
    This has to be the least surprising development to date given ml is a universal function estimator
  • jesse_dot_id
    It will be interesting to see if this changes because presumably AI is using very predictable historical models, but it seems like the climate is shifting into something unseen that we won't have models for?
  • qsbuilder
    The test is when reflexivity kicks in and the prediction itself changes market behavior. LLMs usually melt there
  • johnecheck
    The markets are a highly complex dynamic system. There are many instances of it exhibiting disastrous behavior, especially in response to changes and shocks.AI trading and investment advice meaningfully changes the system and its dynamics. It seems highly probable that this will result in it failing in new ways.
  • attels33
    So my plan to go from a developer to an economist is scrapped. What now?
  • ratelimitsteve
    If 10,000 people guess 10,000 fair coin flips each one of them will get more guesses right than any of the others, one of them will get fewer guesses right than any of the others, and the gulf between the two is likely to be over 4 standard deviations wide. I'm certain that I, being an untutored schmuck from Pittsburgh and having thought of this almost immediately after reading about this contest, cannot be the first person to realize this is a potential problem for a forecasting contest. But I can't find anything they've done to mitigate that problem. Can anyone clue me in?
  • autoexec
    So I guess the AI companies can stop with their plans to infest AI with ads and they'll instead fully fund themselves by using their AI to gamble on stocks and the prediction market right? Surely the chatbots will just print money!
  • mbil
  • 296012
    That is too bad for The Economist. Exor N.V and Agnelli might replace some pundits at The Economist.
  • gyanchawdhary
    At the risk of sounding extremely naieve i have a question for the Wall St / quant / HFT folks lurking here ... but how hard would it actually be to brute force the math/algos behind Medallion Fund (or something in that general class) or even some of the average quant fundsI know it’s not just the math but execution, infrastructure, risk management, data, colocation (if ur an HFT) etc ... but LLMs seem like a pretty powerful apparatus for running experiments that .. a few years ago would have required fairly deep multidisplinary skills across coding .. stats .. and math ..So assuming you have decent intuition for ideas .. how difficult would it actually be to reverseengineer / rediscover some of the underlying stuff?
  • throwaway5752
    The best human forecasters working with artificial intelligence are going to do even better than either alone, the dichotomy is artificial.
  • croes
    Given the training data isn’t that more a win for the wisdom of the crowd?
  • xgulfie
    Hasn't this been true for like 40 years
  • anon48293
    Paywall
  • yeah879846
    [dead]
  • JonathanCross
    [flagged]
  • sehw
    [dead]
  • sehw
    [dead]
  • tolugenius