HN

Comments (35)

• l3x4ur1n
  Would this work for my use case?I need to extract article content, determine it's sentiment towards a keyword and output a simple json with article name, url, sentiment and some text around the found keyword.Currently I'm having problems with the json output, it's not reliable enough and produces a lot of false json.
• chattermate
  The malformed JSON problem is real and tends to be model-specific. We run multiple LLM providers in production (OpenAI, Groq/Llama, Google AI, Ollama) and there's a meaningful gap in schema adherence between frontier models and smaller/cheaper ones. Nested arrays with optional fields are particularly tricky — smaller models will nail 19 out of 20 objects and silently mangle the last one in ways that are hard to predict.One pattern that's helped us: decomposing complex schemas into multiple simpler sequential extractions rather than one large schema. Less impressive as a demo, but noticeably more reliable in production when you're cost-optimizing with smaller models. The partial recovery approach here (keeping valid items even when one fails) is exactly the right instinct for keeping pipelines alive.
• Menu

  sheept
  > LLMs return malformed JSON more often than you'd expect, especially with nested arrays and complex schemas. One bad bracket and your pipeline crashes.This might be one reason why Claude Code uses XML for tool calling: repeating the tag name in the closing bracket helps it keep track of where it is during inference, so it is less error prone.
• Menu

  Flux159
  This looks pretty interesting! I haven't used it yet, but looked through the code a bit, it looks like it uses turndown to convert the html to markdown first, then it passes that to the LLM so assuming that's a huge reduction in tokens by preprocessing. Do you have any data on how often this can cause issues? ie tables or other information being lost?Then langchain and structured schemas for the output along w/ a specific system prompt for the LLM. Do you know which open source models work best or do you just use gemini in production?Also, looking at the docs, Gemini 2.5 flash is getting deprecated by June 17th https://ai.google.dev/gemini-api/docs/deprecations#gemini-2.... (I keep getting emails from Google about it), so might want to update that to Gemini 3 Flash in the examples.
• letier
  The extraction prompt would need some hardening against prompt injection, as far as i can tell.
• Menu

  vetler
  My instinct was also to use LLMs for this, but it was way to slow and still expensive if you want to scrape millions of pages.
• Menu

  plastic041
  > Avoid detection with built-in anti-bot patches and proxy configuration for reliable web scraping.And it doesn't care about robots.txt.
• spiderfarmer
  My platform has 24M pages on 8 domains and these NASTY crawlers insist on visiting every single one of them. For every 1 real visitor there are at least 300 requests from residential proxies. And that's after I blocked complete countries like Russia, China, Taiwan and Singapore.Even Cloudflares bot filter only blocks some of them.I'm using honeypot URLs right now to block all crawlers that ignore rel="nofollow", but they appear to have many millions of devices. I wouldn't be surprised if there are a gazillion residential routers, webcams and phones that are hacked to function as a simple doorways.Things are really getting out of hand.
• Menu

  dmos62
  What's your experience with not getting blocked by anti-bot systems? I see you've custom patches for that.
• Menu

  AirMax98
  This feels like slop to me.It may or may not be, but if you want people to actually use this product I’d suggest improving your documentation and replies here to not look like raw Claude output.I also doubt the premise that about malformed JSON. I have never encountered anything like what you are describing with structured outputs.
• Menu

  zx8080
  Robots.txt anyone?
• warwickmcintosh
  [dead]
• Remi_Etien
  [dead]
• hikaru_ai
  [dead]
• johnwhitman
  [dead]
• openclaw01
  [dead]
• gautamborad
  [dead]