95 points by zerop 3 hours ago
Comments are read-only. Visit Hacker News to participate in the discussion.
lavendermon 1 hour ago | 0 points
Apple has a fascinating history of waiting and seeing how a market develops before entering with a better product and eventually dominating, the iPod and iPhone are two examples. It seems they may be building a strong position to pull that off again in the personal assistant AI category ChatGPT is trying to dominate.<p>I enjoy watching the slow burn approach Apple is deploying as they quietly gear up to offer some % of local inference on iPhones to provide most of the AI services consumers want. Local inference could enable a very low price point for AI services that Apple would leverage alongside their massive ecosystem advantages and privacy claims to compete very effectively for personal assistant AI subscriptions.
QuantumNomad_ 2 minutes ago | 0 points
> the personal assistant AI category<p>Didn’t Apple sort of already enter this market early-on with their subpar Siri?<p>I’ve been an iPhone and MacBook Pro user for years, but when I set up a new Apple device the first thing I do is say no to anything Siri, and also turning off additional Siri related settings. To the best of my knowledge, Siri is worse than useless, and I haven’t regretted staying away from it.
elpakal 1 hour ago | 0 points
Agreed, but I wonder how that plays out without them entering the frontier model space. I just downloaded the new foundation models cli and ran some inference on my Mac, it wasn't great, but it was neat to see. I asked which model it was using and it said gpt 4o mini.<p>An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.
makmanalp 1 hour ago | 0 points
I think it'd be foolish to assume that they aren't already - they have a reputation of doing a lot more internally than ever comes to light. Smaller companies like mistral or thinking machines show that you don't even need that much resources to be semi-competitive, and comparatively Apple is a vertically integrated behemoth.<p>That said in a pinch they could also acquire / acquihire both of those companies without much thought.
andriy_koval 1 hour ago | 0 points
> I wonder how that plays out without them entering the frontier model space<p>bet could be that this space will be commoditized: many good enough open weight models and multiple competing provides.
comandillos 2 hours ago | 0 points
I feel like retrieval approaches, despite how good your ranking mechanisms are, end up biasing LLMs into what a heuristics-based system with way worse understanding of the domain says. I guess is a trade off between speed and quality, but letting the LLM explore without a bias (based on my personal experience) seems to generally provide much better results in terms of quality for the final output.
armcat 1 hour ago | 0 points
Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.<p>From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).<p>[1] <a href="https://research.atspotify.com/2025/9/you-say-search-i-say-recs-a-scalable-agentic-approach-to-query-understanding" rel="nofollow">https://research.atspotify.com/2025/9/you-say-search-i-say-r...</a>
mickeyp 53 minutes ago | 0 points
I'm not sold; that's not to say I can't be swayed, but I've spent a lot of time building RAG solutions as a "one-size-fits-all" against codebases and I am not convinced embedding models or any of the myriad BERT implementations work well enough that they can beat out grep.<p>Most embedding/BERT models are far too small to truly grasp the nettles when an AI ask questions to them; and if you condition the AI to only search for really basic things, then... you're basically back to grep + regex.<p>You need to prune all manner of useless detritus that might gum up your matches like test files, documentation, and sidecars that semantically overlap or you're just going to match your tests. That's fine if you're building a document store for a business, but if you want a "one-and-done" solution for your codebase it's rarely good enough.<p>And if you <i>do</i> use a very large embedding model (which means it'll require running in the cloud as most company computers won't run it well), then... well, you're better off just using a language model as a subagent to scout ahead for you and elide the embedding model to begin with.
brrrrrm 2 hours ago | 0 points
as a recovering RAG-enthusiast, I will share a quote from a peer that has really stuck with me:<p>"why not use grep?"
mejutoco 1 hour ago | 0 points
Grep is not semantic like embeddings, although the llm can expand a concept into multiple literal searches with a similar effect.
0o777 1 hour ago | 0 points
There have been some attempts at "semantic grep" but don't know its current status<p><a href="https://github.com/lightonai/next-plaid/tree/main/colgrep" rel="nofollow">https://github.com/lightonai/next-plaid/tree/main/colgrep</a>
elpakal 1 hour ago | 0 points
Your understanding was applied too strictly and you missed the intent of the joke...
goldenarm 2 hours ago | 0 points
Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.
ozkatz 1 hour ago | 0 points
all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.
jdthedisciple 51 minutes ago | 0 points
> red|maroon|scarlet|crimson<p>still doesn't include typo'ed variants for any of these
dv_dt 2 hours ago | 0 points
If the data is this large then why not ask the llm to build a query
itishappy 1 hour ago | 0 points
That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.
nostrebored 1 hour ago | 0 points
latency is often the reason, distilling interactions into significantly better retrieval is the second.
aaroninsf 1 hour ago | 0 points
My case:<p>With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.
pants2 1 hour ago | 0 points
Can't wait to see how they leverage this research to make Siri even worse!