It seems, that the amount of information that "typically" an person would want to work with, can be handled with pretty much any technique by the compute that most of us have, at a performance level that is satisfying.. It's only when you start to increase the information volume, decrease the compute, or increase the performance expectations, that it is really impactful to start to find methods that really make the most of the deep structures of that information to query it. Complexity of the original information and what accuracy means for your use case also comes into play I suppose.

I've been experimenting with hybrid memory retrieval using SQLite FTS5/BM25 for lexical search and dense embeddings from Nomic Embed Text v1.5 (768D), all-MiniLM-L6-v2 (384D), and Qwen3-Embedding (0.6B), with NumPy cosine similarity, Matryoshka-style dimensional truncation (64D → 768D reranking), and optional graph-based candidate expansion. In one 40-query benchmark, combining retrieval channels improved candidate recall from 30/40 with compact vectors alone to 40/40, although Top-1 accuracy was only 33/40. Another metric I was looking at was number of final candidates found to filter. If its very high, you can get better accuracy, but at the expense of resource use, and it can also lead to worse accuracy if your tying to choose top result efficently.. it makes a needle in a needle stack.

> I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods

Funny, I did exactly this for my last RAG project. Works great for a small dataset.

> found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods

I've been saying this for a while now. Lexical is approximately the same thing as semantic when a frontier LLM is driving the search terms.

The trade-off between speed and quality is everything though. I've found that semantic searching everything that might be relevant and dump the top N into the model is way faster at getting a workable result.

Good point and it’s an often overlooked.

RAG can take the intelligence of an LLM and dumb it down to old school Google search results.

> the personal assistant AI category

Didn’t Apple sort of already enter this market early-on with their subpar Siri?

I’ve been an iPhone and MacBook Pro user for years, but each time when I set up a new Apple device the first thing I do is say no to anything Siri, and also turning off additional Siri related settings. To the best of my knowledge, Siri is worse than useless, and I haven’t regretted staying away from it.

Agreed, but I wonder how that plays out without them entering the frontier model space. I just downloaded the new foundation models cli and ran some inference on my Mac, it wasn't great, but it was neat to see. I asked which model it was using and it said gpt 4o mini.

An on-device, 100% private AI assistant seems like it could be so big that it could be a legitimate threat to the frontier model providers they are partnering with right now, imho.

Apple or Steve Jobs? It seems like that was Steve Jobs doing as Apple hasn't done it without him.

You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.

For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.

RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.

For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.

Grep is not semantic like embeddings, although the llm can expand a concept into multiple literal searches with a similar effect.

Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.

Retrieval is a whole subject of study. For well-structured plaintext datasets, basic filtering may be as good as semantic and lexical searches. Takes measurement to know what you need

the amount of times the clanker has used grep badly and nuked my changes should be illegal, god bless git, god bless AI hook gates, and god bless RAG