Building Hacker Search

Learnings from building a semantic search engine for Hacker News content

Jonathan Unikowski (@jnnnthnn) • May 7th, 2024

Last Thursday I launched Hacker Search, a semantic search engine built on top of the Hacker News dataset.

I built this for fun and profit: I love the content I find on Hacker News, frequently use its lexical search feature, and suspected that semantic search might surface things I was otherwise missing — which is indeed exactly what happened.

After I launched on Hacker News, many have reached out asking specific questions about my approach to inform their own projects, so I figured I’d share some interesting learnings from the experience.

What is it exactly?

A semantic search engine enables users to search by meaning rather than keywords (aka “lexical search”). As an example, where a traditional search engine might fail miserably on a query like “how do diffusion models work” (because most articles on the topic don’t contain that exact string), semantic search engines typically perform much better by effectively searching through the meaning of the query. See Hacker Search results on the topic for more information.

Semantic search is not necessarily “better” than keyword search. Rather, their respective usefulness depends entirely on the nature of the query. If you’re looking for articles about a specific company, just typing the company’s name in a keyword-based search engine is probably all you need. But if you’re looking for something you’re not sure how to frame in terms of keywords (e.g. our sample “git good practices” lookup, which would likely require multiple keyword searches and a fair bit of sifting through them), or finding options you don’t know about (e.g. “dbms like postgres”), semantic search might be much more useful. Ideally you'd want to be able to use both, and for the search engine to effectively pick the right approach (or mix thereof) based on your query and intent.

How it works

Hacker Search works by enumerating links posted to Hacker News, visiting them, ingesting their contents, embedding those, and providing users with a simple client to do lookups against those embeddings. To keep things manageable as a small side-project, I ingested only about 100,000 out of 4,000,000 submissions on Hacker News, prioritizing high-score submissions and ignoring comments. There are many ways to do that, so I picked the one I knew how to spin up the fastest:

• A cron looks up the ID of the latest item available via the Hacker News API and enqueues a bunch of Cloud Tasks for each of the items we haven’t yet ingested
• Each task is executed as a serverless function, fetches the content of the URL using a scraping API, stores it, cleans it, and embeds it (why extract contents? because titles often have too little to work with for semantic search)
• When users type a query on the site, we embed it, look up which documents have the closest embedding via cosine distance, and retrieve those (I use pgvector as the vector store)
• Lastly, we rerank the results using an LLM and present them to the user

My focus was on putting together something that worked, not something that was perfect. This approach likely wouldn’t scale well to much larger dataset — but it worked for this one!

Data quality is everything, and luckily LLMs help

When you scrape links from the internet, you get a lot of junk, even when those links are sourced from a well-curated source. Some are dead, others captcha’ed your bot, and yet others have outdated contents (e.g. ”happy new year 2010”). I didn’t want to embed and surface those, both because they’d create a bad user experience and because they’d represent an unnecessary cost. I used two tricks to deal with that.

The first one is that I simply used LLMs to tell me what was junk and what was stale content. Surely I could have spent hours coming up with clever heuristics for that, but I found that simply pushing the content to GPT-3.5 (for determining junkiness) and GPT-4 (for determining staleness) produced entirely satisfactorily results without needing me to do any custom work.

If you’ve ever used LLMs, you’ve probably surmised by now that the cost of doing that might’ve been stupendous. And indeed, it would be if you were to index a large chunk of the internet in this fashion. Even for my much smaller dataset, pushing raw HTML code through GPT-4 would’ve been pretty expensive.

I reduced those costs by trimming much of the non-core content from the articles I parsed using readability.js (which does a pretty good job with some finicking, though LLMs probably could do even better if you just asked them to identify the containing element), and then summarizing the resulting article contents using GPT-3.5. LLMs are remarkably good at summarizing contents, and this allowed me to substantially reduce the cost of downstream inferences (junkiness, staleness, and embedding) by operating on much fewer tokens. The cost ended up being around $0.01 per article, and conveniently, I had some OpenAI credits expiring at the end of April that I used for this.

Pros and cons of operating on summaries

Of course, operating on summaries means that subtle mentions in the body of articles might simply get lost, and therefore reduce search performance with regards to recall. But that seemed like an acceptable tradeoff to make the project economically reasonable, and in my experience yielded entirely acceptable results.

Another fun aspect of summaries is that they enabled me to generate just-in-time summaries for each result, personalized to the query terms used by the user (though I suspect GPT-3.5, which I use for that, sometimes tries a bit too hard and hallucinates connections that might not exist). But where they came in truly handiest was for reranking.

A sample summary for a search about “Best places to visit in Tokyo”

Reranking with LLMs

Search and information retrieval are canonical topics in computer science, and if you’re an expert in those, you’re probably thinking by now that I’m just reinventing the wheel using LLMs. That might well be the case, but I suspect I wouldn’t have moved nearly as fast without them.

The above is particularly true for reranking: once we’ve fetched the items most similar to the user’s query in the database, it’s likely the case that some are more relevant than others, and mere cosine distance won’t tell us that. So I used an LLM (here, GPT-3.5) to rerank the results, sending it the candidates and the user's query, and asking it to rank results in order of relevance.

Here again, summaries unlocked additional functionality. I provide them to GPT-3.5 when asking it to rerank results, substantially improving the performance of that part of the pipeline (compared to titles alone) while still enabling fast and cheap responses due to the relatively low number of tokens (1000 searches cost less than $1 without any particular optimizations).

What’s next?

I received a ton of encouragement and feedback after launching on Hacker News, and judging by usage, Hacker Search in its present form is already proving to be quite useful to folks. That said, I think I can make it a lot more useful, and am planning to do so in short order. If you’re interested to try new features when they ship, sign up here or follow me on X!

A thank you note

Jeremy Press provided a ton of assistance through this project, from how to efficiently extract contents from documents to reviewing drafts of this post.