Ever felt like search results are drowning in low-quality content and keyword-stuffed articles? You’re not alone. A New York software engineer got so frustrated with SEO spam cluttering his search results that he decided to take matters into his own hands – by building an entirely new search engine from scratch.
Wilson Lin’s two-month journey from concept to working prototype offers fascinating insights into what it really takes to compete with search giants. His experience reveals why combating search spam is so challenging and what alternative approaches might look like in the future.
Why Another Search Engine?
Lin’s motivation stemmed from a growing problem many users face: mainstream search engines increasingly surface irrelevant, spam-filled results optimized for rankings rather than user value. After completing his project, he noted one of the most satisfying outcomes: “What’s great is the comparable lack of SEO spam.”
But building a search engine that actually works requires solving numerous technical challenges that most people never consider.
The Neural Embeddings Approach
Instead of relying on traditional keyword matching, Lin chose neural embeddings as his foundation. This approach uses machine learning to understand the semantic meaning behind queries and content, potentially offering more relevant results than keyword-based systems.
His small-scale testing validated that embeddings could effectively match user intent with relevant content, even when the exact keywords weren’t present.
Breaking Down Content Intelligently
One critical decision involved how to process and chunk web content. Should the system analyze entire paragraphs or individual sentences?
Lin settled on sentence-level processing as the most granular approach that still made sense. This method allowed his engine to identify precise answers within sentences while maintaining enough context for semantic understanding.
However, he encountered issues with indirect references – words like “it” or “the” that depend on previous context for meaning. His solution involved training a specialized classifier:
“I trained a DistilBERT classifier model that would take a sentence and the preceding sentences, and label which one (if any) it depends upon in order to retain meaning. When embedding a statement, I would follow the ‘chain’ backwards to ensure all dependents were also provided in context.”
This approach also helped identify sentences that shouldn’t be matched independently because they lacked standalone meaning.
Extracting Main Content From Web Pages
Every search engine crawler faces the challenge of separating actual content from navigation menus, advertisements, and other page elements. Without clear content identification, search results become cluttered with irrelevant snippets.
Lin’s solution relied on specific HTML tags to identify main content areas:
- blockquote – For quotations and cited material
- dl – Description lists and definitions
- ol – Ordered, numbered lists
- p – Standard paragraph content
- pre – Preformatted text blocks
- table – Tabular data presentation
- ul – Unordered bullet-point lists
This tag-based approach provided a practical way to focus on meaningful content while filtering out navigational clutter.
The Hidden Complexities of Web Crawling
Crawling the web turned out to be far more complicated than expected. Lin discovered that DNS resolution failures occurred surprisingly frequently, creating unexpected bottlenecks.
URL handling presented another layer of complexity:
“They must have https: protocol, not ftp:, data:, javascript:, etc. They must have a valid eTLD and hostname, and can’t have ports, usernames, or passwords.”
Additional challenges included:
- Canonicalizing URLs to prevent duplicate indexing
- Managing extremely long URLs that exceeded system limits
- Handling unusual characters that caused downstream failures
- Processing query parameters consistently
These technical hurdles illustrate why building reliable web infrastructure requires extensive error handling and edge-case management.
Infrastructure Decisions and Scaling Challenges
Lin initially chose Oracle Cloud for its generous data transfer allowances and competitive pricing. With terabytes of data expected, avoiding expensive egress fees was crucial for project viability.
However, scaling issues forced a migration to PostgreSQL, which brought its own set of challenges. Eventually, he settled on RocksDB as the optimal solution:
“I opted for a fixed set of 64 RocksDB shards, which simplified operations and client routing, while providing enough distribution capacity for the foreseeable future.”
At peak performance, this system processed 200,000 writes per second across thousands of clients, handling not just raw HTML but normalized data, contextualized chunks, high-dimensional embeddings, and extensive metadata.
GPU Computing for Semantic Understanding
Generating semantic embeddings required significant computational power. Lin initially used OpenAI’s embedding API but found costs prohibitive at scale.
His solution involved GPU-powered inference using transformer models hosted on Runpod’s infrastructure:
“I discovered Runpod, who offer high performance-per-dollar GPUs like the RTX 4090 at far cheaper per-hour rates than AWS and Lambda. These were operated from tier 3 DCs with stable fast networking and lots of reliable compute capacity.”
This approach provided the processing power needed for semantic analysis while maintaining cost efficiency.
Demonstrating Reduced Search Spam
Lin’s engine showed promising results in spam reduction. Using queries like “best programming blogs,” he demonstrated cleaner results compared to mainstream search engines. The system could also handle complex queries, including full paragraphs of text to discover topically relevant articles.
This capability suggests that semantic understanding might offer a path toward more spam-resistant search experiences.
Four Key Lessons for Search Engine Development
1. Index Size Directly Impacts Quality
Coverage emerged as a critical quality factor. As Lin noted, “coverage defines quality.” A comprehensive index is essential for surfacing valuable content, making crawling scale a fundamental challenge.
2. Filtering Represents the Greatest Challenge
While comprehensive crawling is important, distinguishing valuable content from noise proved exceptionally difficult. Balancing quantity with quality requires sophisticated filtering mechanisms.
This echoes the original PageRank innovation – using human link choices as quality signals. Modern approaches like those used by Perplexity still rely on modified versions of these foundational concepts.
3. Independent Search Engines Face Inherent Limitations
Small-scale operations can’t match the comprehensive web coverage that major search engines provide. These coverage gaps create fundamental constraints on result quality and completeness.
4. Trust and Quality Assessment Remain Unsolved
Automatically determining content authenticity, accuracy, and value across unstructured data continues to challenge search systems:
“Determining authenticity, trust, originality, accuracy, and quality automatically is not trivial. If I started over I would put more emphasis on researching and developing this aspect first.”
Lin believes transformer-based approaches to content evaluation could offer simpler, more cost-effective solutions than traditional multi-signal ranking systems.
What This Means for the Future of Search
Wilson Lin’s experiment demonstrates both the possibility and the challenges of building alternative search experiences. While his prototype shows promise in reducing spam through semantic understanding, it also highlights the enormous technical and resource barriers that protect established search engines.
His work suggests that the future of search improvement may come not from completely replacing existing systems, but from developing better approaches to content quality assessment and semantic understanding that can be integrated into existing infrastructure.
For users frustrated with search spam, Lin’s project offers hope that alternative approaches are possible – even if building them requires overcoming significant technical and economic challenges.
You can explore Wilson Lin’s search engine prototype and read his detailed technical documentation to see these concepts in action.
