Appreciate it. Really looking forward to your feedback.
I have my own infra setup and started with 25k seed domains including major known startups, YC companies and things like that. I collected some of the data via scraping and then merged it with 3rd party enrichment sources. Once the crawling per domain is finished, it is put on a 30 day schedule to repeat the crawl just to keep things fresh.
If you are on desktop and click on a result entry, you'll see expanded data for each domain. In reality that's just a small portion of the total data I collect.
Anyway, really looking forward to hearing your thoughts!
You say, "This type of search engine does not work well for informational search intent". What specifics areas does the search engine work well for? Also great job, this is neat!
It's mostly for navigational search. Say you're looking for a tool that does X, this engine will match your query against all the domains in the database to find you the closest matching domain that solves for that pain point you have.
It's like a discovery engine to help you find tools, products, services, or domains that cater to (or market themselves as a solution for) the problem you have.
Curious to hear if you have any feedback on the concept as is or if you can see any potential changes that would make it work better?
Interesting approach. What made you choose this direction? Was it mainly frustration with SEO spam, or did you discover this approach through experiments? It would be interesting to hear more about this.
I first thought about this approach back in 2011 when I was in college. But never did anything with the idea because I did not have the resources at the time to build something like this.
As of late however, it has become rather clear that the current distribution channels for businesses, both organic and paid, are getting squeezed.
- Google doesnt send you traffic anymore
- Search results are infested with SEO optimized content that no one can compete with
- LLMs cite the same garbage
- Distribution via advertising is getting more expensive each day
- Social channel are all about clickbait and maximize following which encourages quantity over quality
It's clear that the relationship between product-sellers and potential-buyers is completely out of whack today. Most of the time they can't even find each other.
That's why I decided to crawl a small index of 25k websites and try this out.
The idea is to profile a domain as a whole (single entry) and match the user's query to find the best fitting product/service that solves their pain point. This ignores any SEO optimized content, spam, black hat garbage, and instead focuses on finding the best match making the distribution playing field more equal.
Is this a good idea? I still don't know. That's why I'm looking for feedback. I could easily improve the search result quality by like 5-10x but I'm trying to ascertain first if this approach is even helpful to people.
I will try this out over the next day or so and let you know my thoughts. Can I ask how you did the crawling to get this started?
Appreciate it. Really looking forward to your feedback.
I have my own infra setup and started with 25k seed domains including major known startups, YC companies and things like that. I collected some of the data via scraping and then merged it with 3rd party enrichment sources. Once the crawling per domain is finished, it is put on a 30 day schedule to repeat the crawl just to keep things fresh.
If you are on desktop and click on a result entry, you'll see expanded data for each domain. In reality that's just a small portion of the total data I collect.
Anyway, really looking forward to hearing your thoughts!
You say, "This type of search engine does not work well for informational search intent". What specifics areas does the search engine work well for? Also great job, this is neat!
Thanks :)
It's mostly for navigational search. Say you're looking for a tool that does X, this engine will match your query against all the domains in the database to find you the closest matching domain that solves for that pain point you have.
It's like a discovery engine to help you find tools, products, services, or domains that cater to (or market themselves as a solution for) the problem you have.
Curious to hear if you have any feedback on the concept as is or if you can see any potential changes that would make it work better?
Interesting approach. What made you choose this direction? Was it mainly frustration with SEO spam, or did you discover this approach through experiments? It would be interesting to hear more about this.
Cheers!
I first thought about this approach back in 2011 when I was in college. But never did anything with the idea because I did not have the resources at the time to build something like this.
As of late however, it has become rather clear that the current distribution channels for businesses, both organic and paid, are getting squeezed.
- Google doesnt send you traffic anymore
- Search results are infested with SEO optimized content that no one can compete with
- LLMs cite the same garbage
- Distribution via advertising is getting more expensive each day
- Social channel are all about clickbait and maximize following which encourages quantity over quality
It's clear that the relationship between product-sellers and potential-buyers is completely out of whack today. Most of the time they can't even find each other.
That's why I decided to crawl a small index of 25k websites and try this out.
The idea is to profile a domain as a whole (single entry) and match the user's query to find the best fitting product/service that solves their pain point. This ignores any SEO optimized content, spam, black hat garbage, and instead focuses on finding the best match making the distribution playing field more equal.
Is this a good idea? I still don't know. That's why I'm looking for feedback. I could easily improve the search result quality by like 5-10x but I'm trying to ascertain first if this approach is even helpful to people.
Curious to hear if you have any feedback for me?