How Search Engines Work: Crawling to Ranking
By The seo.bike team, SEO & GEO practitioners · Last reviewed July 8, 2026
Search engines run three connected processes. Crawlers discover and fetch pages by following links and sitemaps. Indexing parses, renders and stores what those pages mean in a giant lookup structure. Ranking then scores every eligible page against a query in real time, combining hundreds of relevance, quality and context signals to order the results.
Discovery: how crawlers decide what to fetch
A crawler cannot read the whole web at once, so discovery is really a prioritization problem. It maintains an enormous queue of known URLs, gathered from links on pages it has already seen, from XML sitemaps you submit, and from other feeds. Every new page it fetches adds more links to the queue, so the web is explored outward along its link graph.
Not all URLs are treated equally. The crawler estimates how important and how fresh a page is likely to be, and schedules accordingly. A frequently updated, heavily linked page gets revisited often; an obscure page linked from nowhere may wait a long time or never be reached at all. This is why internal linking matters so much — a page with no internal links pointing to it is effectively invisible to discovery, no matter how good it is.
The engine also budgets its own effort per site, informally called crawl budget. On a small site this never binds. On a large site with millions of URLs — especially one generating endless near-duplicate URLs from query strings — the crawler can waste its budget on junk and never reach the pages that matter. That is one mechanism by which sprawling, low-value URLs actively harm a site: they starve the important pages of crawling.
Fetching, rendering and the JavaScript problem
When the crawler fetches a URL, it first respects the rules you have set. A disallow in robots.txt stops it from fetching at all; a noindex tag, seen after fetching, tells it not to store the page. Assuming neither blocks it, the engine retrieves the raw HTML.
Here is where many sites quietly fail. Modern pages often ship a nearly empty HTML shell and build the real content with JavaScript in the browser. To see that content, the engine must render the page — run the scripts the way a browser would — which is far more expensive than reading HTML. Rendering is often deferred and queued separately, so JavaScript-dependent content can be indexed late or, if the scripts fail or block, not at all.
The practical rule: content that only appears after client-side JavaScript runs is at risk. If it matters for ranking, it should be present in the initial HTML, delivered by server-side rendering or pre-rendering. A page that looks rich to a human but is empty to a raw fetch is a page the engine may treat as thin.
Indexing: what actually gets stored
Indexing is not screenshotting pages; it is turning them into a structure built for instant lookup. The engine parses the rendered page, extracts the meaningful text, understands headings and structure, notes the title and metadata, reads any structured data, and identifies the entities and topics the page is about.
At the core sits an inverted index — conceptually, a giant map from each word or concept to the list of pages that contain it, along with where and how prominently. That inversion is what lets an engine answer a query across trillions of pages in a fraction of a second: it is not scanning the web on demand, it is looking up pre-computed lists.
The index also stores signals about each page: its canonical URL (so duplicates collapse to one entry), the language and region it targets via hreflang, its freshness, and quality assessments. Crucially, a crawled page is not automatically an indexed page. The engine decides whether a page is worth storing at all, and increasingly declines to index content it judges thin, duplicative, or low value. "Discovered — currently not indexed" is the engine telling you it fetched your page and chose not to keep it.
Ranking: how signals combine
When someone searches, the engine does not rank the whole web; it pulls the comparatively small set of indexed pages relevant to the query, then scores and orders them in real time. That scoring blends a large number of signals, and no single one dominates.
Relevance signals ask whether the page addresses the query and its intent — not just matching words, but matching meaning, which modern engines assess with language models that understand synonyms, context, and what kind of answer the searcher actually wants. A query like a definition, a purchase, or a local business each expects a different sort of result.
Quality and trust signals ask whether this source deserves to be believed — the reputation earned from links and mentions, the evidence of real expertise and experience, and the site's overall track record. Context signals personalize the result: the searcher's location, language, device, and sometimes recent behavior.
These are combined by systems tuned against enormous amounts of human quality-rating data, so the weighting shifts by query type. For a breaking-news query, freshness dominates; for a medical query, trusted expertise dominates; for a local query, proximity dominates. There is no fixed formula you can reverse-engineer — which is exactly why chasing a single ranking factor is a losing strategy.
Freshness, re-crawling and why results change
The index is never finished. Crawlers continually revisit pages to catch changes, and the engine re-evaluates rankings constantly, which is why positions drift even when you change nothing.
Freshness itself is a query-dependent signal. For queries where recency matters — news, prices, ongoing events — the engine strongly prefers recently updated content and re-crawls likely-fresh pages aggressively. For evergreen queries — a math concept, a historical fact — an older authoritative page can outrank newer ones indefinitely, and chasing freshness by re-dating unchanged content fools nobody.
Periodically the engine ships a core update: a broad recalibration of how it weighs quality and relevance across the board. These are not penalties aimed at you; they are the system getting better at rewarding what it already valued, which can nonetheless move your rankings sharply in either direction. The durable response is not to chase each update but to keep the fundamentals sound — a crawlable site, genuinely useful content, and earned trust — so that recalibrations tend to move in your favor rather than against you.
Key terms
Related guides
- Does SEO Still Work in 2026? An Honest Take
- On-Page SEO: How to Optimize a Page That Ranks
- Technical SEO: The Crawl, Index & Speed Foundations
- What Is SEO? A Plain-English Guide for Beginners
Frequently asked questions
It maintains a huge queue of known URLs, gathered by following links from pages it has already crawled and by reading the XML sitemaps you submit. Each page it fetches adds its links to the queue, so the web is explored outward along its link graph. A page with no links pointing to it and no sitemap entry may never be discovered at all.
Crawling is fetching a page's content. Indexing is deciding to parse, understand and store that page in the searchable structure. They are separate steps: a page can be crawled and then not indexed if the engine judges it thin, duplicative or low value. The status "discovered — currently not indexed" means the engine reached your page and chose not to keep it.
They can, but at a cost. Seeing content built by client-side JavaScript requires the engine to render the page by running its scripts, which is expensive and often deferred to a separate queue. Content that only appears after scripts run may be indexed late or missed if the scripts fail. If it matters for ranking, deliver it in the initial HTML via server-side rendering.
The index is constantly re-crawled and rankings are recomputed continuously, so positions drift as competitors change, freshness is re-weighted, and the engine ships periodic core updates that recalibrate how it scores quality and relevance across the whole web. These broad updates can move your rankings either way without any change on your side. Sound fundamentals make those shifts tend in your favor.
Put this into practice
Try the free SEO tools, or let the managed service do the work for you — every change checked by a safety linter before it ships.