|
|
|
|
|
|
The Unseen Web David Dorn knows that search engines just don’t search more than a quarter of Web sites out there – the rest is the Unseen Web – but he knows how you can see the unseen! In the beginning was the Web, and every page on it was an individual HTML document, probably lovingly hand-crafted in a text editor, with its creator typing all the various HTML commands and syntax by hand. Then, along came the likes of Hot Metal, Front Page, Dreamweaver and so on, whicb made WYSIWYG editing a reality, and speeded up the crea\tion of Web sites. But still, each page was an individual document. Search engines had no problems with sich sites – they could crawl through each page looking for content, and then build their own databases for you to search through. Now, though, things are different. There are hundred and thousands of Web sites out there that are built on databases. Their page count is minimal. You may have seen URLs that include great strings of numbers, question marks, and other gubbins that look nothing whatever like “http://www.hereyougo.com/index.html”. They’re actually a set of commands to a database, which prompt the database at that location to assemble a Web page for you to look at. But the page is transitory – it doesn’t exist, other than in your browser. So, of course, search engines cannot index a page that doesn’t exist – hence the Unseen Web – it’s unseen by search engines. If you want to search such a site for its contents, you generally have to go to the site itself and use its own search facilities to discover what’s hidden beneath the database front end. The likes of AOL Search, Google, AltaVista and others just can’t burrow down into the database to get you the info. I must admit, I’m not exactly sure how many clicks and hits such a Web site loses by not being massively indexed by the search engines – indeed, I’m not entirely sure just exactly how vital search engines are, although my experience this last week or so would be very different had they not existed. Let me tell you why. I’m in the market for a specific bit of kit – it’s a mount cutter, a device to allow me to cut precisely sized rectangular holes in sheets of mounting card for the purpose of framing digital photos printed at huge size. I’ll freely admit that, although I know such things exist, I know nothing about who makes them or what the price range for them is. So, my better half decided that she would, in her words, “hit the Web” and see what she could find out. Now, she’s a Google fanatic, so she went there, typed in “Mount Cutters” and got 15 pages of links to have a play with. Dozens of UK sites, some of which sell mount cutters, some of which sell side mount grass cutters! That’s fine – I sat with her and we learned a lot – not that I’m that much nearer to making a purchase decision, but that’s more a matter of the most informative sites not being e-commerce enabled than anything else. What I don’t know, though, is how many database-driven sites Google didn’t index, because it couldn’t. I don’t know how much information I’m missing, purely because the search engines can’t get at it. Now, that doesn’t really worry me too much, because I’m very much an old fuddy-duddy, and the kind of shop that deals in mount cutters is, more than likely, not going to have the resources to create an expensive database-driven site, or it’s going to be run by a fuddy-duddy like me. But here’s the thing – if your site is an all-singing, all-dancing database driven job, how much traffic do you lose because the search engines can’t keep up? Some Web experts (I call them Webspurts) tell us that more than fifty percent of a site’s traffic comes from search engines, and that you have to be in the first page of a search result in order to derive any benefit from it. They tell us that Flash-only sites lose out because they’re unsearchable. They tell us that database-driven sites lose out because they’re unsearchable. It’s an interesting thought. There is, though, light at the end of the tunnel. There’s a new search engine currently in beta that claims to be able to search the Unseen Web. I’ll leave it to you to find it as an exercise, by using search engines! Have your say - click here |
