Did you know about the Hidden Web?

[redi]

The Hidden Web (http://en.wikipedia.org/wiki/Invisible_Web). I haven't read it that carefully but what it boils down to I think is that our search engines can only scan static pages and not dynamically generated pages, while the latter embody the majority content of the internet. Hence, most of what's available on the internet cannot be found by/through search engines, but you can only get it by generating the page yourself using a specific search.

Of course that leaves the questions whether this dynamically generated content is really important, interesting or useful :) Or maybe it's all kinds of secret stuff :ohmy::-P

[ravenmorpheus2k]

Deepnet. Sounds like somewhere governments would store secrets online. :lol:

[Eurostc]

:-)

[MAD-MINI]

Deepnet. Sounds like somewhere governments would store secrets online. :lol:

No you couldnt be further from the truth....
It's where I keep my laundry........ :D

yes all 91,000 terabytes of dirty clothes

alas the truth is out. I shall go and wash most of it and "clean" up the conspiracy

[BruceD]

I think is that our search engines can only scan static pages and not dynamically generated pages, while the latter embody the majority content of the internet

Total rubbish cos I can't count the amount of times that a google search will lead me to a site that has automatically generated my search topic into itself. But yes search engines are pretty useless at finding half the webpages out there.

[redi]

Total rubbish cos I can't count the amount of times that a google search will lead me to a site that has automatically generated my search topic into itself. But yes search engines are pretty useless at finding half the webpages out there.

If you have read the Wiki article then you have seen that most search engines have been adapted to start searching the hidden net, but they're not fully capable yet ;)

[AndreiRS]

Hahah my forum is one of those, when I tell people to google it, nobody finds. And it is there... ?!

[gtrNL]

Example (in case you still don;t really understand what this all is):
Type this on google:
"showroom site:http://members.iracing.com/jforum"
You won't find anything.

Use this query on Google:
"site:http://members.iracing.com/jforum"
You will find the website you are looking for.

Reason, Google knows the member website of iRacing exists, but it can't search it, because it is only visible for logged in members. This site is 'hidden'.

NoGrip however, is not, search google for "NoGrip" and you will find it. Search Google for "NoGrip however, is not" and you will find this post in this forum.
Reason: NoGrip can be read when not logged in.

Last one. Search "Facebook", and you will find Facebook. Search "Your friend just posted a cool video and someone liked it" and you won't find anything.

Should the hidden web really be part of our search engines (should it be indexable)?
Why not? Give your favo search engine access to your Facebook, iRacing, what ever, password and username and use the search engine for every single thing you ever want to find on the web. If you are allowed to read it, you can find it.

On the other hand, you will be giving out nearly all your information to the company behind the search engine and some people don't like that (Google-fobia).

[redi]

Ah yes, of course everything hidden behind personal access/passwords cannot be searched. Should not be searched :)