Re: Search Engine questions ...
This WebDNA talk-list message is from 2002
It keeps the original formatting.
numero = 44784
interpreted = N
texte = >> > >HOW do they read the DNA stuff from the server? If the pages are preprocessed by Webcat, all of the DNA is stripped out before being sent to the SE, right??? >Here's what I found in the archives. I think I misquoted JP.>2002/08/03 03:01:34 John Peacock Re: OT: Site Downloaders >Lester Emo wrote: >> Is there a way to prevent a program like Offline Explorer >> by MetaProducts.com to suck up every page linked in a >> site and download it to a persons computer??? >> >If your site is set up properly, any page with webdna will not be served up >except through WebCatalog itself (hence processed). Any actively generated site can >be spidered as long as the spider can generate the appropriate links. There is >nothing you can do about it, since the spiders can ignore the ROBOTS file and >pretend they are an ordinary browser. >John I still say the spiders can recognize dynamically generated links. That's based on my experience of examining log files and searching the se's for my stuff. I know it's so for AllTheWeb. I'm still waiting for Google Bot to come back around so I can see how it spiders the same stuff again.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
>> > >HOW do they read the DNA stuff from the server? If the pages are preprocessed by Webcat, all of the DNA is stripped out before being sent to the SE, right??? >Here's what I found in the archives. I think I misquoted JP.>2002/08/03 03:01:34 John Peacock Re: OT: Site Downloaders >Lester Emo wrote: >> Is there a way to prevent a program like Offline Explorer >> by MetaProducts.com to suck up every page linked in a >> site and download it to a persons computer??? >> >If your site is set up properly, any page with webdna will not be served up >except through WebCatalog itself (hence processed). Any actively generated site can >be spidered as long as the spider can generate the appropriate links. There is >nothing you can do about it, since the spiders can ignore the ROBOTS file and >pretend they are an ordinary browser. >John I still say the spiders can recognize dynamically generated links. That's based on my experience of examining log files and searching the se's for my stuff. I know it's so for AllTheWeb. I'm still waiting for Google Bot to come back around so I can see how it spiders the same stuff again.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Glenn Busbin
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Not really WebCat- (1997)
can WC render sites out? (1997)
Credit Card processing (1998)
refreshing IE with posted .tmpl (1997)
Whats going on? (2000)
take me off mailing list please (2001)
[WebDNA] Version 8 (2014)
Single Link browsing (1997)
[setcookie] & [redirect] (1998)
WebCat2b13MacPlugIn - [shownext method=post] ??? (1997)
HEADER AND FOOTER (1997)
shipcost (1997)
Protect vs Authenicate (1997)
Text data with spaces in them... (1997)
international time (1997)
2.0 Info (1997)
[cart] how is it generated (2002)
Expiration of Carts (1997)
Date Sorting (1997)
WebCat (or other) Indexing (1999)