Re: how do we turn off staticpages page creation?

This WebDNA talk-list message is from

2003


It keeps the original formatting.
numero = 50118
interpreted = N
texte = >Set a robot.txt file in that directory and set whatever pages you do not >want crawled. That will work for good bots, but there are those that look for the names of pages and ignore the DISALLOW tag. That's why it's best to have no reference at all, either on a template or the robots.txt file, to anything you don't want visited.The ^*#&@^$! spammer's address harvesters will crawl anything they can get to while searching for email addresses. Make a honeypot page that does nothing but record hits and mention it only in the robots.txt file with a DISALLOW tag. I'll bet a nickle it gets hit.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/ Associated Messages, from the most recent to the oldest:

    
  1. Re: how do we turn off staticpages page creation? (Kenneth Grome 2003)
  2. Re: how do we turn off staticpages page creation? (Lucas Rockwell 2003)
  3. Re: how do we turn off staticpages page creation? (WebCat @ Inkblot Media 2003)
  4. Re: how do we turn off staticpages page creation? (Glenn Busbin 2003)
  5. Re: how do we turn off staticpages page creation? (WebCat @ Inkblot Media 2003)
  6. Re: how do we turn off staticpages page creation? (Chris List Recipient 2003)
  7. Re: how do we turn off staticpages page creation? (dbrooke 2003)
  8. Re: how do we turn off staticpages page creation? (Charles Kline 2003)
  9. Re: how do we turn off staticpages page creation? (Charles Kline 2003)
  10. how do we turn off staticpages page creation? (Lucas Rockwell 2003)
>Set a robot.txt file in that directory and set whatever pages you do not >want crawled. That will work for good bots, but there are those that look for the names of pages and ignore the DISALLOW tag. That's why it's best to have no reference at all, either on a template or the robots.txt file, to anything you don't want visited.The ^*#&@^$! spammer's address harvesters will crawl anything they can get to while searching for email addresses. Make a honeypot page that does nothing but record hits and mention it only in the robots.txt file with a DISALLOW tag. I'll bet a nickle it gets hit.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/ Glenn Busbin

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

WCS Newbie question (1997) Include Troubles (1998) Putting together a econimic server (2004) syntax question, not in online refernce (1997) Calendar Solutions (1998) WebCat b13 CGI -shownext- (1997) Referrer Problems (2000) Location of Browser Info.txt file (1997) Emailer (1997) Draft Manual, Tutorial, and more (1997) page redirect in webDNA (1997) CSS positioning!!! (2001) webcat and OS 8 (1997) select multiple 2 more cents (1997) $append (1998) test (2000) Emailer setup (1997) wierd [cart] action! (1997) WebMerchant (1997) WebCatalog 2.0 b 15 mac (1997)