Re: Spiders

This WebDNA talk-list message is from

1998


It keeps the original formatting.
numero = 17010
interpreted = N
texte = Kevin,Examples, as well as other information about the robots.txt file and the Standard for Robot Exclusion can be found at:WendellKevin Mueller wrote:>This may seem like a rookie question, but here goes. > >I have noticed that when my site is accessed by a spider it first looks >for a file called SPIDER.TXT. >What is (or should be) contained in this file? >Is the presence of this file a solution the problems indexing sites with >dynamic URLs like those in a WebCat site? >I know this is on the fringes of WebCat but if someone can pull me out >of the dark on this one... : ) > >Thanks, >Kevin Mueller >iFolio Digital Image Library Associated Messages, from the most recent to the oldest:

    
  1. Re: Search Spiders (Glenn Busbin 2002)
  2. Re: Search Spiders (Glenn Busbin 2002)
  3. Re: Search Spiders (Glenn Busbin 2002)
  4. Search Spiders (Alain Russell 2002)
  5. Re: Spiders and Bots (Paul Uttermohlen 2000)
  6. Re: OT - Re: Spiders and Bots (Glenn Busbin 2000)
  7. OT - Re: Spiders and Bots (John Butler 2000)
  8. Re: Spiders and Bots (John Butler 2000)
  9. Re: Spiders and Bots (Glenn Busbin 2000)
  10. Re: Spiders and Bots (Glenn Busbin 2000)
  11. Re: Spiders and Bots (John Butler 2000)
  12. Re: Spiders and Bots (Jesse Proudman 2000)
  13. Re: Spiders and Bots (Bob Sneidar 2000)
  14. Re: Spiders and Bots (John Butler 2000)
  15. Re: Spiders and Bots (Aaron Lynch 2000)
  16. Re: Spiders and Bots (Glenn Busbin 2000)
  17. Re: Spiders and Bots (John Butler 2000)
  18. Spiders and Bots (Glenn Busbin 2000)
  19. Re: Spiders (Glenn Busbin 2000)
  20. Re: Spiders (Rob Marquardt 2000)
  21. Spiders (Glenn Busbin 2000)
  22. Re: Spiders, robots, whatever you want to call them (Christer Olsson 1998)
  23. Spiders, robots, whatever you want to call them (Sue Ann Sanders 1998)
  24. Re: Spiders (Wendell B. Kozak 1998)
  25. Re: Spiders (Bob Minor 1998)
  26. Re: Spiders (Kevin Mueller 1998)
  27. Spiders (Kevin Mueller 1998)
Kevin,Examples, as well as other information about the robots.txt file and the Standard for Robot Exclusion can be found at:WendellKevin Mueller wrote:>This may seem like a rookie question, but here goes. > >I have noticed that when my site is accessed by a spider it first looks >for a file called SPIDER.TXT. >What is (or should be) contained in this file? >Is the presence of this file a solution the problems indexing sites with >dynamic URLs like those in a WebCat site? >I know this is on the fringes of WebCat but if someone can pull me out >of the dark on this one... : ) > >Thanks, >Kevin Mueller >iFolio Digital Image Library Wendell B. Kozak

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

SKU Question (1999) Test (2002) Database changes (1998) [WebDNA] Accepting an XML via HTTP (2013) PIXO support (1997) FAX orders (1996) WebCat editing, SiteGuard WAS:SiteAssociative lookup style? (1997) PROBLEM (1997) NetSplat and WebCat2 (1997) Help! (2000) Separate SSL Server (1997) CC Processors question (2002) Whats going on? (2000) Concatenating Form Variables (1998) OT - quiet on the list today - here's something (2004) Custom Shipping Charges (1997) Rhapsody? (1997) searchable list archive (1997) date formatting for CC card expiration date check (1998) bug in [SendMail] (1997)