More Search Engine

This WebDNA talk-list message is from

2002


It keeps the original formatting.
numero = 44948
interpreted = N
texte = Since the search engine bots don't like to follow the query strings links, I may have a workaround that will let them spider pages anyhow.GoogleBot the FAST crawler, and others are good at obeying robots.txt instructions, so instead of using that to keep bots away from pages, maybe it can be used to attract them to others.I rewrote my robots.txt file to include a lot of Allow: abc.html along with my old Disallow: xyz.html instructions. In fact, I used writefile to create a new robobts.txt file so it can be updated as required.Maybe this will work on those templates that don't depend on a query string to tell them what to display. I'll have to wait for another crawl to find out.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

    
  1. More Search Engine (Glenn Busbin 2002)
Since the search engine bots don't like to follow the query strings links, I may have a workaround that will let them spider pages anyhow.GoogleBot the FAST crawler, and others are good at obeying robots.txt instructions, so instead of using that to keep bots away from pages, maybe it can be used to attract them to others.I rewrote my robots.txt file to include a lot of Allow: abc.html along with my old Disallow: xyz.html instructions. In fact, I used writefile to create a new robobts.txt file so it can be updated as required.Maybe this will work on those templates that don't depend on a query string to tell them what to display. I'll have to wait for another crawl to find out.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Glenn Busbin

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

website hosters using webcat? (1999) Getting total number of items ordered (1997) AuthorizeNet Declines (2005) Math confusion (2004) WebCat b13 CGI -shownext- (1997) 2.0Beta Command Ref (can't find this instruction) (1997) Running 2 two WebCatalog.acgi's (1996) Problems with [Applescript] (1997) webCatalog, OSX, and WebStar (2001) update on include (1997) triggers on Linux? (2000) Sorting by date (1997) [WebDNA] [OT] Kickstarter feedback please? (2015) Hello world ... (2002) Weird problems with [SHOWIF]s (1997) Not really WebCat (1997) [movefile] (1999) [WebDNA] current thinking on architecture of mass email scripts? (2011) Multiple Pulldowns (1997) flushdatabases (1997)