Re: Search Engine bots
This WebDNA talk-list message is from 2002
It keeps the original formatting.
numero = 44758
interpreted = N
texte = >Hi, I need this clarified once and for all. >I have a domain with a default page at the root >called default.tpl. > >Will a search engine robot (specifically googlebot) >see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
>Hi, I need this clarified once and for all. >I have a domain with a default page at the root >called default.tpl. > >Will a search engine robot (specifically googlebot) >see this page?Probably, and it may also follow some dynamically generated links on that page, but it may also refuse to follow some of them. I'd use Default.html just to be a bit on the safe side. Google has info for webmasters about this. See their site map. It doesn't mention .tpl suffixes, but that's no surprise.AllTheWeb won't follow dynamically generated links, so I make them static. I don't know about the other spiders.Your robots.txt file is ok, but there are bad bots who not only ignore the Disallow stuff, they use it to crawl pages they wouldn't otherwise know about.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Glenn Busbin
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Re:listfiles-looking for slick solution (1997)
[HIDEIF] inside [FOUNDITEM] (1997)
[WebDNA] feature request - .hdr files (2011)
Filemaker Pro and [convertchars] (2000)
NT b19 sends extra MIME headers (1997)
Programmer Needed (1998)
[WebDNA] Does [referrer] get lost when server switches from http (2020)
& not allowed in db by definition? (1999)
showif and cart (1997)
Stop page auto. refreshing (2004)
Closing Databases (1998)
The Guru Speaks-Is That Really the Only Manuel? (1998)
username/password dialogue (2001)
(1997)
Wierd problem with 3.08 WIN Version (2000)
Re[2]: 2nd WebCatalog2 Feature Request (1996)
sendmail (2000)
problems with 2 tags (1997)
Site Check (2006)
Permissions (2000)