Re: How do I get Google to crawl a WebCat site?
This WebDNA talk-list message is from 2003
It keeps the original formatting.
numero = 48652
interpreted = N
texte = > > >Don't use the keywords meta tagYou can use keywords META tags all you want. Some spiders ignore them, some use them, none penalize you for their use.Use
in the page headers. Modify as required.> >Do use the Description meta tag > >Code in all the alt tags for graphics.These may help ranking, but do nothing for getting a site spidered.See Google's webmaster tips for getting spidered by Googlebot.Googlebot reads the robots.txt file before spidering and looks to see if a home page exists before ever trying to spider a site. It also obeys the Allow: command in the robots.txt file, even though it's not in the robots.txt RFC. Use it anyway.Some bad bots will use the templates in the Disallow: command to spider pages you want left alone. Don't use that command and do not link to such templates from those that can be spidered unless you have good security for those templates (U/N and P/W's, for example).I doubt if Google or any other bot knows what a .tmpl or .tpl suffixes are. SM should, but prolly never has, tried to educated the bot owners about this. Use .htm or .html instead.Query strings can be read by some bots, but not all. In any event, URL's with query strings do not rank well compared to those without them.Some bots can accept a cookie now, but don't use it. It's just a way of spidering without being restricted to only those pages which require one. They still hit links they find, but without following them from one page to the next. Hence, no referrer for those hits shows in the logs.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Associated Messages, from the most recent to the oldest:
> > >Don't use the keywords meta tagYou can use keywords META tags all you want. Some spiders ignore them, some use them, none penalize you for their use.Use in the page headers. Modify as required.> >Do use the Description meta tag > >Code in all the alt tags for graphics.These may help ranking, but do nothing for getting a site spidered.See Google's webmaster tips for getting spidered by Googlebot.Googlebot reads the robots.txt file before spidering and looks to see if a home page exists before ever trying to spider a site. It also obeys the Allow: command in the robots.txt file, even though it's not in the robots.txt RFC. Use it anyway.Some bad bots will use the templates in the Disallow: command to spider pages you want left alone. Don't use that command and do not link to such templates from those that can be spidered unless you have good security for those templates (U/N and P/W's, for example).I doubt if Google or any other bot knows what a .tmpl or .tpl suffixes are. SM should, but prolly never has, tried to educated the bot owners about this. Use .htm or .html instead.Query strings can be read by some bots, but not all. In any event, URL's with query strings do not rank well compared to those without them.Some bots can accept a cookie now, but don't use it. It's just a way of spidering without being restricted to only those pages which require one. They still hit links they find, but without following them from one page to the next. Hence, no referrer for those hits shows in the logs.Glenn------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://webdna.smithmicro.com/
Glenn Busbin
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
WebCatalog for Postcards ? (1997)
I'm new be kind (1997)
problem: mail changed (1997)
If Empty ? (1997)
[ot] G5 Desktop as Server (2004)
[WebDNA] search command problem (2009)
WebCat2b13MacPlugIn - More limits on [include] (1997)
Can't use old cart file (was One more try) (1997)
Sorting error (1997)
Setting up WebCatalog with Retail Pro data (1996)
40,000+ items = mutiple dbs? (1999)
Hideif on IP range (2004)
DDE feature in webcat (1998)
[sendmail] on NT? (1997)
Nested tags count question (1997)
Running 2 two WebCatalog.acgi's (1996)
Searching (2005)
WebCommerce: Folder organization ? (1997)
Form Authentication (2000)
different show next (1997)