Re: Search Engine questions ...
This WebDNA talk-list message is from 2002
It keeps the original formatting.
numero = 44841
interpreted = N
texte = >Try this: >Write a script that records page access and writes this to a db: >URL, Date, Time, IPAddress > >Stick it at the top of any page on your site that is available to a spider.No ... what you should be doing is sticking it into any webdna-processed page that is available to a spider. Because if you stick it into just any old page -- which can include pages which are not processed by webcatalog -- then of course the webdna will not be processed, and no entry will be made in your db file. >Wait until your web logs show that the page has been hit again by a >spider. There will be no record in the db indicating that a spider >has seen the page.Then I think your webdna code is not working properly, or you're confusing which pages have actually been requested by the spider and which pages you think have been requested.If your webdna code were working properly, and if the page you THINK is being requested is actually the one the spider has requested, then webdna will record every access to that page. It doesn't have any other choice ... does it? >The inktomi spider is an exception but Googlebot, FAST, the usual >spam spiders, and others simply do not appear in the db.Then what pages are these others visiting? Or are Google and the others so fast that they can *outrun* webdna's ability to record every page request? Is that what you're saying? But inktomi is too slow compared to the others, so those requests get logged? I would like to see some proof of your claims -- because if it's actually true, then I think it points to a bug in the webdna software and not some unusual thing going on in the search engine requests! Sincerely, Kenneth Grome--------------------------------------------------- WebDNA Professional Training and Development Center 175 J. Llorente Street +63 (32) 255-6921 Cebu City, Cebu 6000 kengrome@webdna.net Philippines http://www.webdna.net ---------------------------------------------------------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list
. To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Associated Messages, from the most recent to the oldest:
>Try this: >Write a script that records page access and writes this to a db: >URL, Date, Time, IPAddress > >Stick it at the top of any page on your site that is available to a spider.No ... what you should be doing is sticking it into any webdna-processed page that is available to a spider. Because if you stick it into just any old page -- which can include pages which are not processed by webcatalog -- then of course the webdna will not be processed, and no entry will be made in your db file. >Wait until your web logs show that the page has been hit again by a >spider. There will be no record in the db indicating that a spider >has seen the page.Then I think your webdna code is not working properly, or you're confusing which pages have actually been requested by the spider and which pages you think have been requested.If your webdna code were working properly, and if the page you THINK is being requested is actually the one the spider has requested, then webdna will record every access to that page. It doesn't have any other choice ... does it? >The inktomi spider is an exception but Googlebot, FAST, the usual >spam spiders, and others simply do not appear in the db.Then what pages are these others visiting? Or are Google and the others so fast that they can *outrun* webdna's ability to record every page request? Is that what you're saying? But inktomi is too slow compared to the others, so those requests get logged? I would like to see some proof of your claims -- because if it's actually true, then I think it points to a bug in the webdna software and not some unusual thing going on in the search engine requests! Sincerely, Kenneth Grome--------------------------------------------------- WebDNA Professional Training and Development Center 175 J. Llorente Street +63 (32) 255-6921 Cebu City, Cebu 6000 kengrome@webdna.net Philippines http://www.webdna.net ---------------------------------------------------------------------------------------------------------------- This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/
Kenneth Grome
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
WebStar Secure on other machine (1997)
Alternative to Authorize.net (2002)
Redirect frame targets (1998)
PCS Frames (1997)
Using Cookie for client specific info? (1997)
includes and cart numbers (1997)
WCS Newbie question (1997)
setheader and retrieve it immediatly (2001)
Removing and Reinstalling Typhoon (2000)
lookup and two records? (1997)
I assume it is a safe bet this does not help (1998)
emailer setup (1997)
Emailer (1997)
Document Contains No Data! (1997)
2nd WebCatalog2 Feature Request (1996)
Mac Scheduling (1998)
Protect vs Authenicate (1997)
Site Specific Shopping Cart folder and Orderfiles (2000)
Migrating to NT (1997)
WebCatalog Prefs.ref file ? (1998)