Re: [WebDNA] Some code inside a large loops fail ... but why?
This WebDNA talk-list message is from 2010
It keeps the original formatting.
numero = 104434
interpreted = N
texte = Thanks Brian, that's one part of the equation. Now all I have to do is figure out the other parts and I can go back to using webdna directly instead of via the browser ... although I'm not sure I want to given the way the browser technique seems to be working for me right now. Sincerely, Kenneth Grome www.KenGrome.com > Text variables are retained in the spawned thread, but containing context (loop, founditems, etc) status is not. So, you can set a text variable to the loop's index value before beginning your spawn. For example: > > [writefile test.txt][date] [time] > [/writefile] > [loop start=1&end=10] > [text]x=[index][/text] > [spawn][appendfile test.txt][x] > [/appendfile][/spawn] > [/loop] > > > results in the file test.txt containing: > > 01/12/2010 21:19:49 > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > 9 > 10 > > > > Brian Fries > BrainScan Software > > > On Jan 12, 2010, at 9:13 PM, Kenneth Grome wrote: > > >> Looking at the docs for [spawn] I see that you may have to > >> stagger (in time) those spawns since too many at once will > >> weigh down the server too much). > > > > I tried spawn but the loop index was not passed into the spawn context so the tcpconnect which needs that index fails. If it actually worked the huge number of spawns would bring down my webdna server or the data source server unle > > > > The best solution is really the simplest, but only if the tcpconnect has a timeout or never fails. It is very simple to hit the server consecutively with the code I already posted, and if the tcpconnect never failed I could get all 300k records from that simple little bit of code. > > > > > >> Could you spawn/divide the task into batches so that if one > >> tcpconnect fails then it will only spoil that one batch? > > > > Possibly, but first I would have to figure out how to pass the loop index value into the spawn context, then I would have to figure out how to create the groupings so that only a small number of tcpconnects were spawned at one time. And then I would have to figure out how to control the speed that each spawned thread is created so they are not all spawned at once. As you can see this is very complicated compared with the simple code I posted earlier. > > > > For now I'm using a meta-refresh in my browser to hit that same code with a loop of only 10 records at a time. If the browser fails I will know it because the page won't be returned, instead an error will be displayed. Then I can easily look in the db to find the last record appended and continue from there. > > > > The browser technique is possible only because I'm using it to initially populate the db. It wouldn't be practical on a regular basis. After the db gets populated initially, adding new records will be far simpler. > > > > I was hoping to use tcpconnect to do the initial population but without a timeout everything grinds to a halt with no warnings or errors, and I don't think there is any way to create an error message (or send an error email) once the tcpconnect has failed because as soon as that happens no other code is processed. > > > > Sincerely, > > Kenneth Grome > > www.KenGrome.com > > > > > > > > > > > > > > --------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list
. > > To unsubscribe, E-mail to: > > archives: http://mail.webdna.us/list/talk@webdna.us > > old archives: http://dev.webdna.us/TalkListArchive/ > > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 >
Associated Messages, from the most recent to the oldest:
Thanks Brian, that's one part of the equation. Now all I have to do is figure out the other parts and I can go back to using webdna directly instead of via the browser ... although I'm not sure I want to given the way the browser technique seems to be working for me right now. Sincerely, Kenneth Grome www.KenGrome.com > Text variables are retained in the spawned thread, but containing context (loop, founditems, etc) status is not. So, you can set a text variable to the loop's index value before beginning your spawn. For example: > > [writefile test.txt][date] [time] > [/writefile] > [loop start=1&end=10] > [text]x=[index][/text] > [spawn][appendfile test.txt][x] > [/appendfile][/spawn] > [/loop] > > > results in the file test.txt containing: > > 01/12/2010 21:19:49 > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > 9 > 10 > > > > Brian Fries > BrainScan Software > > > On Jan 12, 2010, at 9:13 PM, Kenneth Grome wrote: > > >> Looking at the docs for [spawn] I see that you may have to > >> stagger (in time) those spawns since too many at once will > >> weigh down the server too much). > > > > I tried spawn but the loop index was not passed into the spawn context so the tcpconnect which needs that index fails. If it actually worked the huge number of spawns would bring down my webdna server or the data source server unle > > > > The best solution is really the simplest, but only if the tcpconnect has a timeout or never fails. It is very simple to hit the server consecutively with the code I already posted, and if the tcpconnect never failed I could get all 300k records from that simple little bit of code. > > > > > >> Could you spawn/divide the task into batches so that if one > >> tcpconnect fails then it will only spoil that one batch? > > > > Possibly, but first I would have to figure out how to pass the loop index value into the spawn context, then I would have to figure out how to create the groupings so that only a small number of tcpconnects were spawned at one time. And then I would have to figure out how to control the speed that each spawned thread is created so they are not all spawned at once. As you can see this is very complicated compared with the simple code I posted earlier. > > > > For now I'm using a meta-refresh in my browser to hit that same code with a loop of only 10 records at a time. If the browser fails I will know it because the page won't be returned, instead an error will be displayed. Then I can easily look in the db to find the last record appended and continue from there. > > > > The browser technique is possible only because I'm using it to initially populate the db. It wouldn't be practical on a regular basis. After the db gets populated initially, adding new records will be far simpler. > > > > I was hoping to use tcpconnect to do the initial population but without a timeout everything grinds to a halt with no warnings or errors, and I don't think there is any way to create an error message (or send an error email) once the tcpconnect has failed because as soon as that happens no other code is processed. > > > > Sincerely, > > Kenneth Grome > > www.KenGrome.com > > > > > > > > > > > > > > --------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > archives: http://mail.webdna.us/list/talk@webdna.us > > old archives: http://dev.webdna.us/TalkListArchive/ > > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: http://forum.webdna.us/eucabb.html?page=topics&category=288 >
Kenneth Grome
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
BBEdit and WebCatalog 2.0? (1997)
[WebDNA] [OT] WebDNA Wiki - need grammar check / suggestions (2009)
Running 2 two WebCatalog.acgi's (1996)
Setting up shop (1997)
Is setting a basic lineitem in the cart really magic? (2000)
Slightly OT - SQL- Clustering (2006)
WebCatalog Technical Reference (1997)
Banner Ad (1998)
webDNA engine creating duplicate lineitems?! (2004)
Queertrons? (1997)
case sensitivity in lookups (1997)
RE: Formulas.db + Users.db (1997)
Using [showif] within [lineitems] (2000)
Sendmail doesn't (was Queertrons?) (1997)
Wanted how to (1-2-3) (1999)
Hello??? (1997)
OT: Amazon Patents (2000)
RE: 2nd WebCatalog2 Feature Request (1996)
[OT] Any perl gurus out there? (2006)
WebCat2 - [format thousands] (1997)