Re: [WebDNA] A new popuated field to a DB with 700.000 records
This WebDNA talk-list message is from 2009
It keeps the original formatting.
numero = 104094
interpreted = N
texte = here it is: ----start code snippet------------------------------------------------- [!]-------------------------------------------------------------- ** Hot Merge Example by Donovan ** ** Note: This snippet is within a [listfiles..] context that looks in a "hotfolder/" directory and is fired every 10 min. by a trigger. This snippet is also after integrity checks on the uploaded merge file. ** Note: [tNumRecs_Merge] value is from a [numfound] on the merge file after the integrity checks. -------------------------------------------------------------[/!] [!] ** Merge 500 records at a time ** [/!] [text]tNumLoops=3D[math]floor([tNumRecs_Merge]/500)[/math][/text] [!] ** Set some vars ** [/!] [text multi=3DT]hmt_index=3D1[!] [/!]&tKeyField=3DMERGE_KEY_FIELD[!] [/!]&tDestDB_Key=3DDEST_KEY_FIELD[!] [/!]&tDestDBPath=3Dyour/db/destination.db[!] [/!]&tNumSecs=3D10[!] ^ Edit the above more or less depending on size of merge. [/!][/text] [!] ** need to set some text vars because some WebDNA is not allowed in = spawn ** [/!] [text]tFilename=3D[filename][/text] [!] ** spawn a new process for each block of 500 ** [/!] [loop start=3D1&end=3D[math][tNumLoops]+1[/math]&advance=3D1] [spawn] [search db=3Dhotfolder/[tFilename][!] [/!]&ne[tKeyField]datarq=3Dfind_all[!] [/!]&startat=3D[hmt_index][!] [/!]&max=3D500] [founditems] [replace db=3D[tDestDBPath][!] [/!]&eq[tDestDB_Key]datarq=3D[interpret][[tKeyField]][/interpret]][!] [/!]&DestFirstField=3D[url][Merge1Field][/url][!] [/!]&DestSecondField=3D[url][Merge2Field][/url][!] etc.. [/!][/replace] [/founditems] [/search] [/spawn] [!] ** Wait <[tNumSecs]> seconds to start the next block ** [/!] [waitforfile file=3Dnon_existent_file.txt[!] [/!]&timeout=3D[tNumSecs]][/waitforfile] =09 [!] ** set index to next 500 block ** [/!] [text]hmt_index=3D[math][index]*500[/math][/text] [/loop] ----end code------------------------------------------------------- The basic idea was that I didn't really care how long the merge took, but rather, I wanted to make sure the processor wasn't overloaded. My idea was to use SPAWN and a waiting technique = using WAITFORFILE to "spread out" the task. This turned out to work really well I think. I used 'top -u apache' to monitor the process on a merge with 10,000 records, and I didn't see *any* noticeable heightened processor usage using this code. Just thought I'd pass this experiment along to the list! Donovan disclaimer: :) the above code was snipped out of a live working system, but to make it legible and universal, I rewrote a bit of it above so there could be some syntax errors from the rewrite. On Dec 3, 2009, at 2:24, Donovan Brooke wrote: > Palle, I wrote a merge system a while back that I may have posted to = the list. .. Not sure. The system automatically breaks the tasks into = chunks, only you don't have to baby sit it. >=20 > Maybe check both archives.... >=20 > Donovan >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > d.brooke - mobile > www.euca.us > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > On Dec 2, 2009, at 3:15 PM, Palle Bo Nielsen =
wrote: >=20 >> Dan, >>=20 >> I understand where you are getting but it would be preferable to have = the IDs immediately. >>=20 >> I will let my questions live for 24 hours or so and then move to plan = B (in chunks) unless the better idea pops up from the list or my own = pillow tonight. >>=20 >> Palle >>=20 >>=20 >>=20 >> On 02/12/2009, at 22.06, Dan Strong wrote: >>=20 >>>> 1) Yes, but is not sequential. >>>=20 >>> Ok, next question: do the new, unique sequential IDs need to be in = there immediately? If not, perhaps add the new uniques as the .db is = otherwise being hit? >>>=20 >>> -Dan Strong >>> http://www.DanStrong.com >>>=20 >>> Palle Bo Nielsen wrote: >>>> Dan, >>>> 1) Yes, but is not sequential. >>>> 2) Yes, same result - on it's knees. >>>> 3) This is my next plan, but it would be nice to get it done in one = process. >>>> Palle >>>> On 02/12/2009, at 21.56, Dan Strong wrote: >>>>> 1) Is there any other unique identifier in each record? >>>>> 2) Have you tried [replacefounditems]? >>>>> 3) Maybe "chunk it" to only do, say, 500 at a time? >>>>>=20 >>>>> -Dan Strong >>>>> http://www.DanStrong.com >>>>>=20 >>>>> Palle Bo Nielsen wrote: >>>>>> Hi all, >>>>>> I got a DB which has 11 fields and more that 700.000 lines of = data. >>>>>> Now I need to add a new field and I need to populate this field = with an index from the number 1 to the most recent number which for = example is 700.000. >>>>>> The number 1 is the first lines of data within this BD and the = 700.000 is the most recent added line of data. >>>>>> If have done some test and sand-boxing and every time WebDNA goes = into it's knees. >>>>>> I need some good advise from you all to figure out the best = solution. >>>>>> I look forward to your suggestions. >>>>>> **** this one is rough and breaks WebDNA **** >>>>>> [search = db=3Dl4.db&nel4_statusdata=3Ddummy&asl4_serialsort=3D1][numFound][foundIte= ms] >>>>>> [replace = db=3Dl4.db&eql4_skudata=3D[l4_sku]]l4_index=3D[index][/replace] >>>>>> [/foundItems][/search] >>>>>> **** snip **** >>>>>> Palle--------------------------------------------------------- >>>>>> This message is sent to you because you are subscribed to >>>>>> the mailing list . >>>>>> To unsubscribe, E-mail to: >>>>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>>> --------------------------------------------------------- >>>>> This message is sent to you because you are subscribed to >>>>> the mailing list . >>>>> To unsubscribe, E-mail to: >>>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>>>=20 >>>> --------------------------------------------------------- >>>> This message is sent to you because you are subscribed to >>>> the mailing list . >>>> To unsubscribe, E-mail to: >>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>> --------------------------------------------------------- >>> This message is sent to you because you are subscribed to >>> the mailing list . >>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us >>> old archives: http://dev.webdna.us/TalkListArchive/ >>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>=20 >>=20 >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list . >> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us >> old archives: http://dev.webdna.us/TalkListArchive/ >> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288
Associated Messages, from the most recent to the oldest:
here it is: ----start code snippet------------------------------------------------- [!]-------------------------------------------------------------- ** Hot Merge Example by Donovan ** ** Note: This snippet is within a [listfiles..] context that looks in a "hotfolder/" directory and is fired every 10 min. by a trigger. This snippet is also after integrity checks on the uploaded merge file. ** Note: [tNumRecs_Merge] value is from a [numfound] on the merge file after the integrity checks. -------------------------------------------------------------[/!] [!] ** Merge 500 records at a time ** [/!] [text]tNumLoops=3D[math]floor([tNumRecs_Merge]/500)[/math][/text] [!] ** Set some vars ** [/!] [text multi=3DT]hmt_index=3D1[!] [/!]&tKeyField=3DMERGE_KEY_FIELD[!] [/!]&tDestDB_Key=3DDEST_KEY_FIELD[!] [/!]&tDestDBPath=3Dyour/db/destination.db[!] [/!]&tNumSecs=3D10[!] ^ Edit the above more or less depending on size of merge. [/!][/text] [!] ** need to set some text vars because some WebDNA is not allowed in = spawn ** [/!] [text]tFilename=3D[filename][/text] [!] ** spawn a new process for each block of 500 ** [/!] [loop start=3D1&end=3D[math][tNumLoops]+1[/math]&advance=3D1] [spawn] [search db=3Dhotfolder/[tFilename][!] [/!]&ne[tKeyField]datarq=3Dfind_all[!] [/!]&startat=3D[hmt_index][!] [/!]&max=3D500] [founditems] [replace db=3D[tDestDBPath][!] [/!]&eq[tDestDB_Key]datarq=3D[interpret][[tKeyField]][/interpret]][!] [/!]&DestFirstField=3D[url][Merge1Field][/url][!] [/!]&DestSecondField=3D[url][Merge2Field][/url][!] etc.. [/!][/replace] [/founditems] [/search] [/spawn] [!] ** Wait <[tNumSecs]> seconds to start the next block ** [/!] [waitforfile file=3Dnon_existent_file.txt[!] [/!]&timeout=3D[tNumSecs]][/waitforfile] =09 [!] ** set index to next 500 block ** [/!] [text]hmt_index=3D[math][index]*500[/math][/text] [/loop] ----end code------------------------------------------------------- The basic idea was that I didn't really care how long the merge took, but rather, I wanted to make sure the processor wasn't overloaded. My idea was to use SPAWN and a waiting technique = using WAITFORFILE to "spread out" the task. This turned out to work really well I think. I used 'top -u apache' to monitor the process on a merge with 10,000 records, and I didn't see *any* noticeable heightened processor usage using this code. Just thought I'd pass this experiment along to the list! Donovan disclaimer: :) the above code was snipped out of a live working system, but to make it legible and universal, I rewrote a bit of it above so there could be some syntax errors from the rewrite. On Dec 3, 2009, at 2:24, Donovan Brooke wrote: > Palle, I wrote a merge system a while back that I may have posted to = the list. .. Not sure. The system automatically breaks the tasks into = chunks, only you don't have to baby sit it. >=20 > Maybe check both archives.... >=20 > Donovan >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > d.brooke - mobile > www.euca.us > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > On Dec 2, 2009, at 3:15 PM, Palle Bo Nielsen = wrote: >=20 >> Dan, >>=20 >> I understand where you are getting but it would be preferable to have = the IDs immediately. >>=20 >> I will let my questions live for 24 hours or so and then move to plan = B (in chunks) unless the better idea pops up from the list or my own = pillow tonight. >>=20 >> Palle >>=20 >>=20 >>=20 >> On 02/12/2009, at 22.06, Dan Strong wrote: >>=20 >>>> 1) Yes, but is not sequential. >>>=20 >>> Ok, next question: do the new, unique sequential IDs need to be in = there immediately? If not, perhaps add the new uniques as the .db is = otherwise being hit? >>>=20 >>> -Dan Strong >>> http://www.DanStrong.com >>>=20 >>> Palle Bo Nielsen wrote: >>>> Dan, >>>> 1) Yes, but is not sequential. >>>> 2) Yes, same result - on it's knees. >>>> 3) This is my next plan, but it would be nice to get it done in one = process. >>>> Palle >>>> On 02/12/2009, at 21.56, Dan Strong wrote: >>>>> 1) Is there any other unique identifier in each record? >>>>> 2) Have you tried [replacefounditems]? >>>>> 3) Maybe "chunk it" to only do, say, 500 at a time? >>>>>=20 >>>>> -Dan Strong >>>>> http://www.DanStrong.com >>>>>=20 >>>>> Palle Bo Nielsen wrote: >>>>>> Hi all, >>>>>> I got a DB which has 11 fields and more that 700.000 lines of = data. >>>>>> Now I need to add a new field and I need to populate this field = with an index from the number 1 to the most recent number which for = example is 700.000. >>>>>> The number 1 is the first lines of data within this BD and the = 700.000 is the most recent added line of data. >>>>>> If have done some test and sand-boxing and every time WebDNA goes = into it's knees. >>>>>> I need some good advise from you all to figure out the best = solution. >>>>>> I look forward to your suggestions. >>>>>> **** this one is rough and breaks WebDNA **** >>>>>> [search = db=3Dl4.db&nel4_statusdata=3Ddummy&asl4_serialsort=3D1][numFound][foundIte= ms] >>>>>> [replace = db=3Dl4.db&eql4_skudata=3D[l4_sku]]l4_index=3D[index][/replace] >>>>>> [/foundItems][/search] >>>>>> **** snip **** >>>>>> Palle--------------------------------------------------------- >>>>>> This message is sent to you because you are subscribed to >>>>>> the mailing list . >>>>>> To unsubscribe, E-mail to: >>>>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>>> --------------------------------------------------------- >>>>> This message is sent to you because you are subscribed to >>>>> the mailing list . >>>>> To unsubscribe, E-mail to: >>>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>>>=20 >>>> --------------------------------------------------------- >>>> This message is sent to you because you are subscribed to >>>> the mailing list . >>>> To unsubscribe, E-mail to: >>>> archives: http://mail.webdna.us/list/talk@webdna.us >>>> old archives: http://dev.webdna.us/TalkListArchive/ >>>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>> --------------------------------------------------------- >>> This message is sent to you because you are subscribed to >>> the mailing list . >>> To unsubscribe, E-mail to: >>> archives: http://mail.webdna.us/list/talk@webdna.us >>> old archives: http://dev.webdna.us/TalkListArchive/ >>> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 >>>=20 >>=20 >> --------------------------------------------------------- >> This message is sent to you because you are subscribed to >> the mailing list . >> To unsubscribe, E-mail to: >> archives: http://mail.webdna.us/list/talk@webdna.us >> old archives: http://dev.webdna.us/TalkListArchive/ >> Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288 > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ > Bug Reporting: = http://forum.webdna.us/eucabb.html?page=3Dtopics&category=3D288
christophe.billiottet@webdna.us
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
founditem align (1998)
[CART] inside a [LOOP] (1997)
Error: Can't open order file. Ignoring [OrderFile] context (2000)
[OT] Am I an Idiot? (2004)
[WebDNA] Enhancements to "list" contexts ... (2015)
re: Large databases in WebCat (1997)
Redirect frame targets (1998)
orderfile headers (was: 2nd Request for help/adviceonvariable pricing) (2000)
RE: Displaying Location (1997)
browser info.txt and SSL (1997)
your mail (2000)
Catalogs and W* (1996)
Where's Cart Created ? (1997)
Authenticate (1997)
re: Large databases in WebCat (1997)
Number of Line Items (1998)
Search (1997)
Price Not Appearing (2000)
No subject given (2000)
Sorting Numbers (1997)