Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether?

This WebDNA talk-list message is from

2009


It keeps the original formatting.
numero = 103039
interpreted = N
texte = Just wanted to add how i split very large bases: imagine a base with this kind of data customer name: customer address: customer tel: item title: item description: auction number: price: Splitting the large base in 3 parts: one "customers" base, one "items" base and one extra "bridge" base that lists the owner reference number associated with the item reference number. The [search] tag is used on the "bridge" base and [lookup] tags will recover the proper data from "customers" and "items". Since [lookup] is extremely fast compared to [search], the global result is significantly faster than a single base. Of course, as Patrick wrote, this does does require common sense and strategy, depending on your data and how to use it. - chris On Jul 28, 2009, at 7:38, christophe.billiottet@webdna.us wrote: > My 2 cents: i already worked with projects including 450,000 records > (full descriptions of antics for an auction house) but even this was > far smaller than 1GB (a little less than 200MB if i remember). > Anyway, what i did was split the database in chunks, each items with > its unique reference number, using [LOOKUP] instead of [SEARCH] > whenever possible, using "Only commit databases to disk when > instructed" and manually [COMMITDATABASE] to disk and i must say > WebDNA performs remarkably well: fast, stable, it runs like this on > a 4GB RAM G5 Mac with a 15,000rpms SCSI disk for 4 years now. Since > this customer usually forgets to reboot, i do it myself from time to > time, every 4 or 5 months. > Just remember that writing 1GB to a disk, even a fast one, takes > several tens of seconds. If you get many entries to your database, > it is far better to operate in RAM and carefully commit in order to > do not queue several [COMMITDATABASE] operations. > > - chris > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ Associated Messages, from the most recent to the oldest:

    
  1. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (christophe.billiottet@webdna.us 2009)
  2. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Palle Bo Nielsen 2009)
  3. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Palle Bo Nielsen 2009)
  4. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Govinda 2009)
  5. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Palle Bo Nielsen 2009)
  6. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Govinda 2009)
  7. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (christophe.billiottet@webdna.us 2009)
  8. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (christophe.billiottet@webdna.us 2009)
  9. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Govinda 2009)
  10. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Patrick McCormick 2009)
  11. Re: [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Govinda 2009)
  12. [WebDNA] 3-5 GB of native WebDNA db in RAM?, else MySQL w/WebDNA [SQL] tags, else skip WebDNA altogether? (Govinda 2009)
Just wanted to add how i split very large bases: imagine a base with this kind of data customer name: customer address: customer tel: item title: item description: auction number: price: Splitting the large base in 3 parts: one "customers" base, one "items" base and one extra "bridge" base that lists the owner reference number associated with the item reference number. The [search] tag is used on the "bridge" base and [lookup] tags will recover the proper data from "customers" and "items". Since [lookup] is extremely fast compared to [search], the global result is significantly faster than a single base. Of course, as Patrick wrote, this does does require common sense and strategy, depending on your data and how to use it. - chris On Jul 28, 2009, at 7:38, christophe.billiottet@webdna.us wrote: > My 2 cents: i already worked with projects including 450,000 records > (full descriptions of antics for an auction house) but even this was > far smaller than 1GB (a little less than 200MB if i remember). > Anyway, what i did was split the database in chunks, each items with > its unique reference number, using [lookup] instead of [search] > whenever possible, using "Only commit databases to disk when > instructed" and manually [commitdatabase] to disk and i must say > WebDNA performs remarkably well: fast, stable, it runs like this on > a 4GB RAM G5 Mac with a 15,000rpms SCSI disk for 4 years now. Since > this customer usually forgets to reboot, i do it myself from time to > time, every 4 or 5 months. > Just remember that writing 1GB to a disk, even a fast one, takes > several tens of seconds. If you get many entries to your database, > it is far better to operate in RAM and carefully commit in order to > do not queue several [commitdatabase] operations. > > - chris > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > old archives: http://dev.webdna.us/TalkListArchive/ christophe.billiottet@webdna.us

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Newbie problem blah blah blah (1997) [WebDNA] sum in a search (2009) WebCat2 Append problem (B14Macacgi) (1997) multiple credit card merchant accounts/processors (1997) This might sound silly... (2000) (2000) Exclamation point (1997) webcat restarting script on Linux? (1999) apostrophe in search item (1997) Separate SSL Server (1997) Cancel Subscription (1996) know it all (1998) WebCat for Unix?? (1997) WebCatalog Work (2001) Nesting format tags (1997) Linux Beta4 problems with multirecord returns (2000) How Many SKU's is enough? (1997) About WebCatalog and iTools 7 (2003) Sku numbers (1997) WebMerchant Error (1998)