Re: [WebDNA] Limits?

This WebDNA talk-list message is from

2009


It keeps the original formatting.
numero = 102333
interpreted = N
texte = Psi Prime, Matthew A Perosi wrote: > In relation to this topic. > > One of my production servers has 100 websites on it, all running in each > their own sandbox. > The server has run solid, without a hiccup for 4 years. > > In January I notices that my Triggers were being deleted. Randomly. I > tried to figure out if there was a pattern to it all, and discovered > that they would be deleted between 4:30am - 5:30am right during the > height of my backup process. > > A few weeks later the entire server, all websites, started to experience > a 45-50second delay in loading pages. The WebDNA engine was practically > halting on every page load. > > Here's what I figured out: > Each of the 100 websites has 120+ databases. > Each website had 3 databases dedicated to user tracking (a record of > every product and page viewed), that's 300 tracking databases. > Of the 300 tracking databases, about 10 of them had reached 300MB+ in > size, that's 3GB. > I only have 4GB RAM on that server. > The massive slow down was caused by the databases needing time to > commit, flush and reload. > Once I deactivated the tracking on the 10 databases the server > immediately went back to normal operations. > > Getting back to the trigger issue... > What I later realized was that during the backup process every night the > server creates a zip file of each website and saves it into a temp > folder, then transfers it to my backup machine. Whenever the server > started the zip of a very large website it was causing a memory overflow > and all the WebDNA databases were being pushed out of memory before they > could finish committing and flushing. I would find a few empty database > files on the websites, but I always found all of my active Triggers.db > cleared out. I also would find a partially saved file called > Triggers.db.TeMpOr (i can't actually remember the real temporary file > name, but it was something like that.) in every Sandbox. This Trigger > issue was resolved the same day I fixed the issue with the 300MB > databases I described above. > > Matthew A Perosi JewelerWebsites.com > ------------------------------by Psi Prime------- > Senior Web Developer 323 Union Blvd. Interesting story. In Working *on the edge* with WebDNA databases (tables), I think there are a few things to consider: 1.) Size of DB's 2.) How often they are accessed 3.) the code used to access them 4.) how many databases are being accessed. 5.) Your architecture and amount of RAM I think you are probably pushing WebDNA's table limits regarding some of these points. Which, is interesting for us all, and makes for a good test case. ;-) I suppose if I were you, I would start thinking about migrating one or two of your largest databases to MySQL and using WebDNA's newer SQL contexts to access them. However, if you wanted to be brave and continue.. it sounds like you are one of the rare cases where our new 64bit build, along with more RAM, may help your situation. (If you are on a linux machine) One of our near-top development priorities is to ensure that the MySQL integration is up-to-date and is performing rock solid. It's current state is an API using an older library, but has been solid in the *minimal* tests that we've performed so far. When we do get to updating this, it will take some time as it will require a lot of rewriting. Anyway, if you decide to try out MySQL (Perhaps first in a dev environment, or in limited live situations), it would be good to keep in touch with you so that we can get your feedback. Donovan P.S. the .TMP file is created because you have "safe write" turned on in your prefs. -- Donovan D. Brooke PH: 1 (608) 770-3822 ------------------------------------------------ VP WebDNA Software Corporation 16192 Coastal Highway Lewes, DE 19958 Associated Messages, from the most recent to the oldest:

    
  1. Re: [WebDNA] Limits? (rupert ravens 2009)
  2. Re: [WebDNA] Limits? (christophe.billiottet@webdna.us 2009)
  3. Re: [WebDNA] Limits? ("Psi Prime, Matthew A Perosi " 2009)
  4. Re: [WebDNA] Limits? (rupert ravens 2009)
  5. Re: [WebDNA] Limits? ("Psi Prime, Matthew A Perosi " 2009)
  6. Re: [WebDNA] Limits? (Donovan Brooke 2009)
  7. Re: [WebDNA] Limits? (Dale - Ehome 2009)
  8. RE: [WebDNA] Limits? ("Olin Lagon" 2009)
  9. Re: [WebDNA] Limits? (Bob Minor 2009)
  10. Re: [WebDNA] Limits? ("Psi Prime, Matthew A Perosi " 2009)
  11. RE: [WebDNA] Limits? ("Michael A. DeLorenzo" 2009)
  12. Re: [WebDNA] Limits? (Govinda 2009)
  13. Re: [WebDNA] Limits? (Donovan Brooke 2009)
  14. Re: [WebDNA] Limits? (christophe.billiottet@webdna.us 2009)
  15. [WebDNA] Limits? (Bob Minor 2009)
Psi Prime, Matthew A Perosi wrote: > In relation to this topic. > > One of my production servers has 100 websites on it, all running in each > their own sandbox. > The server has run solid, without a hiccup for 4 years. > > In January I notices that my triggers were being deleted. Randomly. I > tried to figure out if there was a pattern to it all, and discovered > that they would be deleted between 4:30am - 5:30am right during the > height of my backup process. > > A few weeks later the entire server, all websites, started to experience > a 45-50second delay in loading pages. The WebDNA engine was practically > halting on every page load. > > Here's what I figured out: > Each of the 100 websites has 120+ databases. > Each website had 3 databases dedicated to user tracking (a record of > every product and page viewed), that's 300 tracking databases. > Of the 300 tracking databases, about 10 of them had reached 300MB+ in > size, that's 3GB. > I only have 4GB RAM on that server. > The massive slow down was caused by the databases needing time to > commit, flush and reload. > Once I deactivated the tracking on the 10 databases the server > immediately went back to normal operations. > > Getting back to the trigger issue... > What I later realized was that during the backup process every night the > server creates a zip file of each website and saves it into a temp > folder, then transfers it to my backup machine. Whenever the server > started the zip of a very large website it was causing a memory overflow > and all the WebDNA databases were being pushed out of memory before they > could finish committing and flushing. I would find a few empty database > files on the websites, but I always found all of my active triggers.db > cleared out. I also would find a partially saved file called > triggers.db.TeMpOr (i can't actually remember the real temporary file > name, but it was something like that.) in every Sandbox. This Trigger > issue was resolved the same day I fixed the issue with the 300MB > databases I described above. > > Matthew A Perosi JewelerWebsites.com > ------------------------------by Psi Prime------- > Senior Web Developer 323 Union Blvd. Interesting story. In Working *on the edge* with WebDNA databases (tables), I think there are a few things to consider: 1.) Size of DB's 2.) How often they are accessed 3.) the code used to access them 4.) how many databases are being accessed. 5.) Your architecture and amount of RAM I think you are probably pushing WebDNA's table limits regarding some of these points. Which, is interesting for us all, and makes for a good test case. ;-) I suppose if I were you, I would start thinking about migrating one or two of your largest databases to MySQL and using WebDNA's newer SQL contexts to access them. However, if you wanted to be brave and continue.. it sounds like you are one of the rare cases where our new 64bit build, along with more RAM, may help your situation. (If you are on a linux machine) One of our near-top development priorities is to ensure that the MySQL integration is up-to-date and is performing rock solid. It's current state is an API using an older library, but has been solid in the *minimal* tests that we've performed so far. When we do get to updating this, it will take some time as it will require a lot of rewriting. Anyway, if you decide to try out MySQL (Perhaps first in a dev environment, or in limited live situations), it would be good to keep in touch with you so that we can get your feedback. Donovan P.S. the .TMP file is created because you have "safe write" turned on in your prefs. -- Donovan D. Brooke PH: 1 (608) 770-3822 ------------------------------------------------ VP WebDNA Software Corporation 16192 Coastal Highway Lewes, DE 19958 Donovan Brooke

DOWNLOAD WEBDNA NOW!

Top Articles:

Talk List

The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...

Related Readings:

Problem with date and ranges (2002) Reversed words (1997) Tea Room trouble (1997) [OT] Domain Management (2003) Anyone Using 4DMail (2004) Method=GET v Method=POST (2001) [WebDNA] Quick sort/results grouping Q (2008) getchars broken? (1997) Problems with [Applescript] (1997) emailer (1997) Hosting (2000) Protect (1997) WebCat2b13MacPlugIn - [showif][search][/showif] (1997) Just Testing (1997) WebDelivery downloads alias, not original ? (1997) [WebDNA] Mass email using WebDNA (2008) webcat and webkitty votes needed (1997) Automated FTP from Shell - Update (2004) [WebDNA] New problem with [ShowNext] (2010) E-mailer error codes (1997)