Re: SMSI: databases corrupted on crash and permission issues on

This WebDNA talk-list message is from

2002

It keeps the original formatting. numero = 43622
interpreted = N
texte = Scott, just please test this extensively with big DBs.We have a 300 meg database, and doing certain deletes takes *forever*On 9/19/02 7:49 PM, Scott Anderson mashed the following keys :>> In the meantime I have to ask: why doesn't webcat write databases to>> a temp file (i.e. .orders.db.tmp) before replacing the original file?>> If coded robustly, this change should virtually eliminate the>> possibility of truncating db files.> This is currently being considered for the upcoming patch release. What I> found interesting is that there is already code in the engine to do this,> but it was disabled by the original authors, I do not know why.> > The UNIX builds can't benefit from the level of exception handling available> on the MAC and PC platforms which is why when one thread crashes the whole> WebDNA process quits. We have worked very hard to improve the level of> robustness of the UNIX builds, and as a result, the 4.5 builds are a vast> improvement over the 4.0 build. But even with the improvements, I think it> is certainly time to implement the above solution to help maintain db> integrity. In the mean time, the best thing to do, is exactly what you are> doing, using the debug and error logs to help track down which template is> causing the engine thread to quit.> > >> -----Original Message----->> From: WebCatalog Talk>> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf>> Of Dale LaFountain>> Sent: Thursday, September 19, 2002 6:13 PM>> To: WebCatalog Talk>> Subject: SMSI: databases corrupted on crash and permission>> issues on OSX>> >> >> We just upgraded the back end of our store (www.tfaw.com) from a>> G4-867/OS9/Webstar/Webcat3.07 to>> Dual-1Ghz-DDR/OSX10.2/apache/Webcat4.5. Most things are working well,>> but we're having a database problem and lost of problems with file>> permissions.>> >> Our sales database has been truncated on two separate occasions>> today. Problems with this 40+ MB database coincide exactly with>> WebDNAMonitor entries in system.log noting that webcat has exited and>> been restarted:>> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA receives>> signal SIGBUSVal>> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA with pid>> [4338] is alive>> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA with pid>> [4338] is dead>> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [25411]>> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA receives>> signal SIGTERMal>> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA with pid>> [25411] is alive>> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA with pid>> [25411] is dead>> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [10584]>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: Thread receives>> terminate signal>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA with pid>> [10584] is dead>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [10585]>> >> I'm still running my webcatwatch.sh every minute from root's crontab,>> because I have found some situations where WebDNAMonitor is not able>> to recover gracefully from a crash (~1 time in 10). One of our other>> OSX servers still crashes 4-6 times per day, but I'm hoping to>> partially fix that with a hardware upgrade.>> >> We're now watching (debug file enabled - could the filename be any>> more cumbersome??) to see if a specific template is causing the crash>> in 4.x when it ran fine for 2 years in 3.x. We're also going to>> archive some data from this db to speed access and saving of the file.>> >> In the meantime I have to ask: why doesn't webcat write databases to>> a temp file (i.e. .orders.db.tmp) before replacing the original file?>> If coded robustly, this change should virtually eliminate the>> possibility of truncating db files.>> >> I realize this situation has come up multiple times in the past on>> this list, and manual backups of crucial databases hourly or more was>> the only real solution. This is fine except for the size and number>> of databases we use and the frequency of changes (lots of changes).>> Yes, I could write a shell script that saves incremental copies of>> each database and notifies me if the database is suddenly truncated.>> However, I believe that maintaining database integrity should be the>> responsibility of the application to the fullest extent possible, and>> not left as an exercise for each developer to implement their own>> safety net. Life would be much simpler for everyone if Webcat didn't>> mangle our databases in the first place, or even better, if it didn't>> crash in the middle of saving a db...>> >> We have heard from several developers on this topic in the past. I>> would really appreciate a response from SMSI this time around.>> >> >> Permissions: Whenever webcat moves an orderfile, it changes the>> ownership of that file from the shared developer owner with mode 664>> to owner www and the 644 permissions (no group write). User-created>> shoppingcarts are www/www and mode 664, which is fine until webcat>> authorizes them, after which they are www/www and mode 644. Our>> developers often need to manipulate these orderfiles manually for>> various reasons. The developers are in the www group, and the parent>> directories (Orders, Completed, etc) all have 770 permission.>> >> A couple solutions come to mind:>> >> - set up a crontab to chmod all files in these directories every>> minute (did this already, but it's a sledgehammer solution)>> - swap out all the [movefile] contexts with [shell]mv oldpath>> newpath[/shell] contexts to preserve the file settings>> - make developers login as www (same user as webcat process), but I'm>> not too hot about the idea of giving user www a valid shell for>> security reasons.>> - report this as a potential bug to SMSI and hope they fix it in a>> future release>> >> >> On a semi-related note: OSX Jaguar client -> Jaguar client file>> permissions are an equal pain in the a$$. The default permission set>> through personal file sharing are 600, which of course prevents>> apache from serving the files unless you force all developers to>> login as www (which, again, is not desirable). I haven't tried 10.2>> server yet, but that's an expensive solution considering the fraction>> of 10 Server features we would be using. I'm going to try NFS>> instead of AFP since all our developers are running OSX.>> >> Any thoughts on the above?>> >> -Dale>> >> ------------------------------------------------------------->> This message is sent to you because you are subscribed to>> the mailing list .>> To unsubscribe, E-mail to: >> To switch to the DIGEST mode, E-mail to>> >> Web Archive of this list is at: http://search.smithmicro.com/> > > -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to> > Web Archive of this list is at: http://search.smithmicro.com/> -------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Associated Messages, from the most recent to the oldest:

Re: SMSI: databases corrupted on crash and permission issues on (Alain Russell 2002)
Re: SMSI: databases corrupted on crash and permission issues on (Aaron Lynch 2002)
SMSI: databases corrupted on crash and permission issues on OSX (Dale LaFountain 2002)

Scott, just please test this extensively with big DBs.We have a 300 meg database, and doing certain deletes takes *forever*On 9/19/02 7:49 PM, Scott Anderson mashed the following keys :>> In the meantime I have to ask: why doesn't webcat write databases to>> a temp file (i.e. .orders.db.tmp) before replacing the original file?>> If coded robustly, this change should virtually eliminate the>> possibility of truncating db files.> This is currently being considered for the upcoming patch release. What I> found interesting is that there is already code in the engine to do this,> but it was disabled by the original authors, I do not know why.> > The UNIX builds can't benefit from the level of exception handling available> on the MAC and PC platforms which is why when one thread crashes the whole> WebDNA process quits. We have worked very hard to improve the level of> robustness of the UNIX builds, and as a result, the 4.5 builds are a vast> improvement over the 4.0 build. But even with the improvements, I think it> is certainly time to implement the above solution to help maintain db> integrity. In the mean time, the best thing to do, is exactly what you are> doing, using the debug and error logs to help track down which template is> causing the engine thread to quit.> > >> -----Original Message----->> From: WebCatalog Talk>> [mailto:WebDNA-Talk@talk.smithmicro.com]On Behalf>> Of Dale LaFountain>> Sent: Thursday, September 19, 2002 6:13 PM>> To: WebCatalog Talk>> Subject: SMSI: databases corrupted on crash and permission>> issues on OSX>> >> >> We just upgraded the back end of our store (www.tfaw.com) from a>> G4-867/OS9/Webstar/Webcat3.07 to>> Dual-1Ghz-DDR/OSX10.2/apache/Webcat4.5. Most things are working well,>> but we're having a database problem and lost of problems with file>> permissions.>> >> Our sales database has been truncated on two separate occasions>> today. Problems with this 40+ MB database coincide exactly with>> WebDNAMonitor entries in system.log noting that webcat has exited and>> been restarted:>> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA receives>> signal SIGBUSVal>> Sep 19 10:12:32 tfaw WebDNAMonitor[574]: WebDNA with pid>> [4338] is alive>> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA with pid>> [4338] is dead>> Sep 19 10:12:33 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [25411]>> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA receives>> signal SIGTERMal>> Sep 19 15:31:02 tfaw WebDNAMonitor[574]: WebDNA with pid>> [25411] is alive>> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA with pid>> [25411] is dead>> Sep 19 15:31:03 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [10584]>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: Thread receives>> terminate signal>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA with pid>> [10584] is dead>> Sep 19 15:31:04 tfaw WebDNAMonitor[574]: WebDNA restarts with>> pid [10585]>> >> I'm still running my webcatwatch.sh every minute from root's crontab,>> because I have found some situations where WebDNAMonitor is not able>> to recover gracefully from a crash (~1 time in 10). One of our other>> OSX servers still crashes 4-6 times per day, but I'm hoping to>> partially fix that with a hardware upgrade.>> >> We're now watching (debug file enabled - could the filename be any>> more cumbersome??) to see if a specific template is causing the crash>> in 4.x when it ran fine for 2 years in 3.x. We're also going to>> archive some data from this db to speed access and saving of the file.>> >> In the meantime I have to ask: why doesn't webcat write databases to>> a temp file (i.e. .orders.db.tmp) before replacing the original file?>> If coded robustly, this change should virtually eliminate the>> possibility of truncating db files.>> >> I realize this situation has come up multiple times in the past on>> this list, and manual backups of crucial databases hourly or more was>> the only real solution. This is fine except for the size and number>> of databases we use and the frequency of changes (lots of changes).>> Yes, I could write a shell script that saves incremental copies of>> each database and notifies me if the database is suddenly truncated.>> However, I believe that maintaining database integrity should be the>> responsibility of the application to the fullest extent possible, and>> not left as an exercise for each developer to implement their own>> safety net. Life would be much simpler for everyone if Webcat didn't>> mangle our databases in the first place, or even better, if it didn't>> crash in the middle of saving a db...>> >> We have heard from several developers on this topic in the past. I>> would really appreciate a response from SMSI this time around.>> >> >> Permissions: Whenever webcat moves an orderfile, it changes the>> ownership of that file from the shared developer owner with mode 664>> to owner www and the 644 permissions (no group write). User-created>> shoppingcarts are www/www and mode 664, which is fine until webcat>> authorizes them, after which they are www/www and mode 644. Our>> developers often need to manipulate these orderfiles manually for>> various reasons. The developers are in the www group, and the parent>> directories (Orders, Completed, etc) all have 770 permission.>> >> A couple solutions come to mind:>> >> - set up a crontab to chmod all files in these directories every>> minute (did this already, but it's a sledgehammer solution)>> - swap out all the [movefile] contexts with [shell]mv oldpath>> newpath[/shell] contexts to preserve the file settings>> - make developers login as www (same user as webcat process), but I'm>> not too hot about the idea of giving user www a valid shell for>> security reasons.>> - report this as a potential bug to SMSI and hope they fix it in a>> future release>> >> >> On a semi-related note: OSX Jaguar client -> Jaguar client file>> permissions are an equal pain in the a$$. The default permission set>> through personal file sharing are 600, which of course prevents>> apache from serving the files unless you force all developers to>> login as www (which, again, is not desirable). I haven't tried 10.2>> server yet, but that's an expensive solution considering the fraction>> of 10 Server features we would be using. I'm going to try NFS>> instead of AFP since all our developers are running OSX.>> >> Any thoughts on the above?>> >> -Dale>> >> ------------------------------------------------------------->> This message is sent to you because you are subscribed to>> the mailing list .>> To unsubscribe, E-mail to: >> To switch to the DIGEST mode, E-mail to>> >> Web Archive of this list is at: http://search.smithmicro.com/> > > -------------------------------------------------------------> This message is sent to you because you are subscribed to> the mailing list .> To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to> > Web Archive of this list is at: http://search.smithmicro.com/> -------------------------------------------------------------This message is sent to you because you are subscribed to the mailing list .To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to Web Archive of this list is at: http://search.smithmicro.com/ Aaron Lynch

DOWNLOAD WEBDNA NOW!

Re: SMSI: databases corrupted on crash and permission issues on

2002

Top Articles:

Related Readings: