Re: [WebDNA] HTTP Streaming - POSSIBLE!
This WebDNA talk-list message is from 2010
It keeps the original formatting.
numero = 105582
interpreted = N
texte = Although your suggestion will eliminate the counting of files in the bids folder, it still requires a separate disk access by each of thousands of connected bidders every second. It's these disk accesses that I feel are going to overload the server. Sincerely, Kenneth Grome > How about upon the creation of a new sequence file it deletes the old. > Then it knows what file to wait for because you base the file name of > numfound+1 search of auction bids. > > So, client arrives, gets the number of current bids and then it knows > what file to wait on. > > Shooting in the dark, just another idea... > > Scott > > On Jul 13, 2010, at 8:16 AM, Kenneth Grome wrote: > > >> Let's say a folder is created for each auction item. > > > > There is only one auction item so this part is easy ... :) > > > > > >> Then every time a new bid comes in it writes a > >> new sequential file. > > > > We will have anywhere from 1000 to 5000 bidders submitting bids > > during the one-hour auction. My estimate of the average number of > > bids received per second is 55. This means 55 new files written per > > second "on average" ... > > > > But this is only an average. During the early part of the auction > > only one or two bids may come in each second, or perhaps only one or > > two bids each minute. It's when we're nearing the end of the > > auction that I expect to receive thousands of bids per second. > > > > > >> Now when a new bidder views a potential item it > >> would search the number of files in that auctions > >> directory and would keep the connection open with a > >> waitforfile command until a new bid comes in. > >> Then when the file is created it simply refreshes the > >> page and waits for the next bid sequential file. > > > > My concern here is that there will be tens of thousands of "bid > > files" created during the one-hour auction, and maybe webdna won't > > be able to count them fast enough to respond quickly, especially > > near the end of the auction when thousands of bidders are online and > > submitting new bids rapidly. > > > > > >> This solves your server load issues because it doesn't > >> demand packets to constantly be exchanged. > > > > Yes, it solves the network part of the server load issue. But it > > burdens webdna with a task that invokes the OS to count files on > > disk -- and write new files in rapid succession -- instead of > > keeping all this data in a ram-cached webdna database. Because of > > all the disk access inherent in counting files on disk and writing > > new files all the time, I'm not convinced that webdna will keep up. > > > > Webdna is fast when writing to its own ram-cached databases since > > this is all done in memory. If this data is not flushed to disk > > until *after* the auction there would be very few disk disk hits > > during the auction. > > > > But my gut tells me that writing every new bid to a separate file on > > disk, and then invoking the OS to count those files thousands of > > times per second (one count for each of the thousands of bidders who > > are online and waiting for the next file to be written) is just > > going to bog down the server and fail miserably. > > > > I could be wrong of course, but I have enough doubts about this > > type of system to avoid proceeding with it -- at least until someone > > comes up with an argument that convinces me otherwise. > > > > Sincerely, > > Kenneth Grome > > > > > > > > --------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list
. > > To unsubscribe, E-mail to: > > archives: http://mail.webdna.us/list/talk@webdna.us > > Bug Reporting: support@webdna.us > > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > Bug Reporting: support@webdna.us >
Associated Messages, from the most recent to the oldest:
Although your suggestion will eliminate the counting of files in the bids folder, it still requires a separate disk access by each of thousands of connected bidders every second. It's these disk accesses that I feel are going to overload the server. Sincerely, Kenneth Grome > How about upon the creation of a new sequence file it deletes the old. > Then it knows what file to wait for because you base the file name of > numfound+1 search of auction bids. > > So, client arrives, gets the number of current bids and then it knows > what file to wait on. > > Shooting in the dark, just another idea... > > Scott > > On Jul 13, 2010, at 8:16 AM, Kenneth Grome wrote: > > >> Let's say a folder is created for each auction item. > > > > There is only one auction item so this part is easy ... :) > > > > > >> Then every time a new bid comes in it writes a > >> new sequential file. > > > > We will have anywhere from 1000 to 5000 bidders submitting bids > > during the one-hour auction. My estimate of the average number of > > bids received per second is 55. This means 55 new files written per > > second "on average" ... > > > > But this is only an average. During the early part of the auction > > only one or two bids may come in each second, or perhaps only one or > > two bids each minute. It's when we're nearing the end of the > > auction that I expect to receive thousands of bids per second. > > > > > >> Now when a new bidder views a potential item it > >> would search the number of files in that auctions > >> directory and would keep the connection open with a > >> waitforfile command until a new bid comes in. > >> Then when the file is created it simply refreshes the > >> page and waits for the next bid sequential file. > > > > My concern here is that there will be tens of thousands of "bid > > files" created during the one-hour auction, and maybe webdna won't > > be able to count them fast enough to respond quickly, especially > > near the end of the auction when thousands of bidders are online and > > submitting new bids rapidly. > > > > > >> This solves your server load issues because it doesn't > >> demand packets to constantly be exchanged. > > > > Yes, it solves the network part of the server load issue. But it > > burdens webdna with a task that invokes the OS to count files on > > disk -- and write new files in rapid succession -- instead of > > keeping all this data in a ram-cached webdna database. Because of > > all the disk access inherent in counting files on disk and writing > > new files all the time, I'm not convinced that webdna will keep up. > > > > Webdna is fast when writing to its own ram-cached databases since > > this is all done in memory. If this data is not flushed to disk > > until *after* the auction there would be very few disk disk hits > > during the auction. > > > > But my gut tells me that writing every new bid to a separate file on > > disk, and then invoking the OS to count those files thousands of > > times per second (one count for each of the thousands of bidders who > > are online and waiting for the next file to be written) is just > > going to bog down the server and fail miserably. > > > > I could be wrong of course, but I have enough doubts about this > > type of system to avoid proceeding with it -- at least until someone > > comes up with an argument that convinces me otherwise. > > > > Sincerely, > > Kenneth Grome > > > > > > > > --------------------------------------------------------- > > This message is sent to you because you are subscribed to > > the mailing list . > > To unsubscribe, E-mail to: > > archives: http://mail.webdna.us/list/talk@webdna.us > > Bug Reporting: support@webdna.us > > > > --------------------------------------------------------- > This message is sent to you because you are subscribed to > the mailing list . > To unsubscribe, E-mail to: > archives: http://mail.webdna.us/list/talk@webdna.us > Bug Reporting: support@webdna.us >
Kenneth Grome
DOWNLOAD WEBDNA NOW!
Top Articles:
Talk List
The WebDNA community talk-list is the best place to get some help: several hundred extremely proficient programmers with an excellent knowledge of WebDNA and an excellent spirit will deliver all the tips and tricks you can imagine...
Related Readings:
Template transformed itself into a database? (1998)
Counter for SKUs viewed (2002)
Running 2 two WebCatalog.acgi's (1996)
Emailer problem (2000)
WCS Newbie question (1997)
more info on [setlineitems] (1997)
[OT] (waaaay OT) further off topic than before (2004)
Help name our technology! I found it (1997)
Web Developer Product Awards (1997)
Is this possible, WebCat2.0 and checkboxes (1997)
select multiple (1997)
Nested tags count question (1997)
OT: How to get more traffic to a site? (2005)
HideIf ip= OR ip= (1998)
LookUp (2001)
This list needs a digest: rant, rave... (1997)
Setting up shop (1997)
OT: WebDNA and LINUX (2004)
[WriteFile] problems (1997)
Separate SSL Server (1997)