This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Subject: Re: Wish List To: mozilla-webtools-request@mozilla.org From: Terry Weissman <weissman@tellme.com> Date: Mon, 11 Sep 2000 11:16:53 -0700 Matthew Barnson wrote: > I remembered the thing I forgot last time: Queries that are "too big for > Bugzilla's little brain" -- I would absolutely LOVE it if we could work out > the logic for "next page" links according to customizable parameters. This > could help solve the CPU issue Terry talked about with the "google-type" > query page -- if you insisted it return no more than 50 results, and set up a > hidden field on the return page for them to get the next 50, you could keep > the usage down. Once it found 50 matches, it could print and exit, then > resume where it left off for the next 50. Time to clear up some misconceptions: (1) The reason things are "too big for Bugzilla's little brain" here is simply that Bugzilla stores the results of the query in a cookie, and there are limits as to how big a cookie can be. This is fixable in a variety of ways; the simplest that would maintain the exact same functionality in all other ways (which may or may not be desirable) would be to put this list of bugs into a server-side SQL table, and store in the cookie only a pointer to that table. Oh, and do some magic to occasionally expire old things out of the table. (2) It is an illusion of SQL that restricting things to only return 50 results will significantly speed things up. If you do a big query, and you expect the results of your query to be sorted in any way, then SQL has no choice but to do the entire query, sort the results, and only then can it slice off the first 50 and return them. It's the "do the entire query" part that is slow and expensive; communicating the results back is relatively cheap. (The only exception to this is if you don't care about the order of things returned, or if the order is very intimately related to the query itself. These things are generally not true.) === Subject: Re: Wish List From: Brad Roberts <braddr@puremagic.com> Date: Mon, 11 Sep 2000 11:44:24 -0700 (PDT) On Mon, 11 Sep 2000, Terry Weissman wrote: > > (2) It is an illusion of SQL that restricting things to only return 50 > results will significantly speed things up. If you do a big query, and > you expect the results of your query to be sorted in any way, then SQL > has no choice but to do the entire query, sort the results, and only > then can it slice off the first 50 and return them. It's the "do the > entire query" part that is slow and expensive; communicating the results > back is relatively cheap. (The only exception to this is if you don't > care about the order of things returned, or if the order is very > intimately related to the query itself. These things are generally not > true.) Unless you cache the results of the first query in a table presorted for subsequent queries. Later queries will then actually be able to abort early since you can drop the order by clause. It does add a signifigant level of complexity to the system though. === Subject: Re: Wish List From: Terry Weissman <weissman@tellme.com> Date: Mon, 11 Sep 2000 12:54:00 -0700 Brad Roberts wrote: > Unless you cache the results of the first query in a table presorted for > subsequent queries. Later queries will then actually be able to abort > early since you can drop the order by clause. It does add a signifigant > level of complexity to the system though. But this assumes that your cache will remain valid, which it won't be the moment anyone tweaks any bug, which happens *all the time* on big installations on bugzilla.mozilla.org. Coping with that raises the complexity so high that it's not worth it. Caching also assumes that lots of people will be querying for the same thing, which is also probably not true often enough to make it worthwhile. === Subject: RE: Wish List From: jwarnica@ns.sympatico.ca (Jeff Warnica) Date: Mon, 11 Sep 2000 17:38:43 -0700 In theory, whatever SQL backend you are using should do this caching, should it not? === Subject: Re: Wish List From: Brad Roberts <braddr@puremagic.com> Date: Mon, 11 Sep 2000 13:41:29 -0700 (PDT) Definitly there will be issues with cache not matching reality. But bugzilla already faces that with the shadowdb to some degree. I wasn't suggesting that it was necessairly a good idea, just that there are ways to make page breakup's work in the sql world even in the face of orderby's. ===