This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
To: modperl@apache.org From: Milo Hyson <milo@cyberlifelabs.com> Subject: Session refresh philosophy Date: Mon, 18 Feb 2002 18:22:23 -0800 Like my previous question on object caching, this one is potentially a matter of style as well. When it comes to implementing expirations on session data, I've encountered two schools of thought on when is best to refresh the timestamp/expiration. In that the general idea of expiration is to discard information that hasn't been accessed in a while, some feel that updating the timestamp is best done during both loading and storing. After all, both are considered accessing the data. However, taking into account the general pattern of HTTP request processing, I feel that updating only during storage is best, especially when using a database for persistence. Suppose one has a SQL table for saving session data. When a request comes in, the session is loaded and its expiration is examined. Assuming the session is still valid, one could issue another statement to the database to refresh the session's expiration time. That's two database ops before the session is even used. If you count the one at the end for storing the session back in the database it's a total of three per request. My feeling is that if you're going to be writing the session back within (hopefully) a fraction of a second anyway, you might as well wait until then to refresh the time-out. The project I'm working on requires that I design a custom application platform for current and future projects. My proposed solution to the session management problem is as follows: 1) A fix-up handler is called to extract the session ID from a cookie. Assuming a valid ID was found, the session is loaded, de-serialized and checked for expiration. If all is well, the a reference to the session is stored in pnotes for use by the application. 1a) If for some reason no session was found (e.g. no cookie) a new one is created and a new cookie is stuffed in the outgoing headers. 2) During content-generation, the application obtains the session reference from pnotes and uses it as necessary. 3) A clean-up handler is called to re-serialize the session and stick it back in persistent storage (updating the expiration in the process). The handler of course does nothing if the application destroyed the session in step 2. I'm still fairly new to mod_perl and haven't fully taken apart all of the various application servers out there to see how they do it. I would still appreciate any feedback anyone may have on the above. ==== To: modperl@apache.org From: Rob Nagler <nagler@bivio.biz> Subject: Re: Session refresh philosophy Date: Mon, 18 Feb 2002 20:29:26 -0700 Milo Hyson writes: > 1) A fix-up handler is called to extract the session ID from a cookie. [snip] > 1a) If for some reason no session was found (e.g. no cookie) a new one is [snip] > 2) During content-generation, the application obtains the session reference [snip] > 3) A clean-up handler is called to re-serialize the session and stick it back I may be asking the wrong question: is there a need for sessions? This seems like a lot of work when, for most applications, sessions are unnecessary. === To: Rob Nagler <nagler@bivio.biz>, modperl@apache.org From: Milo Hyson <milo@cyberlifelabs.com> Subject: Re: Session refresh philosophy Date: Mon, 18 Feb 2002 23:30:23 -0800 On Monday 18 February 2002 07:29 pm, Rob Nagler wrote: > I may be asking the wrong question: is there a need for sessions? > This seems like a lot of work when, for most applications, sessions > are unnecessary. I don't see how they could be unnecessary for what we're doing. Then again, maybe I'm just approaching the problem incorrectly. If one is doing a shopping-cart-style application (whereby someone selects/configures multiple items before they're ultimately committed to a database) how else would you do it? There has to be some semi-persistent (i.e. inter-request) data where selections are stored before they're confirmed. === To: modperl@apache.org From: Rob Nagler <nagler@bivio.biz> Subject: Re: Session refresh philosophy Date: Tue, 19 Feb 2002 10:17:52 -0700 Milo Hyson writes: > shopping-cart-style application (whereby someone selects/configures multiple > items before they're ultimately committed to a database) how else would you > do it? There has to be some semi-persistent (i.e. inter-request) data where > selections are stored before they're confirmed. As I understand it, the session data is "state" which is committed to the database on each request (possibly). It would seem to me that instead of denomalizing the state into a separate session table, you should just store it in a normal table. If the data needs to be expired, then it can be time stamped when it is written. The point is that it's always simpler to use the existing tables directly rather than making a copy and storing it in the database somewhere else. This usually reduces the code by half or more, because you don't have to worry about making the copy in the first place. Simpler code is more reliable and usually runs faster. To me, sessions are negativist. My expectation is that users will end up clicking OK (making the purchase). If that is the case, you are much better off putting the data were belongs right from start. You may bind it to an ephemeral entity, such as a shopping cart, but when the order is complete the only thing you have to do is free the cart and replace it with an order. The items, amounts, and special considerations have already been stored. If most of your users are filling shopping baskets and walking away from them, it may be a problem with the software. Checkout http://www.useit.com for some ideas on how to improve the ratio. Often you can avoid any server side persistence by using hidden fields in the forms. We use this technique extensively, and we have encapsulated it so that it is easy to use. For example, you might have a sub form which asks the user to fill in an address. When the user clicks on the "fill in address" button, the server squirrels away the context of the current form in the hidden fields of the address form. When the user clicks OK on the address form, the fields are stuffed back into the original form including the new address. If you have a performance problem, solve it when you can measure it. Sessions can mitigate performance problems, but so can intelligent caching, which avoids statefulness in the client-server protocol. Rob P.S. For sample sessionless sites, visit http://www.bivio.com and http://petshop.bivio.biz (which runs on a 3 year old 300mhz box running Apache and Postgres). === To: "Rob Nagler" <nagler@bivio.biz>, <modperl@apache.org> From: "Perrin Harkins" <perrin@elem.com> Subject: Re: Session refresh philosophy Date: Tue, 19 Feb 2002 12:35:32 -0500 > As I understand it, the session data is "state" which is committed to > the database on each request (possibly). It would seem to me that > instead of denormalizing the state into a separate session table, you > should just store it in a normal table. The typical breakdown I use for this is to put simple state information that connects this browser to long-term data in the session, and everything else in normal database tables. So, I put the user's ID (if this session belongs to an identified user), a flag telling whether or not this user has given a secure login so far in this session, and not much else in the session. Actually, even this stuff could be put into a normalized "sessions" table rather than serialized to a blob with Storable. It just means more work if you ever change what's stored in the session. === To: "Milo Hyson" <milo@cyberlifelabs.com>, From: "Perrin Harkins" <perrin@elem.com> Subject: Re: Session refresh philosophy Date: Tue, 19 Feb 2002 12:39:33 -0500 > In that the general idea of expiration is to discard > information that hasn't been accessed in a while, some feel that updating the > timestamp is best done during both loading and storing. If you don't always store the session (Apache::Session doesn't store unless you modify data in the session) then you have to update the timestamp on load to be accurate. If you want to be frugal, you might make the store operation update the timestamp, and then install a cleanup handler that will update the timestamp only if the session was not stored. - Perrin === To: modperl@apache.org From: Rob Nagler <nagler@bivio.biz> Subject: Re: Session refresh philosophy Date: Tue, 19 Feb 2002 11:07:43 -0700 Perrin Harkins writes: > Actually, even this stuff could be put into a normalized "sessions" table > rather than serialized to a blob with Storable. It just means more work if > you ever change what's stored in the session. This is a tough question. If you store it in a blob, you can't query it with an ad hoc SQL query. If you store it in a table, you have to deal with data evolution. On the whole, I vote for tables over blobs. My reasoning is that you have to deal with data evolution anyway. We have had about 200 schema changes in the last two years, and very few of them have had anything to do with user/visitor state. === To: Milo Hyson <milo@cyberlifelabs.com> From: Ged Haywood <ged@www2.jubileegroup.co.uk> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 11:42:59 +0000 (GMT) Hi there, On Mon, 18 Feb 2002, Milo Hyson wrote: > maybe I'm just approaching the problem incorrectly. If one is doing a > shopping-cart-style application (whereby someone selects/configures multiple > items before they're ultimately committed to a database) how else would you > do it? There has to be some semi-persistent (i.e. inter-request) data where > selections are stored before they're confirmed. You can for example send a hidden <form> object back and forth between your Client and the app. === To: mod_perl Mailing List <modperl@apache.org> From: Drew Taylor <drew@drewtaylor.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 17:50:31 -0500 At 11:42 AM 2/19/2002 +0000, Ged Haywood wrote: >Hi there, > >On Mon, 18 Feb 2002, Milo Hyson wrote: > > > maybe I'm just approaching the problem incorrectly. If one is doing a > > shopping-cart-style application (whereby someone selects/configures > multiple > > items before they're ultimately committed to a database) how else would > you > > do it? There has to be some semi-persistent (i.e. inter-request) data > where > > selections are stored before they're confirmed. > >You can for example send a hidden <form> object back and forth between >your Client and the app. And that is what I am doing for a small project I'm working on now. In my case, I'm not sure about the capabilities of the remote server, and I know for sure that I don't have a database available, so session information is saved via hidden form fields. It's primitive, but was actually a bit of a challenge to make sure a (unused) hidden field and a visible form element don't appear in the same <form>. Not my first choice, but it definitely works. === To: "mod_perl Mailing List" <modperl@apache.org>, From: "Perrin Harkins" <perrin@elem.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 17:55:09 -0500 > And that is what I am doing for a small project I'm working on now. In my > case, I'm not sure about the capabilities of the remote server, and I know > for sure that I don't have a database available, so session information is > saved via hidden form fields. It's primitive, but was actually a bit of a > challenge to make sure a (unused) hidden field and a visible form element > don't appear in the same <form>. Not my first choice, but it definitely works. Incidentally, this is mostly the same thing as what Jeffrey Baker mentioned a few days ago about storing state entirely inside a cookie with a message digest. The only difference is that by sticking it in a form element you're attaching it to a specific page. === To: "Perrin Harkins" <perrin@elem.com>, From: Drew Taylor <drew@drewtaylor.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 18:09:36 -0500 At 05:55 PM 2/19/2002 -0500, Perrin Harkins wrote: >Incidentally, this is mostly the same thing as what Jeffrey Baker mentioned >a few days ago about storing state entirely inside a cookie with a message >digest. The only difference is that by sticking it in a form element you're >attaching it to a specific page. True. I was very intrigued by his approach, and might use something like that to increase the security of my app by verifying the hidden form field contents. I suppose I could follow his approach, but the amount of data I need to store could possibly overwhelm the 4KB cookie limit. In this case, simple was better - simple application, simple session. And I know I can count on every browser implementing forms. :-) === To: "Perrin Harkins" <perrin@elem.com>, From: Milo Hyson <milo@cyberlifelabs.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 15:09:25 -0800 On Tuesday 19 February 2002 02:55 pm, Perrin Harkins wrote: > Incidentally, this is mostly the same thing as what Jeffrey Baker mentioned > a few days ago about storing state entirely inside a cookie with a message > digest. The only difference is that by sticking it in a form element > you're attaching it to a specific page. That's not a bad idea. I guess if you're paranoid about snooping you could always encrypt the cookie. === To: "Drew Taylor" <drew@drewtaylor.com>, From: "David Harris" <dharris@drh.net> Subject: RE: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 22:08:23 -0500 Drew Taylor [mailto:drew@drewtaylor.com]: > And that is what I am doing for a small project I'm working on now. In my > case, I'm not sure about the capabilities of the remote server, and I know > for sure that I don't have a database available, so session information is > saved via hidden form fields. It's primitive, but was actually a bit of a > challenge to make sure a (unused) hidden field and a visible form element > don't appear in the same <form>. Not my first choice, but it definitely works. I built and use a module that encodes a session hash into a number of hidden fields with a security MD5 sum. The encoded information is serialized, gzipped, Base64 encoded, and then split into reasonable length hidden fields. It looks like this: <input type="hidden" name="_fc_part000" value="eJx9zVEKwyAQhOEbhSiBhL2MrI1GW3VhR8n1a6HPeR6++ZNZCQd15YYYFIZWWEuQ2G/W4 IKqqDul"> <input type="hidden" name="_fc_part001" value="cm5Ic7Ab3UneXNmL7ym3C33EuLykTmywE0IpLp9jHu/0l2ye5UZ+lM+v/gUagTUd"> <input type="hidden" name="_fc_security" value="e99478182b7c579ce65dddb676bbe52e"> This way, you don't have to worry about creating hidden form fields in your templates for every variable you need to encode. In your perl, simply call the session encode and decode methods. You are also assured that nobody messed with the data. You can easily "pass" arbitrarily complex session information from one page to another without using a database, and the session info is truly tied to the *page*. Use of the back button, therefore, doesn't break anything. I've attached some code. To use the code, you'll have to replace the module FreezeThawLite with Storable. Also, beware the \r\n newlines. (I pulled this out of CVS on my windows desktop.) ------=_NextPart_000_0020_01C1B992.4156B5D0 Content-Type: text/plain; name="FormContainer.pm.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="FormContainer.pm.txt" package Fusion::FormContainer; use strict; use Digest::MD5 (); use MIME::Base64 (); use FreezeThawLight (); use Compress::Zlib (); use Carp; # this respresents a securty hole if we open-source this module.. the = securtiy string # needs to be passed as confguration at that point somehow. sub _create_security_string { my $string =3D shift; my $secret =3D <<EOT; --begin-secret-- enter your own secret binary of base64 encoded data here enter your own secret binary of base64 encoded data here enter your own secret binary of base64 encoded data here enter your own secret binary of base64 encoded data here -end-secret-- EOT my $ctx =3D Digest::MD5->new; $ctx->add($string); $ctx->add($secret); return $ctx->hexdigest; } sub encode { my $info =3D shift; my $prefix =3D shift; my $string =3D = MIME::Base64::encode(Compress::Zlib::compress(FreezeThawLight::freeze($in= fo))); $string =3D~ s/\n$//; my @array; push @array, ("${prefix}_fc_security", = _create_security_string($string)); my $part_number =3D 0; foreach my $part ( split "\n", $string ) { my $part_number_string =3D sprintf("%.3d", $part_number); push @array, ("${prefix}_fc_part$part_number_string", $part); $part_number++; } if ( wantarray ) { return @array; } else { my $html; while ( @array ) { my $name =3D shift(@array); my $value =3D shift(@array); $html .=3D <<EOT; <input type=3D"hidden" name=3D"$name" value=3D"$value"> EOT } return $html; } } sub decode { my $apr =3D shift; my $prefix =3D shift; my $security =3D $apr->param("${prefix}_fc_security"); my @string_parts; my $part_number =3D 0; while ( 1 ) { my $part_number_string =3D sprintf("%.3d", $part_number); my $part =3D $apr->param("${prefix}_fc_part$part_number_string"); last if ( $part eq "" ); push @string_parts, $part; $part_number++; } my $string =3D join "\n", @string_parts; croak "tampered or malformed FormContainer: securty string and string = don't match" if ( _create_security_string($string) ne $security ); return = FreezeThawLight::thaw(Compress::Zlib::uncompress(MIME::Base64::decode($st= ring))); } 1; ------=_NextPart_000_0020_01C1B992.4156B5D0-- === To: "David Harris" <dharris@drh.net>, "Drew Taylor" From: "Perrin Harkins" <perrin@elem.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Tue, 19 Feb 2002 22:51:06 -0500 > I built and use a module that encodes a session hash into a number of hidden > fields with a security MD5 sum. Sounds a lot like CGI::SecureState. Have you ever looked at it? - Perrin === To: David Harris <dharris@drh.net> From: Hans Juergen von Lengerke <lengerkeh@sixt.de> Subject: Re: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 10:16:18 +0100 (CET) David Harris <dharris@drh.net> on Feb 19, 2002: > The encoded information is [...] split into reasonable length hidden > fields. Why not put everything in one field? Are there restrictions? Does it make a difference when using POST? === To: "Perrin Harkins" <perrin@elem.com>, "Drew Taylor" From: "David Harris" <dharris@drh.net> Subject: RE: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 09:50:11 -0500 Perrin Harkins [mailto:perrin@elem.com] wrote: > > I built and use a module that encodes a session hash into a > > number of hidden fields with a security MD5 sum. > > Sounds a lot like CGI::SecureState. Have you ever looked at it? I just installed and played with CGI::SecureState (using the example in the POD) and it is totally different than my module. When I used CGI::SecureState it gave the client a non-versioning (more on that later) key and stored the state information in the filesystem. My module doesn't need to store any information in a database or in the filesystem. The entire state is given to the client in hidden form fields, and is passed back to the server on the next request. In addition, CGI::SecureState does not tie the state information to the *page*. With my module (or any method that stores the *data* in a hidden form field, not just a non-versioning key), state information is tied to the page. Let me explain: Imagine a multi-step order process where the user works through pages A, B, C, and D, (which contain forms) then uses the back button to go back to page B, changes the form values, and submits the form. With CGI::SecureState, page C will receive the state information stored by page D (that was intended for use by page E, we presume), instead of the state originally stored by page B (that was intended for page C). This is because all the pages share the same key, and old "versions" of the state are overwritten by the new "versions" and no longer available. When the back button is hit, a newer version of state may be used where an older version was intended. With my module, page C always gets the state information stored for it by page B, since the state is stored in hidden form fields in page B. The browser is actually storing the state and will always submit that same state to page C. (I have mentioned that CGI::SecureState uses a non-versioning key a few times. A way to make CGI::SecureState tie the state information to the actual page would be to change the key whenever the state changed, thus creating a versioning key. The key could be a hash of the state itself. This potentially means that a huge number of versions of the state would have to be stored on disk. I think this method would only be helpful if the state is large and it's not acceptable to pass it back and forth between the client.) In addition, CGI::SecureState gets fuddled if the user opens a new window (something I do often) and then starts performing different operations in each window using the same state key! It has no way of knowing a new windows exists and generating a new key. If you just store a customer id, customer name, and other rarely changing information in the state, these concerns may not matter to you. If you break a long form or order process into multiple pages, gathering new information on each page and storing it in the state so that you can process the order at the end, then this is a likely problem. David === To: mod_perl Mailing List <modperl@apache.org> From: Rob Nagler <nagler@bivio.biz> Subject: Re: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 07:55:46 -0700 Hans Juergen von Lengerke writes: > Why not put everything in one field? Are there restrictions? Does it > make a difference when using POST? That's what we do. There doesn't appear to be a restriction with POST. For while, we were encoding entire forms in URLs, but the limits got to us for really large forms. Rob === To: "Hans Juergen von Lengerke" <lengerkeh@sixt.de> From: "David Harris" <dharris@drh.net> Subject: RE: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 10:05:49 -0500 Hans Juergen von Lengerke [mailto:lengerkeh@sixt.de] wrote: > David Harris <dharris@drh.net> on Feb 19, 2002: > > The encoded information is [...] split into reasonable length hidden > > fields. > > Why not put everything in one field? Are there restrictions? Does it > make a difference when using POST? The POST encoding dose not have a limit on data length. Heck, people use <textarea> tags with huge amounts of content all the time. However, I didn't feel comfortable assuming that the HTML parser used by the browser could easily parse a potentially 20kb attribute. Basically, I didn't want to make my production application a stress-test for my user's browsers. :-) It was easy to break the data up into multiple hidden fields, because Base64 encoding breaks the data into multiple lines by default. I simply encoded each line in one hidden field. I would *NOT* use my module with a GET form if you expect any size of data. I've seen the query string get truncated at some arbitrary size limit. === To: "David Harris" <dharris@drh.net>, "Drew Taylor" From: "Perrin Harkins" <perrin@elem.com> Subject: Re: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 10:19:54 -0500 > When I used CGI::SecureState it gave the client a non-versioning (more on > that later) key and stored the state information in the filesystem. Okay, I only looked at it briefly and thought it stored the data on the client. Your module is actually more like CGI::EncryptForm I think, but yours may make things a bit more transparent. Maybe you should polish it up for CPAN. I'm well aware of the page-state vs. browser-state problem. I was recently bitten by it again when some consultants built a web app for my company that puts the search results in a session keyed on a cookie. As soon as the user opens two windows, it's absolute mayhem. - Perrin === To: "David Harris" <dharris@drh.net> From: wsheldah@lexmark.com Subject: RE: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 11:21:04 -0500 I can see how your approach adds functionality by performing as expected if the user uses the Back button or opens the app. in more than one browser window. The usual objection I've heard to using form fields is the security risk of people changing hidden fields in ways unforseen before submitting the form back, or of other people finding confidential data hidden in form fields if the user walks away and leaves their browser open, or the web page info gets hijacked somehow. Does your module address this, or is this yet another tradeoff between security and functionality/convenience? Wes Sheldahl === To: mod_perl Mailing List <modperl@apache.org> From: dom@idealx.com Subject: Re: [OT-ish] Session refresh philosophy Date: Wed, 20 Feb 2002 17:48:28 +0100 > The usual objection I've heard to using form fields is the security > risk of people changing hidden fields in ways unforseen before submitting > the form back, or of other people finding confidential data hidden in form > fields if the user walks away and leaves their browser open, or the web > page info gets hijacked somehow. Does your module address this, or is this > yet another tradeoff between security and functionality/convenience? No, this just means that input must be validated once again when the last