Resource combined, minified and compressed each request?

Dec 12, 2009 at 6:58 AM

If you have a particular resource set - say 'siteJs', that is only ever in a fixed set of ways.

it is minified and then delivered by gzip or deflate. 

So, just wondering if when the request comes in you create a file that is already compressed/minified/etc and then return that result.  Meaning future inquires for that same resource never have to be redone again (unless config changes, files change, etc).

You would create sets like 'siteJs.1.gzip' and 'siteJs.1.deflate', etc as they were created the first time - then just deliver them.

This would potentially solve a problem I have Telerik's similar product, webforms based.  (I am moving to MVC, so trying to find a new solution.  Like yours!).  The problem is when you deploy version 1 of your scripts, googlebot and other such sites will have the cached version of your html and request resources off the old version.   This means when I deploy my version 2 of the scripts - it is possible (and ALWAYS happens) that requests come in for version 1!!!

If the files were generated, when new versions were released - the old ones would be there and could be served.

Also, looking at - it appears you do not create the file, but are considering something along these lines.


Dec 13, 2009 at 4:43 PM

@waynebrantley: is it right that you expect that if request for the old version (e.g. from googlebot) should always be served with the old version (of the set Combres already generated)?  If that's the case, then yes, a distributed cache, DB or file system cache provided would potentially solve that.  However, that is possible if auto-versioning is used, if not, it would be quite challenging (I need to think more).  Example of the latter:

  • A resource set is in version 2 (explicitly specified instead of 'auto' version)
  • First request, and the result of version 2 is cached (or stored in siteJs.2.gzip)
  • A filter is added for that resource set (or the minifier is changed, or a new resource is added) and developers for some reason chooses not to increase version number
  • The cache is invalidated because of change to data file (even if Combres doesn't invalidate the cache immediately, it is eventually invalidated after the specified duration)
  • Now, we have 2 different contents for the same version - and thus would have the same problem we described.  Of course, we can say that developer ought to change the version number here

I'm also curious to know whether this feature is important or not (to prioritize the implementation).  Is it a big deal if googlebot or s/t like that always have the latest version (although it requests the old one)?  I suppose some cached pages may break if used with latest CSS/JS, but because they're cached pages anyway, so not sure if it's a big deal for many people.  Let me know.

Dec 14, 2009 at 6:54 PM

I appreciate your replies/responses.

1) I am thinking the results should be cached in files always, so the work of your scripts are as minimal as possible.   We are running your scripts on every page load multiple times, instead of as static file get - so lets make it efficient.

2) You have to consider the CURRENT version and old versions.  Old versions are defined as not the current one.  :-)

Current:  Combres is in charge of managing the cache/stored files.  When it detects a change of whatever, you clear the cache (memory cache, created files, etc)

Old:  When a request comes in for an 'old version' - you simply look for the cached file and deliver it.  If it is not there, you could either give an error or deliver the current version of that resource.

Given the above, It does not seem the versioning would be challenging would it?

It is not a big deal if googlebot or whatever always have the latest version or anything - it is important that when they are refreshing the page/search stuff they actually get back the correct resource though.  If I currently have it labeled as version 3 and a request comes in for version 1 or 2 - what do you do?   An error is what is not acceptable in this scenario - really that is what I am trying to avoid.  That and people with cached content (proxy, browser, output caching, etc) not getting an error while the switch is being made.

AND, just trying to make your product better.  :-)


Dec 15, 2009 at 3:36 PM

1.  I'm not so sure why it has to be file cache?  Currently, Combres uses ASP.NET Cache, so unless there's change to the resource set's content, config file etc. then requests are served from the cache.  I suppose you want something more persistent than the ASP.NET Cache (whose live is at most per app restart)?  If that's the case, then I also have the plan to implement a pluggable cache mechanism, as you seems to already notice.

2.  Currently Combres will always return the latest version even when it receives request for old version, instead of throwing exception.  Even so, I guess you'd prefer to have the correct version served instead (or either exception/latest version is served if the old version does not exist anymore).  I think that makes sense.  On the other hand, I'm not quite sure if that would be the expected behavior.  After all, in the life without Combres, whenever people modify a CSS or JS, then even the cached HTML in Google/Yahoo/etc. would request the latest CSS & JS.  In fact, the answer to this question may help me understand your request better: is this a problem when you just use CSS & JS directly in your application without using Combres?  If not, how do you handle it?


Dec 15, 2009 at 9:50 PM

1.  It does not have to be a file cache, but it should be.   Most resources like this (stylesheets/scripts) are served off of files to start with.  If you are going to 'generate' one of these files - just makes sense it would be stored in a file and then served up....that is all.   IIS can then choose to cache it based on it's settings and the client will cache, etc.   At first glance, do not really want to store these things in my cache, potentially getting kicked out for expiring and or kicking other things out.  The plug-able thing would work, with a file implementation as a default.  People could use velocity cache or whatever then.

2.  Returning latest version when request for old one....that is good, unless old one is around, then that is better - glad to hear you do this though!  I like that.   Think of it like jQuery on CDNs.  A script like this is requested:  When jquery 1.4 comes out it will be So,when requests from older clients, cached clients, or any such thing request the library - they still get the old version.   This is easy for them to accomplish, because they just leave the file there - which is what I was thinking you could easily do.  This also answers your question above - we just version the resource.  

3)  Finally, on a related topic.  Can you explain your 'remote' script option to me.   You even give an example using the google cdn for jquery, much like I have done above.   That seems like a terrible idea.   The main purpose of a CDN is to distribute the content to 'edge' servers all over the world.  When someone requests a file from the CDN a geographically close edge server is used to return the content.   This makes for a faster experience for the user.  The secondary purpose of the CDN is for cross-site caching!   If I have the google ajax reference (or the M$ one above), then anyone that has been to ANY site which used that same CDN can just use their existing cached content!  No download.   Using 'remote' feature of your library eliminates both of those huge advantages for the one advantage of 'having all scripts in one file'.   That is not even that big of an advantage - as long as the number of scripts is kept low, because the browser can download from more than one location at a time.

Got your product downloaded, perused it yesterday and going to try it out today.   I really like what I see and am going to attempt to use it.   For #1 above, it is just an optimization feature I think we need - but not a deal killer.  For #2 above, returning the latest version is a great interim solution and for #3 above, I will just include my CDN hosted scripts on the page by themselves - not part of combres - so that is easy too.

Thanks again....hope I am helping.  :-)

Dec 16, 2009 at 6:00 AM

1.  Totally agree.

2.  Great example.  Agree it's nicer to have this feature implemented.   Tracking ticket is created here:

3.  Great point.  Agree with you completely, CDN isn't the best example to demonstrate the use of "remote" resource.  Remote resource can be used in other cases when you have to download scripts located in another web app or server (not CDN), or when the resources are not static but dynamically generated by your application (e.g. an HTTP handler that returns JavaScript),

>>Thanks again....hope I am helping.  :-)

You definitely are :-).  Thanks a lot!  Don't hesitate to add more suggestions as you use the product - I'll be happy to hear more on how to make Combres better.