Simple view on how proxies work (inc DNS)

[mr-b]

Originally Posted by SMHarman

Http also has a HEAD command and a GET command. GET asks for the whole page, HEAD asks for the headers, which can be used to compare to the cached headers. Head uses a lot less bandwidth as a result.

Correct - but there's a way of using even less bandwidth.

If you issue an HTTP GET and also include an If-Modified-Since header then the web server will only send you the object if it has been modified since the date that you specified. If it hasn't, it replies with HTTP/1.1 304 Not Modified and then you can mark your content as being fresh again having only transferred a few bytes.

You can view this in a trace by seeing lots of 304 responses typically coming back after you hit browser Refresh.

But now we're getting away from the 'simple' aspect of this thread ...

[SMHarman]

Originally Posted by mr-b

Correct - but there's a way of using even less bandwidth.

If you issue an HTTP GET and also include an If-Modified-Since header then the web server will only send you the object if it has been modified since the date that you specified. If it hasn't, it replies with HTTP/1.1 304 Not Modified and then you can mark your content as being fresh again having only transferred a few bytes.

You can view this in a trace by seeing lots of 304 responses typically coming back after you hit browser Refresh.

But now we're getting away from the 'simple' aspect of this thread ...

There are some well formatted caches and browsers out there. Looking at my site stats about 25% of traffic is 304'd. That saves me a good bit in bandwidth costs.

[mr-b]

Originally Posted by SMHarman

There are some well formatted caches and browsers out there. Looking at my site stats about 25% of traffic is 304'd. That saves me a good bit in bandwidth costs.

Absolutely, and if you specify explicit expiry times e.g. by using a header such as:
Cache-Control: max-age=3600
on appropriate images etc. then it optimises your web performance even further since the browsers/caches will know exactly when to refresh the content and won't need to guess by sending periodic IMS requests.

For instance Google's logo.gif expires in Jan 2038! Of course if they want to change it then they just change the name.

[Chrysalis]

what would happen if the page was updated

and Cache-Control: max-age=3600 was after the update?

[mr-b]

Originally Posted by Chrysalis

what would happen if the page was updated

and Cache-Control: max-age=3600 was after the update?

A simplistic approach to this is to consider a web page as a collection of objects referenced by an html object. As the content publisher you apply appropriate expiry times to your objects e.g. images tend not to change very much, so you might expire them in a month. Your html might change more often so give html objects expiry times of 3600s.

If you apply inappropriate expiry times then you risk browsers and caches storing your content for longer than is correct. The usual rule of thumb is to keep the html objects small and with short expiry times (or mark it uncacheable) and mark the larger objects with longer expiry times. The larger objects are the ones that you don't want being downloaded all the time anyway, since it will both load your web server and give users the impression that your page is slow.

If you want to change the page and objects completely then change all the objects' names and then you can start afresh.
More details are at http://www.web-caching.com/mnot_tutorial/ (might have posted that link before)

[asdf]

Yup, I noticed proxies had been turned on in both Leicester and Nottingham in the past week or so. Previously I'd pretty much always been proxy free.

Seems okay so far.

[Chrysalis]

mr-b that would explain why I had so many problems on the development pages I used, the first time I downloaded the file probably cached it and because its a large file it is probably cached for a long time, but the file is updated at the server end numerous times a day, the cache made it pretty much unusable for me to use. This was before I setup my own proxy of course.

[mr-b]

Originally Posted by Chrysalis

mr-b that would explain why I had so many problems on the development pages I used, the first time I downloaded the file probably cached it and because its a large file it is probably cached for a long time, but the file is updated at the server end numerous times a day, the cache made it pretty much unusable for me to use. This was before I setup my own proxy of course.

Size of file has nothing to do with how long browsers or proxies cache the object for. They cache the object until they run out of allocated disk space and something else replaces it.

Just run your dev server's URL through the Cacheablity Engine http://www.ircache.net/cgi-bin/cacheability.py and it'll tell you exactly what's going on (Definitely given that link out before).

If the file is being updated frequently then its Last Modified header will be more recent which will tell your browser or proxy to verify its freshness more frequently.

[Chrysalis]

"The usual rule of thumb is to keep the html objects small and with short expiry times (or mark it uncacheable) and mark the larger objects with longer expiry times. The larger objects are the ones that you don't want being downloaded all the time anyway, since it will both load your web server and give users the impression that your page is slow. "

I thought by that you was reffering to size of file?

I ran a link through that url and it seems to confirm my proxy server is only proxying and doing no caching which is what I want

[theman]

i just ran ntlworld though that site and it said they are stale

[mr-b]

Originally Posted by theman

i just ran ntlworld though that site and it said they are stale

The analysis is not saying that the actual object is stale, more accurately that browsers and caches will treat the homepage html object as being stale and so always fetch a new copy every time it is accessed. ntl have done this by setting a cookie on the homepage which effectively makes it uncacheable.
The gifs have Last-Modified headers on and so will be cached and verified according to age.

[erol]

The problem with proxies is they work on a flase premise. The false premise they work on is that external bandwidth is expensive - and thus it is worth spending time money and resource on reducing external bandwidth requirments. It is very like the false premise that 'storage' was expensive - that led to people coding year date fields as 2 numbers and not 4 - well into the 80's and beyond, thus creating the y2k problems.

I would challenge anyone to prove to me that the cost of running and maintaining the NTL proxies is less than the amount saved on external bandwidth costs. If you add in a cost for the problems these proxies cause for users as well then the figures make even less sense. Of course you can also use such proxies for other objectives - like limiting what and how much people can browse but that is apparently not the objective for NTL.

PS there was an onlinbe petition setup a while back - but it's not had much support so far so maybe NTL's forced proxies are not that much of a problem to most users?
http://www.petitiononline.com/ntlhpp/petition.html

[SMHarman]

Erol - you also need to factor in the bandwidth costs that the supplier of the data pays. A site pays its host bandwidth cost. NTLs (and others proxies) reduce the bandwidth I use at my host by about 25%, meaning I stay within my prepaid bandwidth.

NTLs proxies generally work well for me so I don't complain. Ctrl F5 addresses any problems. Occasionally I need to specify as the transparant Luton/Ware has gone screwy.

I like to think of proxys like warehouses. Just because a pair of shoes is made in China this doesn't mean the entire stock stays there until its called for. They are shipped out to the other continents for quicker delivery.

[erol]

Originally Posted by SMHarman

Erol - you also need to factor in the bandwidth costs that the supplier of the data pays. A site pays its host bandwidth cost. NTLs (and others proxies) reduce the bandwidth I use at my host by about 25%, meaning I stay within my prepaid bandwidth.

This is true but I doubt NTL run the proxies to save your web hosters external bandwidth. For me if you are doing a cost evaluation from NTL's perspective of how much the proxies save vs how much they cost - then what is relevant is how mcuh they save for NTL (and not how much they save for NTL + third parties)

Originally Posted by SMHarman

NTLs proxies generally work well for me so I don't complain. Ctrl F5 addresses any problems. Occasionally I need to specify as the transparant Luton/Ware has gone screwy.

Well I am not an NTL user and never have been so my personal experience is zero. However having hung around this forum and other NTL related forums it seems to me that 'try changing your proxy' is one of the most common pieces of advise given and thus it would appear that NTL's forced proxies do create significant problems for significant numbers of users. For me it is all about choice. I do not think NTL should necessarily scapt its proxies but just that it should allow people to chooses whether they use them or not.

[SMHarman]

Originally Posted by erol

This is true but I doubt NTL run the proxies to save your web hosters external bandwidth. For me if you are doing a cost evaluation from NTL's perspective of how much the proxies save vs how much they cost - then what is relevant is how mcuh they save for NTL (and not how much they save for NTL + third parties).

True however this is indicative of the additional bandwidth of a small site, what would it be like for Amazon or other popular sites? Ultimately it could put up the cost of doing business on line as the bandwidth costs need to be recovered in the selling price.

This is also a reciprocal thing. NTL host sites. If all the proxies on the internet were switched off, then the bandwidth hitting NTLs web servers (their own ntlworld and all their customers) would increase and require more hardware and bandwidth. Bandwidth will become a more used resource, more scarse and more expensive.

Originally Posted by erol

Well I am not an NTL user and never have been so my personal experience is zero. However having hung around this forum and other NTL related forums it seems to me that 'try changing your proxy' is one of the most common pieces of advise given and thus it would appear that NTL's forced proxies do create significant problems for significant numbers of users. For me it is all about choice. I do not think NTL should necessarily scapt its proxies but just that it should allow people to chooses whether they use them or not.

Yes they screw up for me too. I can get mail, message, but not browse - its the proxys that are the problem in that instance.