Simple view on how proxies work (inc DNS)
 

Since it seems to confuse people a lot I thought I'd try and explain it in simple terms;

Techies - feel free to point out any mistakes. :)


First of all - without any proxies - when a client wants a web page ;

1. It does a dns lookup of the required domain using its own (locally) defined dns servers.
2. It sends the http request on tcp port 80 to the destination servers IP address.
3. This request is routed over the network direct to the specified server IP.
4. The requested page is then returned, by the server, direct to the client.


Now if you have a manually defined proxy (such as in IE, on port 8080) - when a client wants a web page ;

1. It does a dns lookup of the proxy servers IP using its own (locally) defined dns servers.
2. It sends the http request on tcp port 8080 to the proxy servers IP address.
3. This request is routed over the network direct to the specified proxy server IP.
4. The proxy server checks if it has a valid copy of the requested page in its cache.
5. If the proxy has the page it returns it directly back to the client, Job done.
6. If the proxy does not have the page it does a dns lookup of the requested domain using it's own defined dns servers.
7. The proxy sends the http request on tcp port 80 to the destination servers IP address.
8. This request is routed over the network direct to the destination servers IP.
9. The requested page is then returned, by the destination server, directly back to the proxy server.
10. The requested page is then returned, by the proxy server, direct to the client.


Now if you have no manually defined proxy, but NTL's proxies are active - when a client wants a web page ;

1. It does a dns lookup of the destination servers IP using its own (locally) defined dns servers.
2. It sends the http request on tcp port 80 to the destination servers IP address.
3. This request is intercepted by NTL's routers and re-directed to one of a group of transparent proxy [cache] servers - the decision on which proxy server gets used is (usually) based on the requested domain.
4. The chosen proxy server checks if it has a valid copy of the requested page in its cache.
5. If the proxy has the page it returns it directly back to the client, Job done.
6. If the proxy does not have the page it does a dns lookup of the requested domain using it's own defined dns servers.
7. The proxy sends the http request on tcp port 80 to the destination servers IP address.
8. This request is routed over the network direct to the destination servers IP.
9. The requested page is then returned, by the destination server, directly back to the proxy server.
10. The requested page is then returned, by the proxy server, direct to the client.

The final scenario is called transparent because the client is unaware that its request as been "hijacked" and still thinks it is talking directly to the destination server.

From the above it can be seen that if you are having web browsing problems and proxies are in use (transparent or otherwise) then ;

1. The DNS servers the proxy is using are just as important as yours - if they are duff then you are in trouble. (However, even if the proxy cannot resolve or contact the destination server - you can still access the requested pages if the proxy has a valid copy of them in its cache)

2. Doing a tracert to the destination server will not show you the route your request is taking as it won't go via any proxy in use.

This is not a precise and exact description of how it all works but I hope it makes it clearer. :D

To workout if you are using an NTL transparent proxy - use this simple link

Pem.

Re: Simple view on how proxies work (inc DNS)
 

Cheers for that. I think that makes sense :) I have a question though. What is a 'valid copy' of a page. I.e. How does the proxy decide if the page it has cached is recent enough? Do all pages carry a time stamp (last modified date/time), and does the proxy query the requested site for the time stamp of the requested page?

Re: Simple view on how proxies work (inc DNS)
 

All pages effectivley have an expiry date & time.

Re: Simple view on how proxies work (inc DNS)
 

Ah, that makes sense, as I was wondering if the proxy had to query for a more recent page, which would increase traffic and defeat the purpose of the proxy unless you only have to retrieve the time stamp of the most recent page.

I presume this expiry time/date is set differently depending on the type of page? (very low for a forum like this) and high(er) for less dynamic pages)?

Or is it just based on recent 'refresh rates'?

Re: Simple view on how proxies work (inc DNS)
 

pem even when a port is specified my pages are still going through the transperent proxies, the guy who told us this in the other thread was right.

Re: Simple view on how proxies work (inc DNS)
 

Believe me - you would not believe how complicated it is for a proxy to determine if it's copy is current ("fresh"). There are pages and pages of specifications and http headers to control it. :D


Sorry, I don't follow what you are saying. Can you explain ?

Re: Simple view on how proxies work (inc DNS)
 

Does CTRL+Refresh Overide the proxy in order to get an updated refresh??

Re: Simple view on how proxies work (inc DNS)
 

Erm ........ if client and proxy follow the rules - Yes.

Do they all follow the rules ...... do they **** :banghead:

Re: Simple view on how proxies work (inc DNS)
 

Ah, the joys of optimisation/efficiency :)

Re: Simple view on how proxies work (inc DNS)
 

just ditch them, the're not there for your convenience!

mind you, i like playing with them at work. but that's not for here!!!!!!!!!!

Re: Simple view on how proxies work (inc DNS)
 

How long to the proxies keep stored pages for?

Re: Simple view on how proxies work (inc DNS)
 

generally, until they change or expire. Some will make checks on a site for change on a daily basis to see if there have been changes that they should cache

Re: Simple view on how proxies work (inc DNS)
 

well right now pem I am running through my own proxy on port 3333, but the ntl transperent proxy is still intercepting my pages, I will look for the post that explains this and paste it here.

Re: Simple view on how proxies work (inc DNS)
 

http://forum.nthellworld.co.uk/showp...2&postcount=89

shows the info

but right after i retrieved this post the ip shown in my sig is no longer ntl proxy server.

I think it is working after all

Re: Simple view on how proxies work (inc DNS)
 

when I use IE or firefox it doesnt use ntl proxy but if I use MYIE2 it does, very weird especially as MYIE2 uses IE core.

--edit--

fixed it :)