NTL proxies alter download file size if it's too large

Legorol · 13-06-2006, 17:09

This is an issue that came up for me as I was trying to download the just-released public beta of Microsoft Windows Vista 64-bit version. It appears that NTL's proxy servers mess up the Content-Length field of an HTTP response if it is too large to be represented in 32-bits. In layman's terms: if you are trying to download a file that is over about 4 gigabytes, the downloaded file's size (and therefore the download itself) gets messed up. This affects you even if you use a download manager.

Here follows a description of the issue and how I found out about it.

Symptoms: when I attempted to download the single gigantic ISO image from MS's official download site, then rather than receiving a file that is 4 Gb in size, I ended up with 13.7 Mb. No matter what browser or download manager I tried.

I have done some investigating by capturing network traffic to and from MS's download server, both when going through NTL's proxy as well as a 3rd party proxy server. The result was that when going through the 3rd party proxy, the Content-Length field returned from the web server is 4309368832 bytes (0x100DBC000 in hexadecimal), which is a number that is a little bit over what can be represented in 32 bits. When going through NTL's proxy, the Content-Length field returned is 14401536 bytes (0x00DBC000 in hexadecimal, missing the leading 1). Hence my download ends up at 13.7 Mb.

In other words, it appears that at the moment NTL's proxies are unable to handle a file download that is larger than 4 gigabytes. I have alerted NTL to the problem and i was promised that their server support team will look into this issue.

The real problem with this is NTL's use of transparent proxies. Even if you are using a download manager program, as long as you are downloading something from a web server via HTTP, you are affected by this issue. If you dont' have access to a 3rd party proxy server that allows you to use large amounts of bandwidth, you are in trouble.

crazybones · 13-06-2006, 18:00

Excellent diagnostic work Legorol. That would explain why so many of us are seeing the 64bit download reported as 13.7Mb.

Could the proxy issue also explain why several people including myself have a problem where the download of the 32bit version of Vista stops at 2048MB? Each time I attempt a download with either IE or Firefox, it terminates at the same point.

Graham M · 13-06-2006, 18:01

Wow, thankyou very much for this explanation!

RazorD · 13-06-2006, 18:27

I think you'll find this issue is also related the the Content-Type.

I downloaded Vista onto my server the night it was released (so i could get a stable version quickly, before the servers became hammered, and then downloaded from my server, which could support just me fine!)

I encountered this error when the Content-Type was application/iso, (i couldn't actually find the offical content-type for an .iso, so i guessed) - Anyway, when this was the content type, the 64 bit version was shown as 13.7mb. But, doing some research of linux file mirrors, they used text/plain as the content-type for .iso's.

Having changed this on the server, it then reported the correct filesize.

Legorol · 13-06-2006, 18:35

I am not sure this is related to the Content-Type. When I tried from the official MS download server, I got a Content-Type of application/octet-stream. This was the case both via NTL proxy, via 3rd party proxy and by trying from an entirely different (non-NTL) connection.

Is it possible that your server, where you stored the file, mishandles Content-Length depending on the Content-Type? It could be suffering from a problem similar to the one that the NTL proxies are. You made an interesting point nevertheless.

crazybones: I'm afraid it seems to be an unrelated problem. I had no problem downloading the 32-bit version, which is a little over 3 gigabytes. I have used Free Download Manager, and I was going (transparently) via NTL's proxies. I suggest you get the direct download link and try different download managers, and not from your browser.

PS: Please note that I didn't try and make this into a thread about Vista, I intended this thread to be a discussion about the problem in general.

RazorD · 13-06-2006, 18:45

I run IIS, exactly the same as Microsoft's (Well, Akamaiaiaiaimaimaia, or however you spell it) servers.

I'll do some playing and research - I wonder if the filename length has anything to do with this too.. the filename is pretty long, maybe that triggers it, if its over 4gb, maybe a buffer overflow in NTL's proxies?

Edit: You're definately right - it is the content length which handles it (The reason it appeared to change is because in order to then get the right file, i changed the content type which it then tried to resume, requesting on from where i had got too (about 2mb) -> EOF - which meant it was under the 32bit filesize limit, and would then work. If you have a download manager, you can always try downloading the first 2mb or so, pausing, then resuming - see if it then wakes up and gets the right filesize

-Liam

slowcoach · 13-06-2006, 19:45

I downloaded the 64bit version using Linux, it was reported as 4.1GB and came down fast and perfect. I saw a lot of people were having a problem with the wrong filesize reported and the download stopping.
I have downloaded various .ISO files larger than 4GB in the past and only once had an incorrect filesize reported and that was due to a corrupt file on the server.
I tend to use HTTP servers rather than FTP servers so something doesn't add up, unless it is a Windows issue

I use 64 bit Linux, bet that's the reason I haven't had the same issue.

nffc · 13-06-2006, 23:01

So, does anyone know any 3rd party proxies that i can point Getright to use, to get the X64 beta?

smiker · 14-06-2006, 08:55

Quote:

Originally Posted by nffc

So, does anyone know any 3rd party proxies that i can point Getright to use, to get the X64 beta?

The NTL proxy at cache1-lang.server.ntli.net port:8080 is an inkotomi proxy and has allowed me to get the right file size (at last!)

Still a problem though, it now dumps out after downloading 13.7MB!

crazybones · 16-06-2006, 12:50

Quote:

Originally Posted by smiker

The NTL proxy at cache1-lang.server.ntli.net port:8080 is an inkotomi proxy and has allowed me to get the right file size (at last!)

Still a problem though, it now dumps out after downloading 13.7MB!

Hi smiker,

Follow nffc's advice in this post. I inserted the proxy and port details in "Free Download Manager" and it reported the correct sizes and downloaded the full files for both the 32 bit & 64 bit versions.

Monty123 · 26-06-2006, 22:44

Hello,
Is there a list of proxies which I could try out to download the 64-bit copy of Vista, as it would appear the 69.19.14.10:3128 proxy no longer works. I've tried googling for a couple but the ones i've tried display the correct file size of 4GB but either drop off at 13.7MB still or just die after downloading a couple of mbs, or do nothing at all

.

Mr J · 27-06-2006, 15:28

And here's me thinking I was going mad!! (probably still am thought)

Props to Legorol for this one, I was wondering why the hell I couldn't get the 64bit ISO down after trying normal download and various downlad managers.

Looks like I'm gonna have to get something from the newsgroups again.

Chris W · 27-06-2006, 15:36

Quote:

Originally Posted by slowcoach

I downloaded the 64bit version using Linux, it was reported as 4.1GB and came down fast and perfect. I saw a lot of people were having a problem with the wrong filesize reported and the download stopping.
I have downloaded various .ISO files larger than 4GB in the past and only once had an incorrect filesize reported and that was due to a corrupt file on the server.
I tend to use HTTP servers rather than FTP servers so something doesn't add up, unless it is a Windows issue

I use 64 bit Linux, bet that's the reason I haven't had the same issue.

when you made that post you were not behind a proxy