Still almost certainly not the network card, cable, or connection. If it were any of the above we'd see reduced speeds and/or packet loss, evidence suggests there is none of either.
Only connection setup time is affected.
You could try Netalyzr:
http://netalyzr.icsi.berkeley.edu/
and either post the results or a link to them (stripping out your IP address), this test shows both connection setup time and DNS lookup time in addition to pure speed.