Knocked up another quick diagram to show how it would be possible to implement a caching proxy with the 3 router solution. Other than the price of the hardware and some learning/configuration time, a linux OS and Squid are going to cost nothing. You can configure squid to run as a transparent proxy relatively easily so there's no manual configuration of browser settings required on the client side.
Proxy has 3x NICs, for example:
eth0 is WAN side and connects to the adsl modem
eth1 connects to the WAN interface of the office router
eth2 connects to the WAN interface of the cafe router
So even the WAN interfaces on your office and cafe routers are physically segmented but you can still control other traffic inbound/outbound such as SMTP and POP3 using the iptables access control rules on the proxy box.