Any change to your package tends to start a new minimum term, so removing or downgrading just Internet would result in a new minimum term and possibly disconnection fees (if you removed Internet altogether) . I'd call retentions and see whether they can do anything.
Early termination fees:
https://www.virginmedia.com/shop/the...onnection-fees