I would say those speeds almost certainly point to a contention issue.
If you can live with this, stay with them, otherwise I'd recommend downgrading back to 10Mbps or so (it depends by area, but some areas use DOCSIS 1 for 10Mbps, others use DOCSIS 3, so if your area uses DOCSIS 1 for 10Mbps then you may find that the contention issue no longer affects you.
For reference, here is a speedtest I got just now on 50Mbps: