its might be interesting for you to also use Mplayer in windows to compare, its not right upto the minute date, but on my travels i came across this one(untryed by me so cant say good/bad) .
http://oss.netfarm.it/mplayer-win32.php
"
...
Please don't use aggressive download managers (wget is fine) there is no single connection bandwith limit.
Mplayer on win32 using mingw32 - Last Build on
Sep 15, 2007
CoreAVC support is not so stable, I suspect it doesn't uninit correctly, if you use the GUI, you will easy get a crash.
Please give me some feedbacks about crashes while encoding x264 and xvid on p4/core2 cpu.
...
"
---------- Post added at 15:39 ---------- Previous post was at 14:44 ----------
unfortunataly many of the lads and lasses on doom9 seem obsessed with the mkv container.
but you can demux/re-mux to another container without re-encoding anything,with some good tools, so its also fast,and no loss of original quality.
i also suspect the mkv's your struggling with on that machine, have lots of extra stuff inside them (more than one audio track,subs etc), so a change of container with just the one video and audio track might make a lot of difference overall to lower end systems.
personally i like the TS (transport stream) container (the one all the broadcasters use for DVB* ,DVB-C being what your VM stb takes splits the channel your watching and stuffs it on your TV)as its faster to split etc.
and on the mailing lists someone just asking for info as hes working "to add support to mplayer and ffmpeg for LATM
AAC as carried in DVB"
so thats another good thing, AAC being the matching audio codec for the AVC Video codec.