General : Virgin Akamai Cache Congestion

[Synthetic]

Hello all

I've been looking at a really annoying streaming video quality issue for the past few weeks which i've tracked down to seemingly being an issue with video served from akamai.

For example channel5, seems to hit akadash0.akamaized.net and a traceroute ends at a virgin media address of cur1-melt2-0-0-cust128.12-1.cable.virginm.net (213.106.24.129)

From my investigation it seems to be this cache is congested and causing the streaming quality to drop.

Was hoping someone else more in the know could tell me if i'm onto something here or way off, i don't fancy my chances raising this with CS

TIA!

[spiderplant]

I'll pick this up if you can provide some evidence, and a way to recreate it.

Can anyone else recreate this?

By the way, that reverse DNS lookup isn't correct, though that isn't the cause of your problem. I guess they have reused an old Melton Mowbray IP address but forgot to update the DNS. I can confirm 213.106.24.129 is actually a link to Akamai from VM in Manchester.

[Jaymoss]

Have you tried changing the route by changing the DNS server or trying a VPN

[Synthetic]

Yeah so if I use cloudflare warp, everything is good.

SP what else can I provide to help? I can reproduce it on my5 and itv hub, I'll try and find out if there's any other on demand that uses akamai and check if I can recreate it there too

[spiderplant]

What device are you running My5 and ITVHub on? Can you quantify the quality drop? Does it only happen at certain times of day? Also please name some specific assets that show the fault.

BBC iPlayer certainly used to use Akamai, but I don't know if they still do.

[Synthetic]

Originally Posted by spiderplant

What device are you running My5 and ITVHub on? Can you quantify the quality drop? Does it only happen at certain times of day? Also please name some specific assets that show the fault.

BBC iPlayer certainly used to use Akamai, but I don't know if they still do.

Noticed it on both my fire sticks originally, my partner has also mentioned she noticed sky go (i rarely watch anything on it) drops in quality quite often but no way to check anything on the PS4 app.

I've recreated it on my5 chrome on my laptop, using the dev console i can see the segments it's requesting end in "1500" (assume the max bitrate) and that changes down to 800,360,180 before the picture and audio become noticeably worse, the assets then go back to 1500 and the quality improves, then drops etc etc

It does seem to coincide with peak times, it seems to be staying at 1500 right now, i'll test again later this afternoon and try and get some logs / screenshots

[Synthetic]

Frustratingly CH5 don't appear to be sending me to akamai today, which is very strange, i've been hitting https://hwdash.channel5.com/cenc/xxx...PEGDASH_1500_a instead and everything has been fine.

ITV seem to be using https://itvpnpdotcom.cdn1.content.itv.com rather than akamai too

[General Maximus]

problem solved then

[Synthetic]

Originally Posted by General Maximus

problem solved then

For now it appears so

Spiderpant - just wondering if virgin have any monitoring on these caches, can they see utilisation on the network links to them / the boxes themselves?

I'd imagine they sync their content OOH overnight, but could they somehow end up doing this peak times and causing congestion?

Just trying to get an idea of how these things work heh

[spiderplant]

The caches are owned and managed by Akamai themselves. VM can only monitor the link capacity, though we'd contact Akamai if we thought the problem lay at their end.

[Synthetic]

Thanks SP

Interestingly, i'm no longer being routed to the VM cache, and everything is fine...

C:\Users\xx>tracert akadash0.akamaized.net

Tracing route to a1160.w10.akamai.net [88.221.87.131]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.88.1
2 12 ms 14 ms 8 ms 10.53.34.249
3 11 ms 12 ms 10 ms gate-core-2a-xe-501-0.network.virginmedia.net [82.2.240.105]
4 * * * Request timed out.
5 * * * Request timed out.
6 27 ms 24 ms 25 ms 86.85-254-62.static.virginmediabusiness.co.uk [62.254.85.86]
7 * * * Request timed out.
8 23 ms 23 ms 26 ms uk-lon03a-ri1-ae-2-0.aorta.net [84.116.135.46]
9 30 ms 24 ms 24 ms 213.46.174.130
10 24 ms 29 ms 26 ms ae8-xcr1.lnt.cw.net [195.2.24.130]
11 30 ms 29 ms 29 ms ae4-ucr1.mcr.cw.net [195.2.27.166]
12 30 ms 30 ms 30 ms 217.161.88.254
13 29 ms 29 ms 27 ms a88-221-87-131.deploy.static.akamaitechnologies.com [88.221.87.131]

Trace complete.

Not really sure what is going on here but pinning down is becoming a nightmare, the trend does seem to be the issue only appears when using the VM cache in particular, very strange.

[spiderplant]

Maybe somebody already identified the fault and has taken the problem link out of use.

[Synthetic]

Originally Posted by spiderplant

Maybe somebody already identified the fault and has taken the problem link out of use.

Fingers crossed, i'll keep an eye on it, cheers SP

[Robertus]

Originally Posted by spiderplant

Maybe somebody already identified the fault and has taken the problem link out of use.

Are you aware of any issues between BT sport and now tv using apps? Recently at my house (350mbits) I’ve been getting buffering and quality drops. The in-laws have too.

My house:

350mbit (speed test usually good/low latency/jitter)
Shub4
Asus rtac68u
Apple TV 4K (not the recent one) hardwired to router
Nowtv app
Btsport app

In-laws:

100mbit (speed test usually good/low latency/jitter)
Shub3
Asus rtac68u
Samsung 50” connected via 5ghz WiFi - have done Speedtest from tv using fast.com and it can max out the connection.
Nowtv app
Btsport app

Issues started roughly at the same time. I’m area 20, they are burnage/didsbury.

[spiderplant]

Originally Posted by Robertus

Are you aware of any issues between BT sport and now tv using apps?

No, I wouldn't know where to begin with that one, sorry