100M : Packet Loss After Maintenance

[daveybm]

Sorry have not been on for a couple of days:-
16th:-
http://www.thinkbroadband.com/ping/s...6-12-2012.html

17th:-
http://www.thinkbroadband.com/ping/s...7-12-2012.html

18th:-
http://www.thinkbroadband.com/ping/s...8-12-2012.html

21st:-
http://www.thinkbroadband.com/ping/s...1-12-2012.html

According to a tech that came round the problem was noise from the recent upgrade work,
he also gave me a work number, according to Virgin Media Support the issues where caused by over utilisation he also gave me a work number which was the same as the one of the tech, so what the issue is/was I don't know as you can see things have improved but there are still dropped packets something I never had before.
The s/hub has been rebooted several times it has also ha a factory reset twice, non of which made any difference.
PS the completion date for the work was given as Wed 19th at 15:45
Dave

[Sephiroth]

Upstream congestion can cause packet loss but that really has to be severe. I still think there's upstream noise.

Can we see your DS/US modem stats again please? I'm interested in the number of T3 timeouts as well.

[horseman]

Originally Posted by Sephiroth

….Can we see your DS/US modem stats again please? I'm interested in the number of T3 timeouts as well.

in addition to seeing if there's any correlation with upstream frequency's and severity of the packet loss?

[qasdfdsaq]

Originally Posted by Sephiroth

Upstream congestion can cause packet loss but that really has to be severe. I still think there's upstream noise.

Agreed. Furthermore upstream congestion causing packet loss will be visibly worse at peak times. His isn't.

[daveybm]

Thanks for taking an interest,here is the latest update from Virgin Support regarding the ticket status,
F002279284 was closed as fixed on Wednesday. That ticket did have a fairly wide scope however (multiple cables), and a more local one has been opened under the new reference F002287892 for a continuing Forward Error Correction issue on your cable.

The estimated fix date for this ticket is the 28th. Please bump the thread for updates.

Here are my latest DS/US Stats

Downstream:-

DS-1 DS-2 DS-3 DS-4 DS-5 DS-6 DS-7 DS-8
Frequency (Hz) 314750000 Hz 282750000 Hz 290750000 Hz 298750000 Hz 306750000 Hz 322750000 Hz N/A N/A
Lock Status(QAM Lock/FEC Sync/MPEG Lock) Locked Locked Locked Locked Locked Locked N/A N/A
Channel ID 42 38 39 40 41 43 N/A N/A
Modulation QAM256 QAM256 QAM256 QAM256 QAM256 QAM256 N/A N/A
Symbol Rate (Msym/sec) 6.952000 6.952000 6.952000 6.952000 6.952000 6.952000 N/A N/A
Interleave Depth I=12
J=17 I=12
J=17 I=12
J=17 I=12
J=17 I=12
J=17 I=12
J=17 N/A N/A
Power Level (dBmV) 0.4 dBmV 0.0 dBmV -0.1 dBmV -0.3 dBmV -0.8 dBmV 0.2 dBmV N/A N/A
RxMER (dB) 40.3 dB 40.0 dB 40.3 dB 40.3 dB 40.3 dB 40.1 dB N/A N/A

Upstream:-

US-1 US-2 US-3 US-4
Channel Type 2.0 N/A N/A N/A
Channel ID 59 N/A N/A N/A
Frequency (Hz) 27400000 Hz N/A N/A N/A
Ranging Status Success N/A N/A N/A
Modulation QAM16 N/A N/A N/A
Symbol Rate (Sym/sec) 5120000 N/A N/A N/A
Mini-Slot Size 128 N/A N/A N/A
Power Level (dBmV) 39.3 dBmV N/A N/A N/A
T1 Timeouts 0 N/A N/A N/A
T2 Timeouts 0 N/A N/A N/A
T3 Timeouts 90 N/A N/A N/A
T4 Timeouts 0 N/A N/A N/A

The T3 timeouts seem to go so high and the reset to zero is there a max limit on the count for them? the modem was rebooted by me at 12.30.pm Friday and after the restart the T3 errors was zero so there has been 90 T3s in about 10 hours.
What is a Forward Error Correction issue?

PS I've also noticed I'm now on a different US channel number (was 58 is now 59)

Dave

[Sephiroth]

They are saying that the error correction count downstream on your modem is high.

But the T3 errors are the giveaway that there is an issue on youre upstream. The VM techs on the community forum would be able to spot this. They would see a high error correction count at the CMTS.

[horseman]

Originally Posted by daveybm

…..
PS I've also noticed I'm now on a different US channel number (was 58 is now 59)

Dave

The channel ID number could vary (by area and/or CMTS) depending on frequency/channel plan and ID convention used across linecards so in user context it's perhaps less ambiguous to also include actual frequency. Level2 having access to what node/cable segment your Hub is attached to will invariably use a different naming convention related to how the stats are reported.

For example on my CMTS I may be attached to linecard 1 and the channel ID numbering may start sequentially at 1 thru the whole chassis.
I may be connected to any 1(or 2 combinations as my connection has upstream bundling enabled) of 3 discrete channels available on my node such that:

Channel ID 1 = 45.8MHz and US0 (Level2 reference)
Channel ID 2 = 38.5MHz and US1
Channel ID 3 = 27.4MHz and US2

Just to muddy the waters further US0/1/2 can obviously alter against displayed headers(US-1,US-2 etc) in webpage of upstream tab in SHub stats as well!

Hence you probably want to take a screen shot of upstream to avoid any ambiguity caused by terminology and subsequent confusion and post that with results of TBB graph.

In your scenario Ch59 is 27.4MHz so previous Ch58 was probably 35.8MHz?
However what we're now interested in seeing is if there's any change in packet loss relevant to upstream channel that is actually locked.

The upstream channel may (but not always) change when rebooting the SHub as CMTS static load balancing attempts to evenly spread CM/SH's that are registering across the available upstreams (that may occur due to utilisation load or noise). Upstream channel can also change due to dynamic load balancing as well.

[Sephiroth]

The Horse has eloquently addressed the change of channel mechanism. The telling thing looking back on your first posted dtats (when you were on US Ch58) is that the number of T3 errors remains noticeable.

So somewhere in your circuit is a common cause point that is introducing upstream noise - still doing so. That's what needs to be taken up with VM. Do you have a thread on the VM Forum? I didn't find it.

[daveybm]

Yes this is the original thread:-
http://community.virginmedia.com/t5/...e/td-p/1609270

This is the thread after I was given a ticket number for resolving the problem :-
http://community.virginmedia.com/t5/...4/td-p/1621704

They only say what I've already said here.

The T3 count is now at 329 this is from zero at the s/hub reboot 48 hours ago.


Dave

[Chrysalis]

packet loss on congestion probably is more likely if low buffers are set as packets then get dropped when queues are full, but if large queues are configured (high buffering) then higher latency/jitter will occur instead of packet loss.

[daveybm]

Update on my problem,on 29/12/12 i opened the following thread on Virgin Community Support :-

Can somebody please give me a status check on F002287892, this was due to be completed on 28/12/12, but I am still gettin packet loss as bad as ever, Seph reckons it could be noise on the system and that you people have the equip to check it out, this problem has been on going since 3/12/12 when upgrades where done on the system, Ticket F002279284 was supposed to cure some of these issues but only resolved the over utilization on the ubr, at the moment my stats are :-
Downstream:-

Downstream DS-1DS-2DS-3DS-4DS-5DS-6DS-7DS-8Frequency (Hz)314750000 Hz282750000 Hz290750000 Hz298750000 Hz306750000 Hz322750000 HzN/AN/ALock Status(QAM Lock/FEC Sync/MPEG Lock)LockedLockedLockedLockedLockedLockedN/AN/AChannel ID423839404143N/AN/AModulationQAM256QAM256QAM256QAM256QAM256QAM256N/AN/ASymbol Rate (Msym/sec)6.9520006.9520006.9520006.9520006.9520006.9520 00N/AN/AInterleave DepthI=12
J=17I=12
J=17I=12
J=17I=12
J=17I=12
J=17I=12
J=17N/AN/APower Level (dBmV)0.0 dBmV-0.3 dBmV-0.4 dBmV-0.7 dBmV-1.1 dBmV-0.2 dBmVN/AN/ARxMER (dB)40.1 dB39.9 dB40.3 dB40.4 dB40.1 dB39.9 dBN/AN/A

Upstream :-

Upstream US-1US-2US-3US-4Channel Type2.0N/AN/AN/AChannel ID59N/AN/AN/AFrequency (Hz)27400000 HzN/AN/AN/ARanging StatusSuccessN/AN/AN/AModulationQAM16N/AN/AN/ASymbol Rate (Sym/sec)5120000N/AN/AN/AMini-Slot Size128N/AN/AN/APower Level (dBmV)39.3 dBmVN/AN/AN/AT1 Timeouts0N/AN/AN/AT2 Timeouts0N/AN/AN/AT3 Timeouts100N/AN/AN/AT4 Timeouts0N/AN/AN/A


As you can see there is a lot of T3 timeouts this number gets quite high and then resets back to zero and then starts counting up again.

I have already had a senior tech out to the house who confimed that the issue is on the network.

Thank you.
Daveybm

Finally my TBB Graph for today:-

TBB Graph :-

http://www.thinkbroadband.com/ping/s...9-12-2012.html

This is the reply I received a couple of days later:-


Hi daveybm,

Fault ref: F002287892 is for increased FEC's (forward error corrections) which is a system that compensates for dropped packets on the network.

Although FEC's are commonly synonymous with low SNR, this time the culprit appears to be faulty hardware which is now being replaced.

So the current target fix date has been set for 02/01/2013. This date may change, but it is dependent on the outcome of the hardware replacement. Please bump this post in a day or two and we'll check for the latest updates for you.
Kind Regards,

Jen_A

So it turns out my issues where caused by faulty hardware, the issue was fixed @ 6.30am
as you can see by this TBB graph:-

TBB graph
http://www.thinkbroadband.com/ping/s...3-01-2013.html

Since then all has been well in the land of internet, and I hope it stays that way, many thanks to all who contributed to this thread in an effort to help me solve my problems

Dave

[Sephiroth]

Great news. Pleased to have been of service.

I've left a technical reply on the VM forum. http://community.virginmedia.com/t5/...1633432#M11614