Having been offered a free upgrade on my large number of ADSL branch connections to ADSL2+ I informed the business that the existing performance problems would be resolved in the next few weeks. I was advised that there would be no business impact and a “seemless” transfer.
One morning a couple of weeks ago several branches started complaing of poor performance. Ping rates were ramping from 19ms to 900ms then dropping to 19ms before continuing the cycle. Some sites exhibited totaly loss of connectivity which could only be restored by powering off the router for 20 minutes. On sites with total loss of connectivity the ISP picked this up on the alerting system they use and asked us to perform standard checks i.e. replace the filter etc etc.
The ISP had not given us a timetable for the upgrade however it dawned on me that this was the possible scenario due to the quantity of problems arrising on what was a previously stable platform. The ISP confirmed that the affected sites had been upgraded to ADSL2+ however the problems experienced were due to the 10 day training period.
Whilst I accepted that a training period would cause a slight degredation in performance it should not be causing the issues experienced i.e. within a couple of hours the line should have settled a reasonable amount yet the continuing ramping of ping rates between 19ms and 900ms continued.
An SNR attenuation was applied to the lines which did have a stabalising effect however approx 7 days after the upgrade these sites degraded in performance again.
The CISCO routers were checked for firmware revision between sites exhibiting the fault and working sites that had been upgraded. Whilst there was no correlation between the fault and the version of IOS the ISP established between CISCO and BT that a patch to resolve the problem was available which we applied to five of the affected sites.
The patch had no effect on the issue and after further discussions it became apparent that BT and CISCO were continuin to work on developing a firmware fix.
The cause of the problem is an incompatibility between the mslam and the cisco router and only effects certain exchanges that are carrying another vendors equipment as the mslam.
Tomorrow we should receive Zyxel routers as a temporary fix on the affected sites until CISCO have created a new release of firmware.
Loading tweets...
6 comments
admin says:
April 15, 2010 at 1:12 pm (UTC 0 )
Zyxel routers have been installed at several of the problematic sites. Even though the routers could be undergoing a training period the initial feedback is a dramatic performance increase.
Further routers at the remainder of the degraded sites will be installed over the course of the day.
There is still no news of a Cisco IOS firmware fix.
admin says:
April 15, 2010 at 1:22 pm (UTC 0 )
Those sites that have had a Zyxel router installed and have had not had an SNR attenuation change are experiencing lower ping rates than those with an SNR attenuation applied however it only ran for approx an hour before experiencing a major dropout.
This would leave us to believe that an SNR attenuation is still requierd even with the replacement Zyxel router to stabilise the site.
admin says:
April 16, 2010 at 5:39 pm (UTC 0 )
Question
Is the SNR / line attenuation a ‘red herring’ , workaround or is it offered as an official resolution from BT? If attenuation has to be applied what are the performance repercussions
Answer
This is not offered as an official resolution from BT. As mentioned above, the main problem was identified as a capacity issue. The stability settings and interleaving settings should only be adjusted only if there are further problems with a site now that we are sure the firmware and capacity issues are not contributing to the problem.
Interleaving is error correction on the line at BT’s side which will try to remove any errors in the data that is passing through the line, this is used to remove errors on the line by correcting any corrupted packets before they reach the EU router and has a side effect of higher ping times due to the time it takes to check and correct any packets before being passed onto the EU router.
SNR is the buffer in the signal between the throughput and noise on the line, the higher this is the bigger the buffer and lower effect any noise or line imperfections have on the connection, a side effect of raising the SNR is that there is less signal for the download speed so downloads will be reduced in speed. SNR can be used to eliminate errors on the line without the need for interleaving if those errors are due to noise.
The stability options reduce the delay on BT’s systems before either of these two options are changed. As a rule the standard stability option will allow for 12 drops within a 24 hour period before increasing the SNR while setting the line to Super stable will cause the SNR to rise after 3 drops in the same time period
Interleaving in detail
http://www.kitz.co.uk/adsl/interleaving.htm
SNR in detail
http://www.dslzoneuk.net/adslmax_explained.php – note, when we refer to changing the SNR it is the SNR margin mentioned here that is changed.
admin says:
April 21, 2010 at 8:56 am (UTC 0 )
After the Firmware upgrade was applied to the problematic sites CRC errors started to appear.
This was diagnosed to a lack of capacity across the ISP’s backbone. The ISP increased capacity and the CRC errors went away.
A combination of the new Firware on the CISCO routers and the increase in capacity across the backbone has stabilised the problematic sites.
The ISP is recommending that the firmware upgrade is applied across the estate to prevent further sites becoming degraded.
frankie says:
August 31, 2010 at 6:06 pm (UTC 0 )
Hi there, sorry I’m new here and not sure how to start a new thread?
I’ve got a very similar problem here with ADSL2+ with our Cisco 837 routers. After reading about it has become apparent a firmware upgrade will perhaps cure this? Does anyone have the latest firmware for this particular router at all please?
Many thanks in advanced.
Paul Brown says:
September 14, 2010 at 11:34 am (UTC 0 )
Hi frankie
I applied 4.0.18 on my 877 routers to resolve the problem
regards