“An authoritative principle, belief, or statement of ideas or opinion, especially one considered to be absolutely true”

When innovators proposed Internet Protocol (IP) as the universal protocol for carriers in the mid 90s, they met with furious resistance from the traditional telecommunications community. This post asks whether the wheel has now turned full circle with new innovative approaches often receiving the same reception.

Like many others, I have found the telecommunications industry so very interesting and stimulating over the last decade. There have been so many profound changes that it is hard to indentify with the industry that existed prior to the new religion of IP that took hold in the late 90s. In those balmy days the industry was commercially and technically controlled by the robust world standards of the Public Switched Telecommunications Services (PSTN).

In some ways it was a gentleman’s industry where incumbent monopoly carriers ruled their own lands and had detailed inter-working agreements with other telcos to share the end-to-end revenue generated by each and every telephone call. To enable these agreements to work, the International Telecommunications Union (ITU) in Geneva spent decades defining the technical and commercial standards that greased the wheels. Life was relatively simple as there was only one standards body and one set of rules to abide by. The ITU is far from dead of course and the organisation went on to develop the highly successful GSM standard for mobile telephony and is still very active defining standards to this very day.

In those pre-IP days, the industry was believed to be at its nadir with high revenues, similarly high profits with every company having its place in the universe. Technology had not significantly moved on for decades ( though this does an injustice to the development of ATM and SDH/SONET) and there was quite a degree of complacency driven by a monopolistic mentality. Moreover, it was very much a closed industry in that individuals chose to spend their entire careers in telecommunications from a young age with few outsiders migrating into it. Certainly few individuals with an information technology background joined telcos as there was a significant mismatch in technology, skills and needs. It was not until the mid 90s, when the industry started to use computers by adopting Advanced Intelligent Networks (AIN) and Operational Software and Systems (OSS), that computer literate IT engineers and programmers saw new job opportunities and jumped aboard.

In many ways the industry was quite insular and had its own strong world view of where it was going. As someone once said, “the industry drank its own bathwater” and often chose to blinker out opposing views and changing reality. It is relatively easy to see how this came about with hindsight. How could an industry that was so insular embrace disruptive technology innovation with open arms? The management dogma was all about “We understand our business, our standards and our relationships. We are in complete control and things won’t change.”

Strong dogma dominated and was never more on show than in the debate about the adoption of Asynchronous Transfer Mode (ATM) standards that were needed to upgrade the industry’s switching networks. If ATM had been developed a decade earlier there would have never been an issue but unfortunately the timing could not have been worse as it coincided with the major uptake of IP in enterprises. When I first wrote about ATM back in 1993, IP was pretty much an unknown protocol in Europe. (The demise of ATM ). ATM and the telco industry lost that battle and IP has never looked back.

In reality it was not so much a battle but all out war. It was the telecommunications industry eyeball-to-eyeball with the IT industry. The old “we know best” dogma did not triumph and the abrupt change in industry direction led to severe trauma  in all sections of the industry. Many old-style telecommunications equipment vendors, who had focused on ATM with gusto, failed to adapt with many either writing off billions of Dollars or being sold at knock-down valuations. Of course, many companies made a killing. Inside telcos, commercial and engineering management who had spent decades at the top of their profession, found themselves floundering and over a fifteen year period a significant proportion of that generation of  management ended up leaving the industry.

The IP band wagon had started rolling and its unstoppable inertia has relentlessly driven the industry through to the current time. Interestingly, as I have covered in previous posts such as MPLS and the limitations of the Internet, not all the pre-IP technologies were dumped. This was particularly so with fundamental transmission related network technologies such as SDH / SONET (SDH, the great survivor). These technologies were 100% defined within the telecommunications world and provided capabilities that were wholly lacking in IP. IP may have been perfect for enterprises, but many capabilities were missing that were required if it was to be used as the bedrock protocol in the telecommunications industry. Such things as:

• Unlike telecommunications protocols, IP networks were proud that their networks were non-deterministic. This meant that packets would always find their way to the required destination even if the desired path faulted. In the IP world this was seen as a positive feature. Undoubtedly it was, but it also meant that it was not possible to predict the time it would take for a packet to transit a network. Even worse, a contiguous stream of packets could arrive at a destination via different paths. This was acceptable for e-mail traffic but a killer for real-time services like voice.
• Telecommunications networks required high reliability and resilience so that in event of any failure, automatic switchover to an alternate route would occur within several milliseconds so that even live telephone calls were not interrupted. In this situation IP would lackadaisically find another path to take and packets would eventually find their way to their destination (well maybe that is a bit of an overstatement, but it does provide a good image of how IP worked!).
• Real time services require a very high Quality of Service (QoS) in that latency, delay, jitter and drop-out of packets need to be kept to an absolute minimum. This was, and is, a mandatory requirement for delivery of demanding voice services. IP in those days did not have the control signalling mechanisms to ensure this.
• If PSTN voice networks had one dominant characteristic – it was reliable. Telephone networks just could not go down. There were well engineered and extensively monitored so if any fault occurred comprehensive network management systems flagged it very quickly to enable operational staff to correct it or provide a work round. IP networks just didn’t have this level of capability of operational management systems.

These gaps in capabilities in the new IP-for-everything vision needed to be corrected pretty quickly, so a plethora of standards development was initiated through the IETF that remains in full flow to this day. I can still remember my amazement in the mid 1990s when I came across a company had come up with the truly innovative idea to combine the deterministic ability of ATM with an IP router that brought together the best of the old with the new still under-powered IP protocol (The phenomenon of Ipsilon). This was followed by Cisco’s and the IETF’s development of MPLS and all its progeny protocols. (The rise and maturity of MPLS and GMPLS and common control).

Let’s be clear, without these enhancements to basic IP, all the benefits the telecommunications world gained from focusing on IP would not have been realised. The industry should be making a huge sigh of relief as many of the required enhancements were not developed until after the wholesale industry adoption of IP. If IP itself had not been sufficiently adaptable, it could be conjectured that there would have been one of the biggest industry dead ends imaginable and all the ‘Bellheads’ would have been yelling “I told you so!”.

Is this the end of story?

So, that’s it then, it’s all done. Every carrier of every description, incumbent, alternate, global, regional, mobile, and virtual has adopted IP / MPLS and everything is hunky-dory. We have the perfect set of network standards and everything works fine. The industry has a clear strategy to transport all services over IP and the Next Generation Network (NGN) architecture will last for several decades.

This may very well turn out to be the case and certainly IP /MPLS will be the mainstream technology set for a long time to come and I still believe that this was one of the best decisions the industry took in recent times. However, I cannot help asking myself whether if we have not gone back to many of the same closed industry attitudes that drove it prior to the all-pervasive adoption of IP?

It seems to me that it is now not the ‘done thing’ to propose alternative network approaches or enhancements that do not exactly coincide with the now IP way of doing things for risk of being ‘flamed’. For me the key issue that should drive network architectures should be simplicity and nobody could use the term ‘simple’ when describing today’s IP carrier networks. Simplicity means less opportunity for service failure and simplicity means lower cost operating regimes. In these days of ruthless management cost-cutting, any innovation that promises to simplify a network and thus reduce cost must have merit and should justify extensive evaluation – even if your favourite vender disagrees. To put it simply, simplicity cannot not come from deploying more and more complex protocols that micro-manage a network’s traffic.

Interestingly, in spite of there being a complete domination of public network cores by MPLS, there is still one major area where the use of MPLS is being actively questioned – edge and or metro networks. There is currently quite a vibrant discussion taking place concerning the over complexity of MPLS for use in metro and the possible benefits of using IP over Ethernet (Ethernet goes carrier grade with PBT / PBB-TE?). More on this later.

We should also not forget that telcos have never dropped other aspects of the pre-IP world. For example, the vast majority of telcos who own physical infrastructure still use that leading denizen of the pre-IP world, Synchronous Digital Hierarchy (SDH or SONET) (SDH, the great survivor). This friendly dinosaur of a technology still holds sway at the layer-1 network level even though most signalling and connectivity technologies that sit upon it have been brushed aside by the IP family of standards. SDH’s partner in crime, ATM, was absorbed by IP through the creation of standards that replicated its capabilities in MPLS (deterministic routing) and MPLS-TE (fast rerouting). The absorption of SDH into IP was not such a great success as many of the capabilities of SDH could not effectively be replaced by layer-3 capabilities (though not for the want of trying!).

SDH is based on time division multiplexing (TDM), the pre-IP packet method of sharing a defined amount of bandwidth between a number of services running over an individual wavelength on a fibre optic cable. The real benefit of this multiplexing methodology is that it had proved to be ultra-reliable and offers the very highest level of Quality of Service available. SDH also has the in-built ability par-excellence to provide restoration of an inter-city optical cable in the case of major failure. One of SDH’s limitations however, is that it only operates at very high granularity of bandwidth so smaller streams of traffic more appropriate to the needs of individuals and enterprises cannot be managed through SDH alone. This capability was provided by ATM and is now provided by MPLS.

Would a moment of reflection be beneficial?

The heresy that keeps popping up in my head when I think about IP and all of its progeny protocols, is that the telecommunications industry has spent fifteen years developing a highly complex and inter-dependent set of technical standards that were only needed to effectively replace what was a ‘simple’ standard that did its job effectively at a lower layer in the network. Indeed, pre MPLS, many of the global ISPs used ATM to provide deterministic management of the global IP networks.

Has the industry now created a highly over-engineered and over-complex reference architecture? Has a whole new generation of staff been so marinaded for a decade in deep IP knowledge, training and experience that it’s for an individual to question technical strategy? Has the wheel has turned full circle?

In my post Traffic Engineering, capacity planning and MPLS-TE, I wrote about some of the challenges facing the industry and the carriers’ need to undertake fine-grain traffic engineering to ensure that individual service streams are provided with appropriate QoS. As consumers start to use the Internet more and more for real-time isochronous services such as VoIP and video streaming, there is a major architectural concern about how this should be implemented. Do carriers really want to continue to deploy an ever increasing number of protocols that add to the complexity of live networks and hence increase risk?

It is surprising just how many carriers use only very light traffic engineering and simply rely on over-provisioning of bandwidth at a wavelength level. This may be considered to be expensive (but is it if they own the infrastructure?) and architects may worry about how long they will be able to continue to use this straightforward approach, but there does seem to be a real reticence to introduce fine-grained traffic management. I have been told several times that this is because they do not trust some of the new protocols and it would be too risky to implement them. It is industry knowledge that a router’s operating system contains many features that are never enabled and this is as true today as it was in the 90s.

It is clear that management of fine-grain traffic QoS is one of the top issues to be faced in coming years. However, I believe that many carriers have not even adopted the simplest of traffic engineering standards in the form of MPLS-TE that starts to address the issue. Is this because many see that adopting these standards could create a significant risk to their business or is it simply fear, uncertainty and doubt (FUD)?

Are these some of the questions carriers we should be asking ourselves?

Has management goals moved on since the creation of early MPLS standards?

When first created, MPLS was clearly focused on providing predictable determinability at layer-3 so that the use of ATM switching could be dropped to reduce costs. This was clearly a very successful strategy as MPLS now dominates the core of public networks. This idea was very much in line with David Isenberg’s ideas articulated in The Rise of the Stupid Network in 1997 which we were all so familiar with at the time. However ambitions have moved on, as they do, and the IP vision was considerably expanded. This new ambition was to create a universal network infrastructure that could provide any service using any protocol that any customer was likely to need or buy. This was called an NGN.

However, is that still a good ambition to have? The focus these days is on aggressive cost reduction and it makes sense to ask whether an NGN approach could ever actually reduce costs compared to what it would replace. For example, there are many carriers today who wish to exclusively focus on delivering layer-2 services. For these carriers, does it make sense to deliver these services across a layer 3 based network? Maybe not.

Are networks so ‘on the edge’ that they have to be managed every second of the day?

PSTN networks that pre-date IP were fundamentally designed to be reliable and resilient and pretty much ran without intervention once up and running. They could be trusted and were predictable in performance unless a major outside event occurred such as a spade cutting a cable.

IP networks, whether they be enterprise or carrier, have always had an well-earned image of instability and going awry if left alone for a few hours. This is much to do with the nature of IP and the challenge of managing unpredicted traffic bursts. Even today, there are numerous times when a global IP network goes down due to an unpredicted event creating knock-on consequences. A workable analogy would be that operating an IP network is similar to a parent having to control an errant child suffering from Attention Deficit Disorder.

Much of this has probably been brought about by the unpredictable nature of routing protocols selecting forwarding paths. These protocols have been enhanced over the years by so many bells and whistles that a carrier’s perception of the best choice of data path across the network will probably be not the same as the one selected by the router itself.

Do operational / planning architecture engineers often just want to “leave things as they are” because it’s working. Better the devil you know?

When a large IP network is running, there is a strong tendency to want to leave things well alone. Is this because there are so many inter-dependent functions in operation at any one time that it’s beyond an individual to understand it? Is it because when things go wrong it takes such an effort to restore service and it’s often impossible to isolate the root cause if it not down to simple hardware failure?

Is risk minimisation actually the biggest deciding factor when deciding what technologies to adopt?

Most operational engineers running a live network want to keep things as simple as possible. They have to because their job and sleep are on the line every day. Achieving this often means resisting the use of untried protocols (such as MPLS-TE) and replacing fine-grained traffic engineering with the much simpler strategy of using over-provisioned networks ( Telcos see it as a no-brainer because they already own the fibre in the ground and it is relatively easy to light an additional dark wavelength).

At the end of the day, minimising commercial risk is right at the top of everyone’s agenda, though it usually sits below operation cost reduction.

Compared to the old TDM networks they replace, are IP-based public networks getting too complex to manage when considering the ever increasing need for fine-grain service management at the edge of the network?

The spider’s web of protocols that need to perform flawlessly in unison to provide a good user experience is undoubtedly getting more and more complex as time goes by. There is only little effort to simply things and there is a view that it is all becoming too over-engineered. Even if a new standard has been ratified and is recommended for use, this does not mean it will be implemented in live networks on a wide scale basis. The protocol that heads the list of under exploited protocols is IPv6 (IPv6 to the rescue – eh?).

There is significant on-going standards development activity in the space of path provisioning automation (Path Computation Element (PCE): IETF’s hidden jewel) and of true multilayer network management. This would include seamless control of layer-3 (IP), layer-2.5 (MPLS) and layer-1 networks (SDH) (GMPLS and common control). The big question is (risking being called a Luddite) would a carrier in the near future risk the deployment of such complexity that could bringing down all layers of a network at once? Would the benefits out weigh the risk?

Are IP-based public networks more costly to run than legacy data networks such as Frame Relay?

This is a question I would really like to get an objective answer to as my current views are mostly based on empirical and anecdotal data. If anyone has access to definitive research, please contact me! I suspect, and I am comfortable with the opinion until proved wrong, that this is the case and could be due to the following factors:

• There needs to be more operations and support staff permanently on duty than with the old TDM voice systems thus leading to higher operational costs.
• Operational staff require a higher level of technical skill and training caused by the complex nature of IP. CCIEs are expensive!
• Equipment is expensive as the market is dominated by only a few suppliers and there are often proprietary aspects of new protocols that will only run on a particular vendor’s equipment thus creating effective supplier lock-in. The router clone market is alive and healthy!

It should be remembered that the most important reason given to justify the convergence on IP was the cost savings resulting from collapsing layers. This has not really taken place except for the absorption of ATM into MPLS. Today, each layer is still planned, managed and monitored by separate systems. The principle goal of a Next Generation Network (NGN) architecture is still to achieve this magic result of reduced costs. Most carriers are still waiting on the fence for evidence of this.

Is there a degradation in QoS using IP networks?

This has always been a thorny question to answer and a ‘Google’ to find the answer does not seem to work. Of course, any answer lies in the eyes of the beholder as there is no clear definition of what the term QoS encompasses. In general, the term can be used at two different levels in relation to a network’s performance: micro-QoS and the macro-QoS.

Micro-QoS is concerned with individual packet issues such as order of reception of packets, number of missing packets, latency, delay and jitter. An excessive amount of any of these will severely degrade a real-time service such as VoIP or video streaming. Macro-QoS is more concerned with network wide issues such as network reliability and resilience and other areas that could affect overall performance and operational efficiency of a network.

My perspective is that on a correctly managed IP / MPLS network (with all the hierarchy and management that requires), micro-QoS degradation is minimal and acceptable and certainly no worse than IP over SDH. Indeed, many carriers deliver traditional private wire services such as E1 or T1 connectivity over an MPLS network using pseudowire tunnelling protocols such as Virtual Private LAN Service (VPLS). However this does significantly raise the bar in respect to the level of IP network design and network management quality required.

The important issue is the possible degradation at the macro-QoS level where I am comfortable with the view that using an IP / MPLS network there will always be a statistically higher risk of fault or problems due to its complexity compared to a simpler IP over SDH system. There is a certain irony in that macro-QoS performance of a network could be further degraded when additional protocols are deployed to improve-micro-QoS performance.

Is there still opportunity for simplification?

In an MPLS dominated world, there is still significant opportunity for simplification and cost reduction.

Carrier Ethernet deployment

I have written several posts (Ethernet goes carrier grade with PBT / PBB-TE?) about carrier Ethernet standards and the benefits its adoption might bring to public network. In particular, the promise of simplification. To a great extent this interesting technology is a prime example of where a new (well newish) approach that actually does make quite a lot of sense comes up against the new MPLS-for-everything-and-everywhere dogma. It is not just a question of convincing service providers of the benefit but also overcoming the almost overwhelming pressure brought on carrier management form MPLS vendors who have clear vested interests in what technologies their customers choose to use. This often one-sided debate definitely harks back to the early 90s no-way-IP culture. Religion is back with a vengeance.

Metro networks

Let me quote Light Reading from September 2007. “What once looked like a walkover in the metro network sector has turned into a pitched battle – to the surprise, but not the delight, of those who saw Multiprotocol Label Switching (MPLS) as the clear and obvious choice for metro transport.” MPLS has encountered several road bumps on its way to domination and it should always be appropriate to question whether any particular technology adoption is appropriate.

To quote the column further: “The carrier Ethernet camp contends that MPLS is too complex, too expensive, and too clunky for the metro environment.” Whether ‘thin MPLS’ (PBB-TE / PBT or will it be T-MPLS?) will hold off the innovative PBB intruder remains to be seen. At the end of the day, the technology that provides simplicity and reduced operational costs will win the day.

Think the unthinkable

As discussed above, the original ambition of MPLS has ballooned over the years. Originally solving the challenge of how to provide a deterministic and flexible forwarding methodology for layer-3 IP packets and replace ATM, it has achieved this objective exceptionally well. These days, however, it seems to be always assumed that some combination or mix of Ethernet (PBB-TE) and/or MPLS-TE and maybe even GMPLS is the definitive, but highly complex, answer to creating that optimum highly integrated NGN architecture that can be used to provide any service any customer might require.

Maybe, it is worth considering a complementary approach that is highly focused on removing complexity. There is an interesting new field of innovation that is proposing that path forwarding ‘intelligence’ and path bandwidth management is moved from layer-3, layer.2.5 and layer-2 back into layer-1 where it rightly belongs. By adding additional capability to SDH, it is possible to reduce complexity in the above layers. In particular deployment scenarios this could have a number of major benefits, most of which result in significantly lower costs.

This raises an interesting point to ponder. While revenues still derive from traditional telecom-oriented voice services, the services and applications that are really beginning to dominate and consume most bandwidth are real time interactive and streaming services such as IPTV, TV replays, video shorts, video conferencing, tele-presence, live event broadcasting, tele-medicine, remote monitoring etc. It could be argued that all these point-to-point and broadcast services could be delivered with less cost and complexity using advanced SDH capabilities linked with Ethernet or IP / MPLS access? Is it worth thinking about bringing SDH back to the NGN strategic forefront where it could deliver commercial and technical benefits?

To quote a colleague: “The datacom protocol stack of IP-over-Ethernet was designed for asynchronous file transfer, and Ethernet as a local area network packet-switching protocol, and these traditional datacom protocols do a fine job for those applications (i.e. for services that can tolerate uncertain delays, jitter and throughput, and/or limited-scope campus/LAN environments). IP-over-Ethernet was then assumed to become the basis protocol stack for NGNs in the early 2000s, due to the popularity of that basic datacom protocol stack for delivering the at-that-time prevailing services carried over Internet, which were mainly still file-transfer based non-real-time applications.”

SDH has really moved on since the days when it was only seen as a dumb transport layer. At least one service provider company, Optimum Communications Services offers an innovative vision whereby instead of inter-node paths being static, as is the case with the other NGN technologies discussed in this post, the network is able to dynamically determine the required inter-node bandwidth based on a fast real-time assessment of traffic demands between nodes.

So has the wheel has turned full circle?

As most carriers’ architectural and commercial strategies are wholly focused on IP with the Yellow Brick Road ending with the sun rising over a fully converged NGN, how much real willingness is there to listen to possible alternate or complementary innovative ideas?

In many ways the telecommunications industry could be considered to have returned to the closed shutter mentality that dominated before IP took over in the late 1990s – I hope that this is not the case. There is no doubt that choosing to deploy IP / MPLS was a good decision, but a decision to deploy some of the derivative QoS and TE protocols is far from clear cut.

We need to keep our eyes and minds open as innovation is alive and well and most often arises in small companies who are free to think the the unthinkable. They may might not be always right but they may not be wrong either. Just cast your mind back to the high level of resistance encountered by IP in the 90s and let’s not repeat that mistake again. There is still much scope for innovation within the IP based carrier network world and I would suspect this has everything to do with simplifying networks and not complicating them further.

Addendum #1: Optimum Communications Services – finally a way out of the zero-sum game?