In telecom, the "demarc" represents the point in the network where the responsibility differs; it is the interface between the service provider and the customer.
- Traditional Analog POTS: The Demarc is often a test port on the side of the building. If something is broken between the test port and the customer's phone, then it's the customer's responsibility to fix it. Otherwise, it's the service provider's responsibility.
- ISDN T1 service: The Demarc could be an RJ45 smartjack installed in a wiring closet. If something is broken between the smartjack and the customer's PBX or desk phone, then the customer must resolve the problem. Otherwise, the problem must be fixed by the service provider.
The Demarc clearly represents the point where the customer's responsibility lies. If his cabling inside his building goes bad, he can clearly understand that it is his responsibility. But if the T1 smartjack itself is struck by lightning, then he clearly knows that it is not his responsibility, even though that smartjack is within his building.
The service provider benefits by the clear line of separation. He can deliver a robust product, predictably and reliably, up to the demarc point. With a demarc, he does not have to hope that the customer buys the right kind of cable, provides the right sort of battery, or the proper grounding.
In the era of Hosted IP PBX with platforms like Metaswitch and BroadSoft, many service providers are offering services with no real demarc established.
The Excellent VoIP Service Provider & The Terrible Service
Consider the case of a Metaswitch service provider, with a full CFS, Media Gateways, EAS and Perimeta platform. They have what it takes to offer Hosted PBX. So their salesmen sign contracts. The customer buys a shiny new Polycom phone, which receives its configuration from the Metaswitch Provisioning Server.
Two weeks after installation, they have problems with calls being disconnected. After extensive research, they find that the customer's firewall had been upgraded to the latest version -- and it has an incompatible SIP ALG.
A month later, the audio quality is poor. They discover that the ISP, Time Warner, is having some congestion problems certain times of the day. But to find the problem, they had to travel to the customer site and setup a device to monitor the packet performance through the Internet.
The ISP gets their problem settled, and six months later, one of the Polycom SIP phones starts going crazy every few days. It just locks up. After extensive work, and negotiations between the customer, the Service Provider, and the customer's IT provider, they discover a problem with the PoE unit on one of the customer's ethernet switches powering the phone. The problem only affected that one phone -- but it meant that user could not receive service reliably for two weeks.
Why can't this service provider offer a quality service? It's because their service relies on components that are not robust. The Hosted IP PBX Core Platform might be rock solid, well designed and maintained perfectly. But the Hosted IP PBX product fails due to problems with:
- The customer's LAN cabling (Layer 1)
- The customer's PoE switch power, VLAN, switching performance, software stability, and configuration (Layer 2)
- The customer's DHCP server, and possibly DNS servers (Layers 2-4)
- The customer's routing and internal network congestion or QoS implementation (Layer 3)
- The customer's firewall performance, reliability, NAT consistency, and SIP ALG behavior (Layer 3 and 4)
- The Customer's ISP's DNS server's ability to properly handle NAPTR and SRV records (Layer 4)
- The Customer's ISP's routing performance and congestion (Layer 2-3)
- The Service Providers ISPs' routing performance and congestion (Layer 2-3)
Over The Top, But with a Hard Landing
Many call this carrier "Over The Top", OTT, meaning that the Hosted PBX VoIP service is provided using the Internet link from another ISP. The advantages for the VoIP Service Provider are numerous: They can offer service to customers that are geographically in areas outside their own. And they can take advantage of lower-priced WAN bandwidth than they can offer themselves. Of course, there are the disadvantages of relying on another infrastructure.
But beyond the concept of Over-The-Top, this Service Provider is relying on the customer's local network extensively. Many impressive companies have numerous problems with their network, and VoIP is even more demanding of perfection than most IT applications. When you rely on a customer's internal switches, routing, DHCP, and DNS, then you're essentially hiring the worst IT technician the customer will ever hire.
It gets worse. When you rely on a customer's internal network and their ISP, you lose visibility into the problems. In most cases, the VoIP Service Provider has no ability to login to the the customer's routers, switches, and firewalls to actually diagnose and troubleshoot problems. A simple problem like the failing PoE power unit may be difficult to resolve because the VoIP Service Provider cannot access the ethernet switches to check their status or health. The VoIP Service Provider, however, has accepted responsibility for providing robust service.
Five Facts of Hosted PBX.
Whether you use BroadWorks or Metaswitch (or something else - my condolences), you will eventually discover five facts about Hosted PBX.
1. You are responsible for quality audio all the way to the handset of the desk phone.. The customer wants to buy a simple PBX replacement. They expect you to do what it takes to make the phones work all the time, every time. The expect you to not to offer them a service that is impossible to provide reliably.
2. You cannot provide a VoIP service that is more robust than the network it is built upon. You may have geographic redundancy in your core network. You may have high-availability designs with sub-second failover. But you cannot overcome faulty cabling, switching, DHCP, DNS, routing, and firewalls that disable your SIP phones.
3. You cannot improve a network you do not operate. Even when you know precisely what is wrong with a customer's network, they haven't hired you to run their network. They are only vaguely interested in your advice on what needs to be changed in their network.
4. Corporate networks are not typically robust enough to support VoIP reliably.Many company networks seem to work fine, mostly because the staff learn to avoid the things that don't work well. Problems are common with poor cabling, duplex mismatch, periodic congestion, firewalls and switches that need occasional reboots, and momentary Internet outages.
5. Internet Service Providers are often not robust enough and are often too congested to support VoIP reliably. ISPs are getting better, but many experience momentary congestion that only barely affects data traffic, but practically disables VoIP. Many have DNS servers incapable of properly handling SRV and NAPTR.
Living The Lies
The Big Lie of Hosted PBX is: "You can succeed Hosted PBX over any network, using any good SIP Phone."
The Bigger Lie is: "In this economy, you must offer Hosted PBX over any network you can, and using whatever device the customer wants!"
People believe these lies because their falsehood is revealed slowly. You can, in fact, get many phone calls to work across almost any network. Yes, you can in fact get some service with nearly any SIP Phone. And it'll probably work well enough, long enough for the Salesman's Commission payment to clear the bank.
But in the long term of weeks and months, these installations nearly always have grave problems. The cost of not supporting the customer's LAN is borne by many hours of the Service Provider Support Staff banging their head against the wall trying to troubleshoot a network they just can't actually fix. In far too many cases, the customer is lost. They get fed up with terrible telephone service and leave the VoIP Service Provider.
You Can Have a Reliable Hosted PBX Service
After assisting with over 100 VoIP Service Provider designs, deployments, and operation, here's the secret: It's Broken By Default. If you cannot actually guarantee that the phone and everything that it depends on is functioning properly, then it's safe to assume that the network is broken, and will stop working.
Someday, ISPs might have enough overcapacity that QoS Engineering isn't needed. Someday, corporate networks may be robust enough to support Hosted VoIP from any Service Provider without a problem.
But if you intend on offering Hosted PBX services before those halcyon days, get ready for some old-fashioned risk-averse engineering.
I have been to the mountaintop, and I have seen over to the other side:
- Customers only using the phones we actually support.
- Ethernet switches that we provide, and we maintain, and we monitor and troubleshoot.
- Routers and Firewalls that we maintain and control.
- Diagnostic equipment that allows us to see SIP Phone behavior.
- Access links to customers that we can troubleshoot and test.
- A robust and maintained VoIP Core network.
- PSTN carriers that reliably get calls in and out, without degrading the audio.
- Customers that keep the VoIP service because it works so well.
- Sales and deployment processes that actually work every time, consistently delivering a quality, proven, tested service.
- A support team that can reliably find and fix problems when they do occur, without extended hassles for the customer.
Argument Against: Data Network Support is Not Our Forté
Clients have told me that this is not for them. They say they're a phone company, not a network integration company. But as long as you are troubleshooting your customer's phones going through somebody else's data network, then you're doing the network integration anyway for their data network. You have to find and resolve DHCP, DNS, cabling, switching, routing, NAT and firewall problems -- and yet you want to believe you're not doing network integration.
Argument Against: Deep Network Integration is Too Expensive
Even more have balked at the apparent cost. "You can't go in there and replace the customer's Ethernet switches, cabling, routers, and firewalls, just so you can sell them VoIP service!" That's true, but there are two misleading elements in this conclusion:
- To offer VoIP service, you don't have to be the full network integration provider for the customer. To offer a robust service, you do need to control the cable that the phones are connected to, and the devices on both ends of that cable (typically the phone and the Ethernet switch.) But you don't necessarily need to interfere with any of the cables, switches, or routers currently in use at the customer.
- Further, the support costs of troubleshooting a customer's network is quite high, both in labor costs, and in goodwill lost due to a failing service.
I don't expect all of you to believe me.
One last note: I don't really expect everybody who talks to me about this to believe what I'm saying. Some of you will go on selling Hosted PBX Over The Top, and Over the Customer's Network. In fact, Verizon just in October 2012 decided to deploy an Over-The-Top, Over-The-Existing-LAN Hosted PBX service..
And that's fine, because there are customers out there with crazy SIP ALGs in their routers, and cabling that runs across florescent lights, and erratic firewall rule changes, and the oldest ethernet switches known to man, and a taste for the cheapest deal in Internet service this month.
And those customers are going to need somebody from whom to buy Hosted VoIP. And if you don't want to adopt a Deep Integration model of network support for Hosted VoIP, these customers will be all yours someday.