Ask HN: Why isn't VoIP better?

143 points by scandox ↗ HN
I'm very sensitive to bad call quality. I'm very easy going about most things, but poor sound quality or an intermittent connection on a call, make it completely impossible for me to maintain normal communication and also makes me extremely cross.

As a result I am very attuned to the general quality of different call services. I always know when I'm on a PSTN that terminates to another PSTN - yes maybe the voices are a bit trebly but the connection is so consistent and wonderful. I actually appreciate the quality of those calls.

Skype, Viber, our own corporate VOIP system (whatever the hell it is) all drive me completely nuts. I have 240MB/s at home, but still VOIP lets me down: about 1 in 3 calls is of a quality I am happy with.

I guess my question is: will this ever be solved to my satisfaction, or is it simply the nature of the beast and "good enough" for most people?

140 comments

[ 3.3 ms ] story [ 143 ms ] thread
Where do you live where you can get 2 gigabit speeds at home?
It's more likely he meant 240Mbit/s
(comment deleted)
(comment deleted)
240 Mbps. So AFAIK 240 megabits per second. Is my original notation ambiguous?
A capital B indicates "byte", and a lower-case b indicates "bit". 240 MBps is eight times faster than 240 Mbps.

When it comes to speeds in bits per second, M almost always means 1000000, and not 1048576.

Networking speeds are pretty much universally represented in bits (per second) -- "little b". Storage, on the other hand, uses bytes -- "big B".

So, yeah...

> I have 240MB/s at home ...

You're saying you have a 240 MBytes x 8 -- or 1920 Mbit/s -- connection.

Possibly OP was incorrectly confusing Mbps and MBps but in Japan Sony offers 2Gb/s for home connections.

I had their 1Gb/s plan for ~6000JPY/Month when I was living in Kyoto. Was great

How is a 2 Gb/s home connection presented? 10GbE?
It's probably because most people have horrible quality microphones.
Sorry to down vote, but that's completely the wrong vector. Its a network issue (of some sort - bandwidth, latency, throttling).
Yeah, if you see the quality (or lack thereof) of microphones in old telephone handsets you'd be amazed.
How are games and voice software targeted at gaming managing to make things so much more clear?

If it's a network issue, it's likely unsalvageable because the network is just crap. We need to be using whatever those are using. It is usually crystal clear and people are easily understandable even during large gatherings (EVE Online battles).

I'm the same as the OP, and what's worse, people talk 20 feet from the cisco conference phones (because those are everywhere) and it makes everything sound completely terrible. I don't listen half the time. That is definitely a microphone issue.

If you're audio sounds great for a network game and crappy for a phone call (to the same person you are playing a game with), its because some one (your ISP, your VoIP software, the other parties ISP, or VoIP software, or any network hop in the middle), is intentionally choosing poor quality codecs for you to lower your bandwidth utilization.

I've heard directly of novice VoIP engineers unintentionally taking out corporate networks trying to "help" improve audio quality, by picking a higher quality codec. It worked great when they tested phone calls in the middle of the night maintenance window. It really sucked the next day when packets started colliding. Crappy sounding codecs sound a heck of a lot better than 50% packet loss.

Is it? Why do people with headsets sound 10x better over Google Hangout than people using the integrated laptop microphone?
I find I agree.

- lossy vs lossless codecs (PSTN isn't analogue end-to-end any more, but is lossless as far as I'm aware. VoIP seems to have been about squeezing as much traffic down the pipe as possible, at the expense of quality.)

- circuit vs packet switched networks (Congestion, contention, jitter on shared packet switched networks. While I'm sure PSTN doesn't have "physical" circuit switching any more, I do get the impression that high-bandwidth bearers are chopped into fixed-width chunks, like 64k ISDN, and will reach capacity and stop accepting demand rather than degrade individual connections)

For me, wideband audio helps, even with the jitter and lossy codec. This is starting to become common in the UK on cellular calls between smartphones, even between different network providers. Audio bandwidth is comparable to Facetime Audio.

The lossy part of PSTN occurs at the front-end where only a 3KHz band of audio is passed through the network.
VOIP calls are very sensitive to intermittent connectivity.

Sometimes wifi will work in batches. No packets will get through for a few hundreds of milliseconds (or seconds), then they will get sent all at once.

Sometimes UDP traffic is blocked completely and the call has to get relayed over TCP. Now no packets can get dropped and you get delays due to retransmits.

Sometimes 3g/4g connection disappears because someone entered a building an overpass or passed a corner in a city.

Sometimes the sides of the conversation are far and the audio delay is high.

If any side of the conversation encounters a problem described above both will get a bad quality.

Among other things, I'm a Skype for Business admin. We've found that having our users use Ethernet when working from home instead of WiFi eliminated a lot of audio quality complaints.
At home I use Ethernet, a Blue Yeti USB microphone and a solid pair of Philips USB headphones. Of course I don't know what my counterparts on the other end are using.

I guess my problem seems to be that we swapped a system that had a consistent call quality for one that is inconsistent, for a whole bunch of reasons. It seems to me, as if the general quality of telephony took a huge step backwards. And I don't understand, in an age of technical progress, why we should (or do) put up with that.

Are the 1 in 3 good calls pretty random, or do they tend to be with the same counterparts? Do you notice a difference between the quality of your calls into conferences (if on the same platform) and direct calls with colleagues?

WiFi can be a source of the problem even within the corporate network if the access points weren't designed for VoIP.

It seems random to me, but of course when I'm in User-mode I'm probably quite a poor observer.

I'm not a power user of VOIP, but these are my common scenarios:

1. At home, Skype and Google Hangouts. Hardware setup as above. 240 Mbps connection. Making calls to family abroad. This is the 1 in 3 scenario - we just keep trying till we get a connection that seems to work. Usually the call ends whenever the line finally "goes bad".

2. At work: Skype, Hangouts, Upwork. Mostly one to one calls with potential hires, colleagues, outsource people etc. WiFi. About a 20Mbps connection. Actually now I write it down Upwork is the best - which is weird because in general their systems suck. But Skype and Hangouts are both a consistent source of pain, dropped calls, bad quality, sudden total loss of audio etc...

3. At work: Company phones. Generally the quality is OK when calling out to landlines and mobiles (cell), but I have colleagues who I know are on VOIP at other end and these calls are like 1 in 2 where we just give up.

4. On mobile (4G): Viber, WhatsApp, Skype. Generally I'll call. Within a few seconds we'll be talking like people in a disco and I give up and call back on regular phone.

I also observe colleagues doing a lot of: "can you repeat that", "sorry repeat that" when they're using Skype at the office. Even hearing that contributes to my sense of VOIP-malaise.

Scenario one stands out to me. There will be a lot of hardware between you and the other party in your call and trouble can happen at any of them. The connection quality you get will depend on the worst performing hardware in that chain.
Sounds like your problem isn't with VOIP, it's with softphone VOIP clients and/or OTT carrier. The company phones are probably VOIP (do they just have ethernet connections?), but connected up to a telco which handles traffic nicely.

Most telcos are sending a huge amount of traffic round over IP. The difference is that they are controlling the bandwidth so stuff just-works, whereas when you use a purely OTT service, you're probably stuck with your packets getting the same treatment as any old web request. Most of the calls you think you can detect as POTS probably have VOIP legs in them somewhere if they're long distance.

Which of the services are the most expensive? My guess is the company phone. Are you paying for any of the others?

The only thing that could make VoIP voice quality as consistent as PSTN is a consistent application of QoS.

And that is why I have very mixed feelings about Net Neutrality. Sure, I don't think that Verizon should be allowed to penalize NetFlix or $randomVideoStartup by favoring its video traffic over theirs, but there is no technical way for voice to be consistently good without a consistent handling of its data traffic.

Network neutrality doesn't really have anything to do with QoS. Network neutrality says you can't prioritize packets from Comcast over packets from Netflix. It doesn't say you can't prioritize packets a customer has marked as more important over packets the same customer has marked as less important.
How much of the VoIP contention is on the user end vs the ISP end though?

I have a 1 Gbps connection at home and can still have a VoIP call across the world mess up since I might not have the best route to whoever I'm calling.

I don't believe what you ask for can ever exist. "Clear, Distortion Free, Low-Latency. Pick any two" comes to mind.

When you're doing voip you're doing so over UDP because TCP has far too much latency to be agreeable, but this means a lost packet will be a digital distortion. The nature of the internet is that packets get lost or arrive out of sequence- that's why TCP exists in the first place.

So what you're saying is: VOIP will never match PSTN? Because that's the mood music I get from the sysadmins at work too.
Never is a long time. What if some day the technology exists to interpolate dropped packets (e.g., using ML) fast and cheaply enough to fill in the distortion? That doesn't seem too outlandish in the next 20, or even 10 years.
That sounds fine, except for the timeline. What I'd like is an affordable, ubiquitous solution next week :)
Uh, the modern codecs are already using linear prediction and correction. (essentially a rephrased more advanced perceptron) This is why SILK (Skype), Speex and Opus (SILK-derived part) can work even mildly well on bad links. They predict syllables and also long term speech characteristics. Going to full sentences or words is much harder and not likely to bring gain other than free subtitles. And would require really good voice recognition comparable to Siri at least. Good luck running that on an embedded device reliably.

Remember, the codec has to handle less than 20 ms latency. Running a sizable ANN (fast) or HMM in that time is not easy or even possible. A small one will be no good for various speakers.

Instead the codecs use various specialised error hiding tricks and adaptive strategies.

Heck, Codec2, one of the strongest voice codecs around could be used if all else fails. It is beginning to be used in SDR and amateur radio - optimised for availability and intelligibility not quality. It would be nice to have a variant integrated with CELT for higher quality.

VOIP is considerably better than PSTN if configured correctly. There's no comparison. Your sysadmins don't understand VOIP.
Their ignorance is always just slightly less than mine. Which is a sign of their intellectual "efficiency".
just want to point out: vast knowledge of telephony/VOIP is not something every sysadmin should be expected to know. If you don't have telephony expertise on staff I hesitate to suggest someone build out a VOIP infrastructure on their own. It can be extremely time consuming to learn and it is definitely painful to troubleshoot.
Shouldn't it be possible for me to get some IP phones in my office and tunnel through my router to a competent online voice provider? If so, can you recommend such a provider?
Yes, that is exactly what we do at work. We have ~20 IP phones in the office plus a few in remote locations. A few of us also use softphones (X-Lite or Zoiper) & USB headsets.

Our pbx is hosted by http://sipworxx.com/ and it works great.

http://voip.ms is pretty decent and offers a few nice features that can avoid building an on-site PBX.

I use this with good Polycom "HD voice" devices. I'm also rather picky about voice quality.

Eh, I'd argue this varies a lot. In some cases you are more likely to run into issues setting up multiple SIP clients behind a NAT trying to connect to a server out on the public internet than you are to install your own PBX on-premises.

Grandstream, Yealink, etc. all provide pretty easy to administer Asterisk-based PBX appliances that can be paired with a SIP trunk of your choice (shout out to Flowroute here, they're great).

If you're feeling more adventurous sipXecs from sipfoundry is pretty easy to configure relative to the additional complexity it brings over the more simplified alternatives. I went from having 0 real world experience with SIP to managing a small installation for my aunts shop as well as one at home in a couple days.

For proprietary offerings there's of course Skype for Business, etc. and I would argue they aren't terribly hard to configure or maintain either.

Most of the work is just in network configuration once you have multiple sites and you need to ensure QOS going over your MPLS or whatever to ensure your voice packets take priority - that can be more difficult but that's something a competent network admin can help you with.

I think that statement needs some qualification. If you're sending the VoIP traffic across networks you don't control all bets are off. VoIP across an enterprise's private network or through a leased network where SLA's can be mandated and QoS can be controlled is one thing. "Wild west" VoIP across the Internet, especially through sketchy networks, is another.

Edit: "canbe"

Its bandwidth efficiency, latency, and jitter, actually.

Just like in the old days of 9600 baud leased lines it was hard to get acceptable console performance, and now people have megabit connections and CLI performance is not unusually constrained over the megabit connections. Of course its only using a fraction of 1% of the capacity.

Also remember that end users might have megs of bandwidth to carry K of voice traffic, but somewhere along the line there's someone constrained. Just like work offices don't have to be dumps, but its hard to put a beancounter $ value on not working in a dump, leads to a lot of open office foolishness.

> When you're doing voip you're doing so over UDP because TCP has far too much latency to be agreeable,

Lots of wifis, corporate especially block all UDP traffic. Sometimes they even block all non http/https traffic. Apps like Skype or Biocoded will find a way to route the call, but that can sometimes mean it has to go over https!

Could it be the upload speeds or QoS settings? I don't use voip, my wife skypes long distance and it seems pretty good for both video and audio and our internet is only 2ish Mb down and 1ish Mb up!
Jitter is one of the words you're looking for. 20 milliseconds does not sound like much, but when it's a delay in some (but not all) packets in a call getting to you, you can sure hear it!

Jitter is why you could have a 1 GB/s connection at home, and still have the same occurrence of poor call quality.

QoS (Quality of Service) can help, but only if it is applied at every router the traffic for a call goes through. QoS-assured bandwidth is more expensive, and cost-conscious VoIP providers don't seem to be willing to pay for it.

(edited to add) I'm a Skype for Business admin.

> QoS (Quality of Service) can help, but only if it is applied at every router the traffic for a call goes through.

You really only need it on routers where a bottleneck is. If a router is forwarding 100% of packets immediately there is nothing for QoS to do.

In practice what this means is that the most important place to have it is at the border where you go from high bandwidth LAN to lower bandwidth internet.

In particular if you control a router there, make sure it knows to limit traffic to what the next hop link speed actually is. In will often see the link as gigabit ethernet even if it's slower than that and then forward all traffic when you really want it to be the device that chooses which traffic gets dropped.

Unfortunately if your ISP supports burst bandwidth it's not always easy to get it right without breaking that.

QoS depends on tagging (DSCP), and if one of the routers along the way doesn't support it or isn't configured for it, those tags are often dropped - or at least that's what I've seen in our company's network.
Like so many other things in tech, "it depends".

Network gear can be configured to treat frames/packets in various ways -- leave the ToS/DSCP bits alone, strip them out, or remark them. At the edge of a network, it's common to mark them. In the core of a network, you'll usually leave them alone (but honor them). On connections to third-parties (such as ISPs), it's not unusual for them to strip your QoS markings -- unless you're paying extra for them not to.

For a random packet that leaves your home network and hits the Internet (even VoIP traffic), I'd be surprised if it ends up at its destination with those same QoS bits set.

Like zrm said, though, if there's no congestion on links, prioritizing traffic typically isn't going to do you much good. It's only when there's contention that moving packets to the front of the line helps out.

Except Internet routing is not smart, the routers cannot send you through a low bandwidth but uncontested and low latency pipe when your packets are unmarked because you didn't pay a premium for a peering agreement.

It actually can hurt the Internet performance as a whole, as the tiny VoIP packets prevent Jumbo throughput ones from working really well.

It takes just one bad hop to ruin latency.

Sure they can, especially on MPLS circuits sold for exactly this reason -- and with SLA commitments to deliver packets w/ specific markings within specific parameters (i.e. up to 5 Mbps of traffic marked ef will have maximum latency of 30 ms between A-Z).

Not on your home Internet connection, perhaps, but if you're a company and you want guaranteed performance levels (i.e. for VoIP traffic), you can certainly get it.

Single anecdotal counterpoint:

My corporate VOIP system is made of Polycom phones connecting via SIP to local Asterisk servers, and then those have interconnects to PSTN gateway services.

The quality on-campus is uniformly extremely high.

Some people have the Polycom phones at home (for working remotely) and calls to/from those are of similarly high quality, even in conferences.

Connections to cellphones range from mediocre to bad.

Connections to POTS land lines range from bad to pretty good.

Overall I would say that our VOIP system doesn't have a problem.

PSTN uses a digital backbone these days; it's just supersized carrier-grade digital backbone, not 'what hacky software can we get on the cheap'. Phone companies prioritise their voice traffic over their own backbone and have network admins monitoring it closely, whereas your skype calls are just 'more traffic from the proles' and get to compete with everyone else.
So VoIP can be better... at the expense of needing more bandwidth. During the session initialization the two VoIP endpoints handshake over which codecs will be used (kind of like the Accepts header in HTTP). Just like images, higher quality codecs tend to use more bandwidth (though like images its much more nuanced than that). Similarly like image formats there's different amounts of loss in the encoding/decoding of the voice data.

Voice has a major difference though. Its generally a real time protocol (its data packets are called RTP and sent over UDP generally). With images you can afford to wait longer for high quality images to load over a slow network. In a phone call that delay gets a bit annoying (to put it lightly).

Most people would rather trade quality for timeliness. You ISP would rather throttle your bandwidth and may not care about your quality (to a point).

Its quite a bit more complicated than that, but in a nutshell with VoIP, "higher quality" = "more bandwidth". If you don't have the bandwidth you'll have a very hard time getting better quality. If you have the bandwidth locally, then your provider or some network link along the way either doesn't have the aggregate capacity or may be actively throttling your rate to limit usage.

It really depends on which codec you're using. We had an implementation of G.711, which is not bad, but far from great. However, the license costs for implementing G.729 were so outrageous that my previous company simply couldn't afford it (or wouldn't). G.729 offers better call quality for the same bitrate, or similar call quality for much lower bitrate. Typically, G.711 would use ~64Kbps, whereas G.729 can achieve the same call quality for ~8Kbps (there's also a variable encoding that scales between 8 and 32Kbps). This being said, G.729 also requires a lot more CPU processing power.

If I'm not mistaken, each party of each leg of a call needs a license, which means that for a typical call centre with an IVR server between the agent and the customer, you need:

Summary:

- PSTN converter to IVR: 2 licenses / per customer

- IVR to agent: 2 licenses / per agent

- IVR to supervisor: 2 licenses / per supervisor.

It's pretty normal to see 300-agent call centres, and you typically want at least 300-900 calls waiting. Let's say there's between 10 and 30 supervisors.

300 * 2 + 900 * 2 + 30 * 2 = 2460 licenses. At $6-10 per license, that's more expensive than most hardware PBX solutions alone!

For IVR companies that handle multiple hundreds of thousands of calls per day, it's not really worth it.

Edit: Formatting

That licensing scheme is INSANE, especially since codecs like Opus exist, and do a much better job.

Once you touch the PSTN though, you're uLaw anyway. Why would you pay that much just for in-network voice quality?

Yes, codec licensing costs are one of the reasons people choose to not use the better codecs. Also, yes there is a CPU trade off as well.

The decision is more complicate with many trade offs. But in general, the OPs issues are either fundamental network ones or an intentional choice of one of his providers (typically for cost reasons - manifested in some manner).

Old fashioned PSTN lines would secure the full bandwidth needed end-to-end when the original call was established (which was really in efficient from a data utilization perspective).

With VoIP you are:

1. Beholden to the bandwidth/latency issues of the crappiest link in your end-to-end path.

2. Intentional use of poorer quality codecs (by your ISP and/or VoIP software to reduce aggregate bandwidth use). Ironically this can be inverted, there are codecs that sound much better than PSTN calls (but use more bandwidth).

> 2. Intentional use of poorer quality codecs (by your ISP and/or VoIP software to reduce aggregate bandwidth use). Ironically this can be inverted, there are codecs that sound much better than PSTN calls (but use more bandwidth).

I mean, people use G.729 for example because, yes, it is more efficient than G.711. But ultimately, if you want to communicate with the PSTN those are your only two choices and it's easier to just say "G.729 for everything" instead of wasting processing time on the server re-encoding everything that goes off the network.

You want better codecs? Bitch at the phone companies who have drug their feet on implementing G.722 while more and more IP phones and mobile devices continue to add support for wideband audio.

Both ends need to support wideband codecs, as does your phone (hard or soft phone). This includes the VOIP providers needing to be able to use wideband codecs between each other.

If you achieve a wideband setup the call quality is amazing. I've used it for years. It really sucks when you have to call someone on a legacy POTS line or using a narrow band codec (bad voip, cell phones, etc)

On that note it's unfortunate that cell phones which can do wideband codecs within the same carrier (HD Voice!) won't work if your call leaves their network.

> On that note it's unfortunate that cell phones which can do wideband codecs within the same carrier (HD Voice!) won't work if your call leaves their network.

This is quite annoying, but it's because the PSTN overall is stuck on G.711 and G.729 - my SIP provider doesn't support anything else because they would be forced to re-encode the signal anyway and lose quality doing so.

Ultimately, we need to be putting pressure on the telco's to enable support for G.722 - until that happens we're going to be stuck with 4KHz audio for a long, long time.

Little trolling: so you say you have 1.6Gbps connection at home?

Maybe your connection has some latency issues?

Yes sorry I'm a dufus about these things. The issue has been thoroughly gone into down below.

Mea culpa.

Lots of good answers, but there is another one that's important too: echo cancellation. Most telco equipment has very good echo cancellation hardware in it. It's not impossible to do a decent job in VOIP, but not all of the software does it well. This can be very important when you have high latencies or if you are doing conference calls.

The other main overlooked problem is automatic gain control in software. This one drives me crazy. Most people are too lazy/clueless to set the gain properly on their microphone, so VOIP client makers tend to use automatic gain control to adjust the input. However, it works very differently that what you are used to in a telephone. If someone laughs loudly, or coughs you end up losing the beginning of the next sentence. It can work extremely badly in conjunction with echo cancellation depending on the software -- to the point where the audio can be very difficult to understand.

It's probably my biggest gripe with Google Hangouts because there is no consistent way to shut of AGC on all platforms.

I've written VOIP clients before and it is possible to have incredibly good audio (much better than normal PSTN -- mainly because the codecs are better). But it's very tricky to get it to work on random hardware, with random network conditions and with users who don't want to configure their client optimally.

Got any link to good echo cancellation hardware? Not that I'd every buy it, I'm just interested in tech (and businesses) that's not for average consumers.
I work at a big telco in the voice group and we still have tons of DMS 100/200/250s. Anyways some interconnections we have require Ditech echo canceller (no idea what model but apparently when we need them, we use spare ones or have to buy them on the used market). We have been migrating traffic to newer equipment and the new gateways have echo cancel built into them (e.g. Genband G9s)
I'm going to agree with you and take this opportunity to plug http://cleanfeed.net/ We run a free 'VOIP' system that does zero processing on your audio.

It's actually for broadcasters who just want to get live audio from A to B, untouched. But as it's two way (and multi-way) we're finding users contacting us to say they're using it for regular 'conferencing'.

I suppose we are also doing it ourselves; we used Cleanfeed to communicate when developing Cleanfeed.

Elsewhere in the thread people mention jitter etc. issues. In my experience, 200-300ms of latency has much less negative impact on rapport than aggressive audio processing.

Thanks for the recommendation!

FAQ #1 should be "how does this remain free/why is this not going to disappear"?

Thanks, good suggestion. It's actually on the front page that our free offering is a spin-off of something commercial. But a FAQ to clarify this particular point would be sensible.
Not all uses of VoIP are done with software clients, mind you.

I manage SIP setups for myself at home as well as at my aunts shop, both of which have physical IP phones instead of softphones. While softphones are certainly useful, most people on a VOIP service of some kind are probably more likely to still use a physical handset, headset, or proper conference phone where echo cancellation is less of an issue since it's either handled by the hardware itself or non-existent in the case of users on a headset.

We have several VoIP lines on several locations. The quality is actually great. If you want guaranteed quality, use a dedicated voice VLAN or some sort of QoS.

Even in places where we don't have this in place, it seems to be working just fine for us. I can also say that I've used Skype to a PSTN in a separate continent over a slow mobile link and it also worked.

However: I have to mention that the quality is never quite as good as normal Skype (so called "HD" quality).

I always thought that cellphone calls were routed over ip networks and were using exactly the same low-quality codecs "real" voip is using.
All quality problems would disappear if QoS would be implemented across the entire public internet. Your ears are so much more sensitive to small anomalies.

I've been deploying VoIP since 2003, both in the US and in Europe. In Europe it is easier to find cheap DSL with QoS applied, which allows laying in a separate circuit for VoIP only. In theses cases there are no complaints.

VoIP also works well on fiber. But as soon as one part of the network path travels the public internet all bets are off because there's no guarantee you packets aren't dropped because of traffic spikes.

Everybody who thinks the internet is stable in anyway is way off. Route flaps, packet loss, overloaded hops, last-mile problems happen all the time.

I could talk about this for hours.

Skype used to be fantastic when you could establish a point-to-point call. Today, I think everything gets routed through Microsoft so they can provide access to law enforcement. Some days it's good, often it's pretty bad.
I believe FaceTime is still point-to-point.

I've heard good things about WeChat and it's support for low-bandwitdth networks but haven't worked up the courage to try it out.

Probably rather because the p2p algorithm was not included when they bought Skype, they only licences it...
I'm surprised that no one has mentioned bufferbloat yet. In most home routers, people can see seconds of delay when there's significant other traffic on the link. (And remember, "other traffic" can be ordinary web browsing, where pages average 2 mbytes these days...)

The latency (and jitter) caused by bufferbloat has been a solved problem for almost five years: fq_codel, and the newer cake qdisc's are are in the Linux kernel, and people's home routers now could install LEDE/OpenWrt (https://lede-project.org). Two links:

What can I do about bufferbloat? https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a...

Test your network for bufferbloat: http://dslreports.com/speedtest

Bufferbloat is a side affect of congestion or microbursts, and not the primary cause of poor voice calls. Either packets get dropped (shallow buffer) or queued (enough buffer), and in both cases the quality of the call will suffer.

Mitigation: Don't congest. If you need to, mark VOIP traffic with a DSCP value that has queues that are serviced prior to all others (e.g., EF, DSCP46).

Applications do it already, but edge routers are adept at ignoring these flags or applying them only to their own service.

So that you use it and not competition.

Net neutrality would be the antidote.

You're correct, most applications do.

QoS/CoS services are best when clearly spelled out in service agreements. Providers may clear markings, but likely because either they don't want to have a free for all when it comes to their queues, or because they didn't purchase the gear that offers the rich queuing needed.

If those flags provide better quality of service, why am I not applying them to all of my traffic? Any attempt to "know" if my use is legitimate would undermine network neutrality.
Sure, you could make yourself crazy trying to prioritize all the traffic with varying kinds of flags.

Or you could just let an algorithm (SQM - fq_codel or cake) automatically determine which flows are sending more than their fair share of traffic into the bottleneck link, and offer backpressure to slow those applications down, so other applications work well.

re: net neutrality. That applies to whether my ISP (or any provider upstream in the Internet) should prioritize packets/data from various sources toward (or from) me. The answer should be, No. My contract with my ISP is that I get X mbps of data. The contract doesn't say the ISP will provide higher priority or data rate for service Y over service Z...

SQM has no effect on net neutrality: I'm simply configuring my own router to prioritize the data that I'm requesting (or sending). My ISP should handle them in a totally neutral way...

Yes. I'm a Republic Wireless (WiFi VoIP phone) customer. They mark their RTP packets with DSCP 48 (CS6 / ToS 192) and their SIP packets with DSCP 56 (CS7 / ToS 224). I limit ingress & egress bandwidth on my home router (a Mikrotik), and prioritize flows with those DSCPs above all other traffic.

The result is that even when my pipe is completely full, VoIP packet latency is practically unaffected. Compare to ~3 s latencies I would incur by Comcast's awful traffic shaping.

I have almost no bufferbloat when downloading (about 1ms) but about 300ms when uploading.

What does this mean?

I didn't look at that tool, but just reading this suggests that you're being shaped/policed more aggressively on egress (in the direction of your provider) than you are on ingress. Shaping and policing (typically) both use the same mechanisms as interface buffers to temporarily store packets in order to prevent congestion/discards. The distinction between them is that shaping actively buffers packets, where policing will more aggressively drop them (and aim to trigger a TCP congestion event) - in most cases, policing will involve a burst bucket that is somewhat similar to a shallow queue.

This would make sense if you have a higher bit rate from your provider for down vs up.

What speed connection do you have, and what speed connection did that tool report?

If your connection is fast enough that that tool is unable to max it out for whatever reason (say you're on a poor WiFi connection), you will experience no bufferbloat.

Yes, Don't Congest. But you can spend your life marking various kinds of traffic (many apps already do mark it, as someone down-thread mentioned), and pray that your router (and remote equipment) handles the markings properly.

Or just use SQM with Cake qdisc. It actually looks at the "sojourn time" for packets in various streams/flows of data and offers backpressure for sessions that are generating queues of data: all the other flows "go right through" without a lot of other settings.

Thanks for these! I have the Omnia Turris with Openwrt at home, which has everything needed to fix this. I'd also hope that problems playing music in our home network from the NAS when moving big files in the lan could be solved with these settings.
Anyone know if Mikrotik's RouterOS suffers from this?
Bufferbloat is typically a disease of your ISP (which throttles traffic), not of your router. A Mikrotik in its default configuration, like most other routers, will do nothing to address – nor exacerbate – bufferbloat caused by ISP throttling.

However, it is fairly simple to address ISP bufferbloat using RouterOS's "simple queues". In the most basic configuration, just add a simple queue targeting your WAN interface, with bandwidth limits in each direction slightly (~5-10%) lower than what your ISP throttles you to. If you have a slower connection (<10 Mbps), select "default-small" as the queue type. That's it, bufferbloat solved. (You can get fancier with multiple queues or different queue types, but the above alone gives you 90% of the benefit.)

Not so - Bufferbloat can occur everywhere there's a bottleneck link.

Your home router's link to the ISP is likely to be one, and most don't have any mitigation, and suffer from high latency. (And various ISP's equipment has bufferbloat for data coming toward your home.)

Some sites suggest that you knock your bandwidth down to 70% of the link speed to "leave some headroom" for other packets. That's fine if you want to give away 30% of your capacity.

SQM (implemented through fq_codel or cake qdisc's) takes off a few percentage points from the rated link speed, and achieves great results with minimal setup and configuration. Check out https://www.bufferbloat.net/projects/bloat/wiki/What_to_do_a... for more info.

That is… exactly what I said? Bufferbloat occurs where there's congestion. That's generally on the ISP's router, since they're the ones throttling your connection to 10 Mbps or whatever.

Yes, if you have a gigabit fiber connection and your home router can only handle 50 Mbps, you can get bufferbloat on your home router. But most people aren't in that scenario.

SQM is great, but not available on RouterOS, which is what the GP was asking about.

Yes, but it can occur on both ends of the bottleneck. Your ISP's equipment has a bottleneck link toward your home, and can have bloated buffers on that side. But if your router doesn't handle the uplink well, it'll be bloated as well.

Fortunately, SQM is available (from LEDE) for many MikroTik routers, so there may be a way out for the GP.

That's not how the Internet works. If your ISP is throttling, both upstream and downstream traffic is buffered on its router.

This is a consequence of how congestion is signaled in IP networks: by dropping packets. If the ISP is the bottleneck (it almost always is, due to throttling in both directions), it signals this to upstream equipment (i.e. the source of the packets) by dropping packets. They aren't sent back to some random router to be queued up.

Bufferbloat comes in because the ISP is trying to make use experience a little less crappy by hanging on to bursts of packets which would otherwise be dropped, with the intent of forwarding them on at the throttled rate. The consumer's router does not do this because it has no way of knowing this is happening. (But see below.) This is fine, except ISPs make their buffers HUGE.

There are two ways around this: one is, as you suggested, to use fq_codel, which monitors flows to infer packet buffering or loss at the ISP. So then, it is able to avoid sending packets which would be buffered or dropped by the ISP. The other is, as I suggested, to make your router the one dropping and buffering packets, which it normally does NOT do. It has no reason to: the links to the modem and the client are typically much faster than what the ISP throttles to, and those links are all it can see. (Though it's not unlikely that the user has subscribed to a higher speed than their wireless can handle, in which case inbound traffic is queued on the router, NOT outbound traffic.)

Yes, if your router is underpowered and you have a high-speed connection through the ISP, then your router could become a bottleneck and show bufferbloat. That is almost never the case, and when it is, it can happen in both directions.

The GP has "a way out", it is exactly to perform the steps I indicated. fq-codel is nice to have but NOT necessary.

I'm sorry, but I have to disagree. The laws of physics (of the internet) dictate that the piece of equipment at a bottleneck is responsible for handling bloat/congestion on its own end. Your ISP's head end/DSLAM/etc. has an opportunity to prevent download bloat (with data being sent to the home), but it has no way to know what's happening on the consumer end of the link.

Your router at home needs to have the same logic: it needs to control the buffering of data being sent toward the ISP. fq_codel/cake actually measures the time packets dwell in the queues (their "sojourn time") and drops/marks with ECN some of the packets of flows that build up a queue.

A cool feature of fq_codel/cake qdisc in LEDE is that it can control bloat in both directions. It imposes a bottleneck that's slightly (a few percent) below the actual ISP download link speed so that traffic queues up within your home router (instead of at the DSLAM/head end). Consequently, it can do the fq_codel/cake algorithm on the download (as well as the upload) direction, keeping your link unbloated at a very small loss of link speed.

You seem to be working under the fundamental misunderstanding that the link is the bottleneck. IT IS NOT. The ISP's router is the bottleneck, because it typically throttles the connection.

For example, my home network has a 100 Mbit bidirectional Ethernet connection between my router and my modem, and my modem has a 152 Mbps downlink (4 bonded DOCSIS channels) and an 81 Mbps uplink (3 bonded DOCSIS channels).

Neither of those is the bottleneck, because my ISP throttles my connection entirely on their router to 3.5 Mbps downstream, 1 Mbps upstream. I, like most in the US, cannot afford a 100+ Mbps home connection.

What you said would hold true IF, say, I had a gigabit connection (but still used a 100 Mbps Ethernet link). Then, yes, upstream traffic would build up in the upstream queue on my own router or modem. But that's not the case, and neither my modem nor router see anywhere near the 100+ Mbps of traffic needed to saturate the link.

If you do not understand this, I suggest looking at the queue depth on an actual router, with all traffic shaping turned off, while running a speedtest. You will see the interface queues in both directions be completely empty.

Yes, but... A "bottleneck" occurs wherever there is a high speed link going to a lower speed link. In your case, the 100 Mbps Ethernets in your home feed through the router to the 1 Mbps upstream your ISP provides. That's another bottleneck.

At this point in a conversation, I always recommend people measure their actual network, to see if they're happy with the situation. If it's good, then everyone's happy.

What results to you get from the DSLReports Speed Test (www.dslreports.com/speedtest)? It measures latency (lag) during the download and upload parts of the test, and will show if your router (or your ISP's router) is buffering too much data, and giving you undesired latency. Best regards.

The 100 Mbps is from the router to the modem. There is no such thing as a 1 Mbps "link" to an ISP. That bottleneck is imposed by traffic shaping on the ISP's router and nowhere else. (An exception would be something like ADSL, where the uplink truly is the bottleneck. But even then, the queue builds up in the modem, NOT the router!)

Here's a picture, since you seem to be ignoring my words:

     150 Mbps WiFi
           |
      home router
           |
    100 Mbps Ethernet
           |
         modem
           |
    152/81 Mbps DOCSIS
           |
    +-ISP router-----------------+
    |      |  <-- QUEUE          |
    | 3.5/1 Mbps traffic shaping |
    |      |  <-- QUEUE          |
    +------|---------------------+
           |
     10 Gbps fiber (or whatever)
Or, in the case of ADSL:

     150 Mbps WiFi
           |
      home router
           |
    100 Mbps Ethernet
           |
         modem
           |  <-- QUEUE
     8/1 Mbps ADSL
           |
    +-ISP router-----------------+
    |      |                     |
    | 3.5/1 Mbps traffic shaping |
    |      |  <-- QUEUE          |
    +------|---------------------+
           |
     10 Gbps fiber (or whatever)
In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.
> In either case, neither queue builds up in the home router -- that is exactly the problem, since we can't control the queue size in the modem or ISP! But by traffic shaping to 3.25/0.85 Mbps in the home router (either manually or automatically with fq-codel), we force the queues to build up there, thus giving us control.

YES! I understand what you're saying. (I had been treating the cable/DSL links and their terminating equipment as a black box; your description was being far more specific. Thanks for sticking with me to make yourself clear.)

And as you point out, the fix in all cases (at least, until our ISPs get serious about solving bufferbloat) is to force the home router to take control of the buffering by shaping the traffic a bit below the actual link speed. Thanks.

Not in Germany, kernels and routers need! To be locked by law.

I accidentally updated without know. Now I'm locked...

citation needed. If that were true, quite a few router models available would be illegal, so it probably isn't.

(TP-link started locking down at least some models, in response to FCC rules, and some manufacturers don't allow you to downgrade, but there is no legal requirement to do so and often ways around it. Other manufacturers like Linksys openly advertise their routers for use with OpenWRT)

Oh boy, direct down vote instead of looking on your own or asking...

http://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELE...

http://www.tp-link.de/download/Archer-C7_V2.html#Firmware 1.For Archer C7(EU)V2. 2.The EU firmware was specialized for CE certification and can't be downgraded to other version, please click here for choosing your region and selecting the most suitable firmware version to upgrade. 3.Old firmware's configuration file can be imported into this new firmware; 4.Your device's configuration will be lost after upgrading, which means you need to configure your device again

All routers are doing the same. EU law.

Sorry, was a bit to quick and I misread your post as describing the current situation, where this not (yet) applies.

And last I've heard about it they extended the validity of the old rules for undefined time (due to most of the standards not being ready), but I can't find a good source for it now, so I might have gotten that wrong. My mistake for not checking better.

> All routers are doing the same. EU Law.

Citation still needed.

a) I reviewed your eur-lex.europa.eu link, and didn't find anything that prohibits installing other firmware. Please cite the chapter/article that's relevant.

b) re: TP-Link firmware. It is true that TP-Link has added a technical means that makes it impossible to install certain other TP-Link firmware (the "downgrade" they mention). However, there are other firmware distributions (LEDE/OpenWrt/DD-WRT/others) that have the capability to install over the TP-Link factory firmware.

This is almost always a latency issue, and because nobody has figured out a way to milk extra cash for providing inter-network packet prioritization. In other words, this isn't an engineering problem, it's a money problem.
I've been in the VoIP industry for over a decade now, and in almost all cases of 'bad VoIP' I can trace the blame to a bad implementation of it. Let me give you an extreme examples...

- It's fairly common practice in countries with poorly enforced telco law for people to resell access to the PSTN. They will connect to the PSTN (either via a GSM gateway or a server with an E1 card) and resell that access to a 'wholesale VoIP business'. - That 'wholesale VoIP business' will have an infrastructure, cobbled together with open source technologies with no ability to scale, in the operational or technical sense. - The same 'wholesale VoIP business' will undercut the cost to access the countries PSTN (compared to accessing it directly via a tier 1 carrier) and sell the access to other tier 1 carriers. - The other tier 1 carriers lap up this cheap access, an embrace it because international calls fulfil a small portion of their multi-million revenues. - Therefore, when someone in another country tries to call that country with poorly enforced telco law, it will route via the resold route until it's shut down by the law or it breaks because it's poor infrastructure. - This is currently happening on a big scale in lots of African countries, some central american countries and in the middle east.

I'll leave the context of 'office to office' comms to the other answers as it's been covered in detail elsewhere. But I will say VoIP, all it's forms, is an old technologies, and problems relating to poor bandwidth.. voice quality.. backwards compatibility have been solved for many years.

It depends. I've noticed that sometimes my AT&T cell calls are of a higher quality, depending on who I'm talking to. I think this has something to do with their HD Voice product.

I like it, the quality reminds me of what I've experienced in the past with the Cisco UC platform on a well-provisioned network and UC platform.

The goal of Skype, Viber, etc. are to always give you some level of audio across all networks, so the codecs they are using have a different goal than Cisco UC.

All these codecs have low bandwidth requirements, much lower than what you have. If you have a residential ISP, QoS might be an issue with some of the higher sampling rate codecs, but you probably won't notice them with Skype.

Try running an open source VoIP MCU at home on your LAN to see how good it could be.

I think your complaint is about VoIP over the internet. And that's simple, the internet is best effort. Packets get dropped and that's expected. Also, packets can take a long time to reach their destination and there are no guarantees for jitter. This absolutely sucks for voice quality and there's nothing anyone can do about it.
Your landline is now VOIP too, it may be twisted pair to the central office, but then it's digitized and transmitted on the carriers owned networks.
Then how does Google Fi work? I don't know if it's VoIP but it is phone calls over the internet and I've never had a problem with call quality.
First, as others pointed out, you will need a decent internet connection, with minimized packet loss and jitter. Most metro area data connections and quality WiFi access points will be sufficient. However, the wifi router that comes from your provider and stuffed away on the other side of the house might not work very well.

Secondly, good codecs make a world of difference. This is where I'll add the disclaimer that I'm in the collab side at Cisco. But I have the opposite experience using systems on my phone, my mac and with big telepresence units: the quality in all 3 settings far surpasses what's on PSTN, and if you're using good speakers or headsets, you can really appreciate the good quality.

In your case, it sounds like you might benefit from a better network access point at home.

Shameless plug #1: check out Cisco Spark. We still have some rough edges but I think the call quality is what you're looking for. I'm able to carry on a call and switch from wifi to LTE without dropping, or turn on VPN in the middle of a call and just get a few seconds pause before it recovers.

https://www.ciscospark.com/

Shameless plug #2: check out the new Sparkboard. It's got 12 microphones and does beamforming in order to filter our room noise away from whoever is speaking.

http://www.cisco.com/c/en/us/products/collateral/collaborati...

Does it automatically report your political and religious affiliation to chinese ministry of public security? Lol, zing.
It's end-to-end encryption. For large sites, customer owns the encryption keys on their hardware so we can't intercept/decrypt even if we're ordered to.
I've tried using spark but unfortunately video calls don't seem to work at all when I'm on my corporate VPN (which is Cisco, btw). I really want to use it because it otherwise works on Linux (compared to webex...) but I need to be able to use it on the VPN.
> Secondly, good codecs make a world of difference [...] quality in all 3 settings far surpasses what's on PSTN [...]

To be fair, this is probably true of any decent VoIP solution on the market. The problem is always that moment you have to connect to the PSTN and you're forced into a choice of G.711 or G.729.