4. QoS Testing

Revised 6/25/2005

NOTE: The original test methodology used TestYourVoip.com's TestMyVoIP feature. This is browser-based Java applet that tests both send and receive paths between the computer the applet runs on and a selection of test endpoints located around the world.

Shortly after the review was published, Sipura pointed out that this was not a correct test of the SPA-2100's QoS features because it used data and simulated VoIP traffic that both originated from the product's LAN port. Sipura noted that the 2100 will only prioritize the VoIP traffic originating on the SPA-2100 FXS (phone / fax) ports.

The test described below uses a corrected, valid method using data originating from the LAN port and actual voice from a telephone connected to an FXS port. I apologize to Sipura and our readers for the error.

One of the reasons I wanted to test the 2100 instead of one of Sipura's non-routing models was to check out its QoS features. I used the "Golden Phone" feature of Brix Networks' TestYourVoip.com as my test tool. The "Golden Phone" allows you to make an actual VoIP call to a test point (I used the one in Boston), talk for about 15 seconds and get a MOS (Mean Opinion Score) number for the voice quality measured from your phone to the test point. The test results returned also include packet loss, discards, latency and Codec-related errors.

I ran four "Golden Phone" tests to see how the 2100's QoS capability fared:

with no other apps running on my computer and QoS disabled with a long FTP upload running and QoS disabled with a long FTP upload running and QoS set to TBF (Token Bucket Filter) with a long FTP upload running and QoS set to CBQ (Class Based Queueing)

	No apps/ no QoS	FTP up / no Qos	FTP up / TBF QoS	FTP up / CBQ QoS
MOS	4.4 / 5.0 (Best with G.711 is 4.4)   Degradation Sources 0.60 100.0% 0.00 0.0% Packet Discards 0.00 0.0% Packet Loss 0.00 0.0%	3.4 / 5.0 (Best with G.711 is 4.4)   Degradation Sources 0.60 37.5% 0.67 41.7% Packet Discards 0.33 20.8% Packet Loss 0.00 0.0%	4.3 / 5.0 (Best with G.711 is 4.4)   Degradation Sources 0.60 85.7% 0.10 14.3% Packet Discards 0.00 0.0% Packet Loss 0.00 0.0%	3.7 / 5.0 (Best with G.711 is 4.4)   Degradation Sources 0.60 46.2% 0.00 0.0% Packet Discards 0.70 53.8% Packet Loss 0.00 0.0%
Codec	G.711 (PCM at 64kbps,  20ms RTP payload, 80kbps IP BW)	G.711 (PCM at 64kbps,  20ms RTP payload, 80kbps IP BW)	G.711 (PCM at 64kbps,  20ms RTP payload, 80kbps IP BW)	G.711 (PCM at 64kbps,  20ms RTP payload, 80kbps IP BW)
Round-Trip Latency	11 ms	11 ms	11 ms	11 ms
Packet Discards	0.0%	1.7%	0.0%	2.1%
Packet Loss	0.0%	0.0%	0.0%	0.0%
Loss Periods	Min: 0 ms Avg: 0 ms Max: 0 ms No Loss	Min: 0 ms Avg: 0 ms Max: 0 ms No Loss	Min: 0 ms Avg: 0 ms Max: 0 ms No Loss	Min: 0 ms Avg: 0 ms Max: 0 ms No Loss
Jitter	Min: 0 ms Avg: 1 ms Max: 2 ms	Min: 0 ms Avg: 27 ms Max: 166 ms	Min: 4 ms Avg: 27 ms Max: 160 ms	Min: 0 ms Avg: 3 ms Max: 58 ms
Table 2: QoS Test Results

Table 2 summarizes the results from all four tests.

You can see that with the FTP upload, latency, packet discards and jitter all increase. Enabling TBF-based QoS (Sipura's recommendation), essentially eliminates packet loss, but doesn't do much for jitter. On the other hand, switching to CBQ-based QoS appears to increase packet discards, but significantly lower jitter.

To put all the above results into perspective, note that the 3.4 MOS obtained with no QoS and FTP running would result in noticeable degradation and a "Many users dissatisfied" rating according to the table toward the bottom of this page. That table also points out that the 4.3 score obtained with TBF-based QoS enabled falls into the "Very satisfied" classification and is 0.1 point shy of the 4.4 perfect score possible when using a G.711 codec.

The bottom line is that it looks like the recommended TBF-based QoS did the trick of restoring near-perfect call quality with simultaneous continuous uplink data traffic.