Open Source SIP Testing (and more?)

A little background

Here at Reperio, we utilize the open source Kazoo platform as the basis for our realtime voice/video collaboration platform. We try to contribute everything we do back to the original code base. We are finding that testing this platform (since it’s not just a single codebase), both from our work and from upstream releases, is becoming more important now more than ever. The original author, and maintainer/steward of Kazoo, 2600hz, has built a testing framework called MakeBusy to accomplish this. However, it is written in PHP and very difficult to provide multiple, distributed, asynchronous actions and measurements that scale out.

We are strongly considering developing a new testing platform that can accomplish all of our goals with infrastructure and tools we’re most familiar with (and are also well adopted else wise). Another consideration is to build a platform that is not completely Kazoo centric (although our initial implementation will most certainly be).

What needs to be tested?

The needs for testing a complicated API based soft switch are much more than just sending SIP messages back and forth. We are most concerned with full functionality testing from a fully deployed system. Here are the many things we need to be able to test for:

Softswitch API (Crossbar)
- We need to be able to test all of the API functions required to deploy and manage the soft switch: from accounts (resellers), to devices, to registrations, to call flows, etc.
- It would be best to test end to end. In other words, be able to start with a blank slate and utilize the API to create an entire deployment with all if its data. This is going to be required to test things like device registration, call groups, presence, voicemail, etc. All of these platform specific concepts will have to be provisioned via API calls to ensure that everything is aligned.
Devices (SIP Phones)
- In the context of Kazoo, as well as most PBX’s or Softswitches, there is the concept of device registrations. This is the core of the “device” in the system. We need to be able to make sure that devices can register correctly.
- Once we have a registered device (UAC), we can then test the functionality of sending and receiving calls, calling to other devices, and calling to and from carriers. These tests will require sending/receiving media (audio). The best tests would include both RTCP (to measure call quality via packets and latency), as well as audio processing to determine audio quality directly with samples.
- Devices (SIP Phones) typically can have both presence (who is doing what) and other features like MWI (message wait indicator). These are just SIP messages typically, and we need to make sure they work.
Call Flows
- There are many different types of call flows in Kazoo. Technically, we would want to make sure to test every part of each module.
Conferences
- There is a extremely well developed and feature rich conference application.
Call Queues
- There is a extremely well developed and feature rich call queue application.
Faxing (T38)
- There is a extremely well developed and feature rich fax2email and email2fax application.
Many, many, other applications
- Some test cases will inherently test other parts of other applications, e.g. outbound/inbound calls to/from carriers.

Other testing considerations:

There are many other things that should be tested as well (not an exhaustive list, just to get an idea):

Load/Rate
- How many registrations per second?
- How many registrations in total?
- How many presence subscriptions?
- How many calls concurrently?
- How many calls per second?
Tenancy
- We should test in different scenarios of accounts (master, sub, reseller, etc).
Concurrency
- Run different tests concurrently
- Run different ratios of cps/and # of calls/messages per test
Video
- We want to start qualifying the video portions of Kazoo. Calls to/from devices, conferencing, etc.
WebRTC
- We need to be sure that WebRTC clients can also work
Internal Data/Processing
- Webhooks
- Websockets
- CDRs
- Some tests are needed for cross application interactions, e.g. caches. Here’s a real world example of a bug that was fixed.
Zones
- Kazoo allows for zones with concepts of federation of data between them. We should definitely be able to register in one zone, but send a call in another.

What exists now?

Because we’re an open source centric company, we’re only interested in tools/frameworks that can be released as open source software for everyone to use and contribute back to. There are quite a few open source tools available for testing SIP specifically:

MakeBusy
- This is a PHP + Freeswitch based testing framework. It accomplishes many of the goals set forth.
SIPp
- This is an open source (C/C++) based application that can be scripted to send SIP messages, as well as send/receive RTP
- Can be further integrated with sippy_cup.
- This is very SIP centric, and doesn’t really cover any of the other non-SIP cases.
VoIPER
- For SIP security testing

Unfortunately, none of those tools can meet all of our needs.

Logical Diagram - Proposed Test System (Single Zone Kazoo)

Core Test System Parts

Freeswitch
- Just like MakeBusy before it, utilizing Freeswitch as a core component will allow us to test almost all of the SIP + RTP cases. Specifically, we can test carrier and UAC (calls and SIP messages), as well as T38. We propose utilizing mod_amqp for events, allowing the Freeswitch implementation to be as simple as possible (also to be language/framework independent on the control side).
- Transcoding, DTMF, and RTCP are supported and measurable
- Supports video
Node.js
- This will allow us to utilize standard test infrastructure: chai, etc
- Asynchronous by default
- Promises allow for the ability to capture events when they occur and to coordinate states as they occur
- This is an internal skillset for us, and a well adopted technology globally
RabbitMQ
- AMQP will allow us to coordinate events from multiple Freeswitch instances asynchronously
- Freeswitch instances can be scaled out as needed (trivially)
- This is also a core part of Kazoo, so we’re already familiar with it
- Binding to Kazoo’s AMQP will allow for measuring kazoo specific AMQP events as well, utilizing the same software infrastructure

What doesn't exist?

We plan to begin the node.js codebase as soon as possible. We will likely use hapi.js for the REST/Websockets side of things (mostly because that’s what we use internally for everything else). There are implementations of WebRTC clients: jssip for example. However, this does not provide the RTP portion of the client. Either we will need to find/extend an existing NPM module for this, or write one ourselves for this use case.

Another goal would be to provision the Freeswitch instances via mod_xml_curl. This way we can provide a single or a few UAC’s for tests, and then re-provision with 1000’s if needed.