Innov8r: how QVCCS tests Genesys Cloud CX numbers, IVRs and DTMF from the caller's side

Inside Genesys Cloud CX, a great deal can be checked before a single call is made: call routes, schedules, Architect flows, queues and skills. But a customer never sees any of that. They see a number, and what happens in the first few seconds after they dial it. A number that reaches the wrong flow, a closed-hours message that plays at lunchtime, or a menu that never hears a key press is invisible from inside the platform until somebody calls in from outside, the way a customer does.

So we test from the caller's side, and we do it for every number. On global deployments that can mean hundreds of service numbers and direct dial-in numbers across many countries, each with open-hours and closed-hours behaviour, menus and self-service paths to prove before go-live and again after every significant change. That work has to be fast, repeatable and evidenced, because a test that cannot be shown to have passed has not really been done.

We looked hard for a SIP softphone for the Mac that was robust on both Intel and Apple silicon, and could not find one that suited this work. The options we tried were sold per user, light on the diagnostics an engineer needs, and often tied to one provider's PBX or VoIP service. We wanted a phone that was independent of any provider and rich in the detail that matters when testing: one that makes IVR and customer experience testing of complex Genesys Cloud CX solutions quick and repeatable.

So the QVCCS team built one. Innov8r implements SIP signalling, RTP audio, RTCP reporting and DTMF itself, written from scratch, with no third-party SIP library and no browser telephony layer in between. On the network it behaves like a hardware IP phone, which is exactly what makes it a faithful test caller. It runs natively on Apple silicon and Intel Macs from a single build, and it has grown release by release as our engineers asked more of it on real projects.

A test call starts on an engineer's Mac. Innov8r registers to the QVCCS VoIP server, our own PBX, which sends the call out over a carrier SIP trunk and across the public telephone network to the Genesys number under test. From there the call takes exactly the road a customer's call would: it is offered to Genesys Cloud's media services, hosted on AWS in the org's region, and matched to the org by the number dialled, whether that is a geographic, national, non-geographic or toll-free number.

Inside the org, the call route for that number starts its inbound call flow. The flow plays its announcements, reads the caller's key presses and makes its decisions. Some calls end there, handled completely in self-service by the IVR, a bot flow or a Genesys virtual agent. Others pass to an ACD queue and on to an agent. Because the test call travelled the whole road, every hop along it is proved, from the carrier to the agent's headset, rather than assumed.

Typing hundreds of numbers by hand is slow and error prone, so Innov8r reads them from the org itself. Its Genesys tab connects with an OAuth client using the client credentials grant, which works without a user signing in, and makes read requests only. It fetches the org's call routes and inbound call flows and lists every number beside the route it belongs to and the flows it reaches. Engineers filter the list, click a number to dial it, or save it to speed dial for a regression run.

Each org an engineer works with is remembered as a named profile, and its client secret is held in the macOS Keychain rather than in the app's own storage, so switching between orgs takes a moment. Beside the dial pad, the same screen shows a millisecond-stamped SIP trace, a live spectrogram of both directions of audio and the call quality gauges, so the evidence of each test is captured while the call is still in progress.

Key presses are the classic example. Innov8r sends DTMF as RFC 4733 telephone events in the RTP stream, the same mechanism a hardware IP phone uses, and falls back to SIP INFO only when the other end does not offer telephone events. Getting this right end to end matters: when the method agreed on one leg of a call does not match the next, an IVR can sit waiting for digits it never receives. Testing with a phone that behaves like a real handset shows the problem before customers do.

Many failures happen before a call is even answered, so Innov8r plays early media: the ringing, announcements and messages a network or flow sends before the call connects. Its audio path is built for imperfect connections too. An adaptive jitter buffer grows when packets arrive unevenly and shrinks again once the connection settles, and short gaps from lost packets are concealed, so the engineer hears what a caller on ordinary broadband would hear, and can tell a network problem from a flow problem.

Quality is measured rather than guessed. During every call Innov8r estimates a Mean Opinion Score and R-factor using the ITU-T G.107 E-model, updated ten times a second, alongside jitter, packet loss and throughput. When a test call sounds poor, the gauges show whether the cause is jitter, loss or something else, and the SIP trace shows exactly what each side said and when. That turns "it sounded a bit choppy" into evidence an engineer can act on.

Writing a SIP stack from scratch means owning every fault in it, and Innov8r's release history reads like a set of small investigations. One of the hardest was one-way audio that appeared about four minutes into a call. The cause was the phone's routine re-registration with its server: a successful registration reset the phone's internal state while a call was still live, and a safety check then stopped the outbound audio. The fix makes re-registration leave a live call alone, and the safety check now needs two consecutive mismatches before it acts.

Another was calls that dropped for no visible reason. A server can keep retransmitting responses for a call the phone has already cancelled for up to 32 seconds, and a late one could be applied to the next call and end it. Every response is now matched to its own call before it is acted on, and a stray answer for a call that no longer exists is acknowledged and closed at once, so the server releases its channel in milliseconds instead of holding it open.

Each fix is written up with its symptom, its root cause and how to recognise it in the SIP trace, so the next engineer to see a similar pattern on a client's platform knows what to look for. The same care runs through the rest of the app: secrets held in the macOS Keychain, no call data written to disk, recovery with an immediate re-registration after a network change or wake from sleep, and versioned releases that each say exactly what changed and why.

Innov8r is an internal QVCCS tool. It is used only by QVCCS employees, is not offered for sale or licensing, and is not a Genesys product. It exists because our engineers wanted a better way to prove the work they deliver, and because building it taught us a great deal about how calls really behave between a carrier, a network and Genesys Cloud CX. That understanding now goes into every design, test plan and fault investigation we take on.

We share how it works because the ideas travel, with or without a phone like ours. Keep a register of every number with the call route and flows it should reach. Test each one from outside, over the public network, before go-live and after every significant change. Agree the DTMF method end to end, from carrier to trunk to flow. Measure the quality of every test call and keep the SIP evidence. If that sparks an idea for your own platform, it has done its job.