Streaming conversational AI

A conversation, not a sequence of requests

I was Technical Lead for a consultancy prototype that used conversational AI to complete a structured booking over an ordinary phone call.

The central challenge was not simply connecting speech recognition to an LLM. A phone conversation is continuous: audio arrives while other work is still happening, callers pause or change direction, and a response that arrives too late feels broken even when its content is correct. The entire path needed to behave as one live system.

From Python proof of concept to streaming .NET

I first proved the conversational approach in Python using an early version of LangChain. At that point, many of the asynchronous paths needed for genuine end-to-end streaming had not yet been implemented. The prototype established the behaviour we wanted, but also exposed the need for an architecture designed around continuous asynchronous flow.

I then led and built the C#/.NET implementation using async/await throughout. Twilio routed the phone call to a WebSocket endpoint in a .NET API. Audio moved through a pluggable speech-to-text engine, into a Semantic Kernel conversation layer, then through a pluggable text-to-speech engine and back to the caller over the live connection.

My formal background in music technology and audio programming helped here. I already understood digital audio as a time-based stream of samples, so treating the call as continuous data—with timing and interruption constraints—was familiar rather than an entirely new media problem.

Each stage could process and forward work without waiting for the entire conversation turn to finish. That reduced avoidable latency and made it possible to react to what was happening while the call was still in progress.

Designing for conversational timing

Real conversations contain awkward gaps and interruptions that a request-response demo can ignore. This system could:

• Insert a short spoken acknowledgement when model processing risked leaving a long silence.
• Recognise extended pauses from the caller and repeat or clarify a question when appropriate.
• Stop or redirect generated speech when the caller interrupted.
• Send new text into the speech pipeline at any point in the conversation.

Those behaviours made timing and cancellation part of the product design rather than implementation details. The system had to coordinate incoming audio, recognition results, model work, generated text, outgoing speech, and cancellation without letting one slow stage block everything else.

Agent orchestration before standard tool calling

This work predated today's mature agent frameworks, standardised tool calling, and MCP. I built an explicit information-gathering loop to steer the conversation through the details required for a booking.

The model extracted structured information such as time, locations, and caller details from ordinary speech. A caller could provide one answer at a time or several details in a single sentence. The orchestration tracked what was already known, identified what was still missing, selected the next useful question, and continued until the request was complete.

Modern agent tools would make that flow more conventional to implement, but the underlying engineering problem was the same: give a language model controlled access to application state and use its output inside a deterministic business process.

Comparing providers without rebuilding the system

Speech recognition and synthesis providers differed in latency, accuracy, voice quality, and streaming behaviour. I designed both sides behind replaceable components so the team could compare services without rewriting the call flow.

That separation also let us trial a custom Azure text-to-speech voice trained from a short recording session. The result was strikingly natural and demonstrated how much the perceived quality of a conversational system depends on the complete speech pipeline, not only the language model.

Building for the QA team

Real-time faults can be difficult to reproduce because recognition, model work, speech generation, timing, and cancellation all interact. Without being asked, I built a dedicated log viewer that gave the QA team a practical way to inspect what the system was doing while they exercised different call scenarios.

That decision came directly from my early career in testing. I still think about how engineers can help QA understand a system and report useful evidence, rather than treating testability as somebody else's concern after development is complete.

The result

The prototype demonstrated an end-to-end, interruptible AI phone conversation at a point when the supporting libraries and agent abstractions were still immature. It combined real-time media handling, asynchronous .NET engineering, conversational UX, provider evaluation, and controlled LLM orchestration in one working system.

Why I include this work

• Technical leadership combined with hands-on delivery of the difficult real-time path.
• End-to-end asynchronous and streaming design in C#/.NET.
• Practical understanding of speech recognition, synthesis, latency, interruption, and conversation state.
• Building controlled agent-like workflows before standard tool-calling abstractions existed.
• Designing provider boundaries so rapidly changing AI services could be evaluated and replaced.
• Proactively improving observability and testability for the QA team.