Self-hosted Whisper: Breakthrough in Medical Transcription

Optimization of Self-Hosted Whisper for German Medical Speech

A Challenge for Medical Transcription

In the field of artificial intelligence applied to medicine, a widely held belief is that cloud-hosted speech transcription services offer the highest level of accuracy. However, this perception is being challenged by recent advancements in self-hosted systems, particularly for transcribing medical consultations in German. A study has shown that these systems can not only compete with cloud solutions but also surpass them in terms of accuracy while ensuring data privacy. For example, AWS Transcribe, a cloud transcription service, achieves a recall of 0.82 for medical terms. In comparison, a self-hosted speech model, although generalist, achieves a recall of 0.75. But it is a layered self-hosted pipeline that stands out with a recall of 0.91, thus exceeding the cloud benchmark while keeping audio files under local control.

Context and Constraints of the Project

The goal of this project is to develop an ambient medical scribe for medical practices in Germany. This system must be capable of listening to consultations, transcribing them, and producing structured clinical notes as well as billing codes. Two major constraints have guided the development of this system: first, it must operate on-site, meaning that audio cannot leave the premises of the medical practice, thus excluding the use of cloud transcription services. Second, transcription errors in this context are critical, as they can have serious clinical consequences, such as misinterpreting a medication name. Therefore, it is crucial to measure accuracy on specific medical vocabulary rather than on the overall transcribed words. The challenge is significant: to match the accuracy of a cloud service like AWS Transcribe, which achieves a recall of 0.82 on medical terms in simulated consultations.

Rigorous Evaluation Methodology

To evaluate the performance of the models, the average word error rate is not sufficient. Indeed, a model may perform very well on conversational German while failing on medical terms, which are crucial. The evaluation thus focuses on the recall of medical terms. Two datasets were used for this evaluation: a synthetic capability map and a real-world anchor. The synthetic capability map includes 86 German medical terms, each rendered by four synthetic voices in various contexts, totaling 1,376 excerpts per model. Across four candidate models, 5,504 transcriptions were rated. The real-world anchor consists of five complete synthetic doctor-patient consultations, covering 27 key terms. This approach allows for verifying that the results obtained on synthetic data hold up in real-world contexts.

Choice of Base Model

The choice of the base model proved to be counterintuitive. Initially, a speech model specialized for German seemed to be the best choice for processing German medical audio. However, this model turned out to be the least effective and the hardest to improve. During testing, it showed a recall that was 8 to 10 points lower on medical vocabulary than expected. Moreover, during a medication-rich consultation, it produced about half the words compared to a general model, thus losing a significant part of the conversation. Two other architectures were tested but rejected: Gemma LLM, which failed on long audios, and NVIDIA Canary, which could not recover enough medication names. Ultimately, the general Whisper model was selected, as it improved the recall of the single model from 0.65 to 0.73, thus getting closer to the cloud benchmark.

Error Analysis and Adjustments

Transcription errors are not uniform, and their analysis is crucial for targeting improvements. Brand names and medication ingredients are the most problematic categories for each model. In contrast, diagnoses, lab values, and anatomy are generally well transcribed by competent models. A small number of terms, particularly anticoagulants and antidiabetics, elude all tested models, including the cloud service. This distribution of errors influenced the design of the final architecture, which uses the surrounding context to enhance the detection of critical medical terms.

Improvement Strategies and Results

To achieve a recall of 0.91, a layered architecture was implemented, integrating four deliberate design choices. The first choice, although counterintuitive, was to start with the Whisper model fine-tuned for German, which turned out to be the weakest starting point. However, by integrating additional layers and adjusting the model to better capture specific medical terms, the system was able to surpass the cloud benchmark. The use of surrounding contexts allowed for the recovery of terms that would otherwise have been missed, particularly in complex sentences or consultations rich in medical terms. This approach demonstrated that accuracy and privacy are not mutually exclusive, thus providing a viable solution for medical practices that wish to maintain control over their data while benefiting from accurate transcription.