BMJ Innovations 2020;6:13-17. 12 March 2020
- This study was performed to assess the performance of three popular ‘artificial intelligence’ symptom checkers currently available to the public (Babylon (London), Your.md (London) and Ada (Ada Health GmbH, Munich) and evaluate their accuracy as a screening tool for head and neck cancer symptoms.
Aim The technology of online and mobile-based symptom checkers is developing on a background of unprecedented demand for General Practioner (GP) and medical services. This study was performed to assess the performance of popular ‘artificial intelligence’ symptom checkers currently available to the public and evaluate their accuracy as a screening tool for head and neck cancer symptoms.
Method We selected three of the most popular and previously compared mobile application (app) symptom checkers available including Babylon (London), Your.md (London) and Ada (Ada Health GmbH, Munich). A recent Pan-London Suspected Cancer Referral Guide was obtained and distilled into a list of qualifying symptoms for referral to secondary care on a 2-week wait cancer pathway. A generic patient background was devised and each symptom was passed through the selected symptom checker apps. Differential diagnoses and recommended triage times were compared.
Results Overall a cancer diagnosis was suggested possible in the differential diagnosis in 33% of symptoms. As a screening tool, Babylon was the most accurate for including a potential cancer diagnosis in the differential diagnosis, with a sensitivity of 45% cases, Ada with 32% and Your.md 23%. Ada attempted the most diagnoses, able to provide a differential diagnosis in 95% scenarios, but was also the most risk averse triage assessor, with 55% outcomes recommending medical advice immediately or within hours.
Conclusion The ability of modern symptom checkers to promote health awareness and self-checks is promising, but we have identified a number of areas of potential improvement. At present the sensitivity of the tested symptom checker apps remains low for head and neck cancer.
