AI models are applied to healthcare to provide diagnostic predictions. However, they are often considered black boxes, as there are no information about the underlying decision making process. Here I present a diagnostic tool that identifies gynecological tumors from ultrasound. The tool is built through the collaboration between AI and human experts. The tool delivers accurate predictions, and also shows the rationale behind each decision as an understandable output, for transparency, interpretability, and accountability. By providing comprehensible explanations, also patients can increase trust and engagement.