Providing access to structured information obtained from multi-modal data in patient records and their temporal evolution patterns enables clinicians to make better prognostic and treatment planning decisions.

Clinicians obtain, summarize, and correlate information snippets from various multimodal data sources in brain tumor patient records to synthesize a mental model of the patient status prior to making prognostic and treatment planning decisions. Temporal (co-)evolution patterns of various factors derived from clinical, laboratory, imaging, and genomics data sources are essential to the decision-making process. However, the current “piecemeal” use of data is very time-consuming and leaves the task of threading and correlating information entirely to the clinician.

We have built and tested a prototype system for analyzing and displaying patient images and text-based oncology reports of brain tumor patients accumulated over time. The text analysis includes the extraction of neurological symptoms, drug and dosage information, lab results, vital measurements, and assessments. This is coupled with the ability to scroll through 3D MRI images, plot and highlight edema volumes, and plot any recurring numerical value, such as weight, platelet count and blood pressure. Limited clinical evaluation has been performed to validate the usability of the prototype system in clinical decision making.

The system was initially designed and developed on the basis of several rounds of discussions between computer scientists, radiologists, and oncologists with the common goal of providing the most useful information on each screen. Then it was demonstrated to other members of the team. We conducted a cognitive evaluation with two residents (one in oncology and one in radiology) to determine the relative ease of use in obtaining, coordinating, and integrating patient information to make treatment decisions. The subjects were asked to look at combined text and image data for two patients and to answer several questions. Did the patient experience seizures at any time? When did the patient have the most edema? When did the patient have the lowest platelet count? The residents were asked, at the point of maximum edema, whether the patient had any other significant symptoms and whether the images indicated anything significant about the patient's prognosis. Finally, the residents were asked whether the drug Avastin effected the amount of edema. During the study, they were asked to think aloud and comment on the utility and usability of the system. At the conclusion of the study, they completed a brief 6 item Likert survey.

During the initial presentation to members of the team, several remarked that the ability to plot data, such as edema volume and drug dosage, led to immediate and unexpected insights. Both subjects were able to use the system effectively with minimal training. The experimenters provided assistance only when the subjects were stumped. The complete session with each subject was recorded, and notes were taken by the investigators. Although the participants were able to employ most system functions, they experienced a range of problems. The semantic mappings used to label buttons that subsume findings were not always intuitive. For example, both subjects expected that the “seizures” findings would be subsumed under the Neurology tab when, in fact, they were listed under “Symptoms” as it was in the oncology notes. Both subjects had some difficulty discerning the date of each time point they were examining, as that date was displayed just under the accordion control rather than closer to the actual time data point. If placed in such proximity, they would have overlapped each other.

Initally, the neuro-oncology resident experienced some difficulty selecting and juxtaposing images in the 2 panels. However, after a short period of time, she had no difficulty finding the edema information and was able, without any assistance, to make a comparative plot of edema volume and Avastin dosage. She was also able draw appropriate inferences about changes in the patient’s condition over time.

In summarizing the patient’s status in question 7, the resident readily grasped that the current status would be available by clicking on the last data point. She noted that the best part of the interface was the way it integrated views of text and image data, and noted it was most useful to be able to look at any pairs of images together. She remarked “Being able to determine your thrombocytopenia at that point and the edema volume feature is the best aspect of this whole thing.”

The radiology fellow was more interested in examining the images. He showed less interest in the symptoms reported as text in the accordion tabs. He readily selected among the FLAIR, T1C+, and T1C- images, and found them easy to study. In general, it took him more time and mouse clicks to find what he was looking for and longer to develop a basic mastery of the system. The system works by placing the image from the first thumbnail selected in the left image box and the second on the right. Then if you select a third image, that image replaces the first one. The system provided insufficient feedback and guidance, and he found it to be a source of confusion. He also noted that the “Lock” checkbox, which decouples one image from the other, worked exactly the opposite of the “Link” checkbox provided in the GE PACS system. The fellow was more conservative in drawing inferences from the images. In this system, the total edema volume was computed semi-automatically as described above. He commented, “Everything that this machine is calling edema is not necessarily relevant to the patient’s survival, because that could just be the radiation if in the radiation field. The part of FLAIR signal abnormality, which is in the field of the tumor, or on the tumor, may be more relevant, but once it becomes confluent, you can’t tell which one is the tumor and which one is the radiation change.” He scrutinized the images more carefully and commented on the resolution and potential noise that made clinical inferences less certain. He was also less sure about the value of integrating the text and image modalities, because his work was almost entirely with images. Despite his critical comments, he remarked that the system offered great potential as a clinical tool. His comments offered many excellent insights into the improvement of the interface.

On the Likert survey, both subjects rated the system very highly in terms of ease of use and learnability, and were especially appreciative of the ability to integrate disparate sources of data. They were less convinced that this would be an effective tool to discover novel dimensions of patients’ problems. Both residents agreed emphatically that this was a tool with great clinical potential.

We undertook this study to determine the utility of a multi-modal temporal display, based on both image and text analytics. Since this is preliminary work to lead into a multi-year study, much of the text analytics were carried out manually, and the image analytics carried out semi-automatically.[2] However, the web client-based system is a complete working system using Ajax, Dojo, the Derby database, and JavaServer pages. We tested the system on members of our committee, as well as on two medical fellows, and found that they were able to obtain patient history, image, and patient change information much more rapidly than when using previous systems. Further, the resident's enthusiasm for the prototype system made it clear that this is a system they would definitely want to use in clinical contexts. The ability to look at multi-modal patient profiles clearly leads to more rapid and effective decision making. It is significant to note that, while the two experimental subjects felt they could obtain all the same information without this system, albeit more slowly, the neuro-oncologist actually treating one of the patients found it provided information had not previously been obvious to him. The testing also provided considerable insight into improving semantic category mappings, clarity of temporal representations, and ease of access to various tools supported by the system.

We have developed a compelling prototype system for the temporal display of text and image information of brain tumor patients. Preliminary user studies confirm its value in determining patient treatment.

[1] Cooper JW, Ebadollahi S, Eide E, Neti C, Iyengar G. “A Medical Imaging Informatics Appliance,” in proceedings of Computer Assisted Radiology and Surgery (CARS'08), Barcelona, Spain, 2008.
[2] Lee N, Caban J, Ebadollahi S, Laine AF. “Interactive Segmentation in Multi-Modal Brain Imagery using a Bayesian Transductive Learning Approach.” Submitted to Medical Imaging 2009: Computer-Aided Diagnosis, Proceedings of the SPIE. 2009.