Daniel L. Rubin, MD, MS, Stanford University; Chris Beaulieu, MD, PhD; Cesar Rodriguez, MD; Danny Korenblum; Sandy Napel, PhD

Advances in radiology increasingly depend on collecting and analyzing quantitative and qualitative features of images. However, this information is currently not recorded in a way that can be processed by computers. The annotations that radiologists make on images--to identify lesions, to delineate their borders, to measure ROIs, and to describe the visual features in those ROIs--are currently captured as image graphics and are not directly accessible for future analysis (Figure 1). Thus, for example, radiologists cannot easily process measurements made from images, such as the sum of linear dimensions on serial scans, to evaluate criteria of treatment response, such as RECIST.[1] In addition, the visual observations made by radiologists are recoded in radiology reports that are disconnected from the images and not easily searched for specific anatomy, findings, and disease. For example, it is not possible to query a PACS to display images showing specific findings. In order to achieve such functionality, methods are needed to make quantitative and qualitative information associated with images ("image metadata") accessible for search.

Figure 1: Image Annotations

Images contain rich information, little of which is accessible to computers for query and analysis. Left: A radiologist has graphically identified a lesion in the liver, delineated its margin, measured its size, and described its visual features. None of this information can be readily accessed for query and analysis (e.g., “find all images with ROIs drawn in right lobe of the liver”). For that to happen, a structured representation of the same information is required so computers can understand what types of information are contained in images, their data types, and values.

Our goal was to develop an approach to making the key quantitative and qualitative information in images and their associated reports searchable and linked to the images to enable query and large-scale analysis of image databases. We developed an open-source tool, iPad, to enable researchers and clinicians to create computer-accessible “semantic annotations” on radiological images. The annotations are compliant with a new standard called AIM (Annotation and Image Markup), which is emerging from the National Cancer Institute’s cancer Biomedical Informatics Grid (caBIG).[2,3,4]. We also created a resource, Biomedical Image Metadata Manager (BIMM) for collecting, storing, and searching the rich image information our tool collects. BIMM enables radiologists and researchers to search for images in flexible ways, by anatomy, findings, or other rich image metadata. Together, these tools are ultimately intended to improve radiologist interpretation by enabling on-line decision support, and to facilitate biomedical researchers accessing image information as part of their investigations of the biology and the manifestations of disease.

iPad

We built iPad as a plug-in to OsiriX, an open-source image viewing application.[5] OsiriX provides the basic functionality of an image viewing workstation, while iPad provides the ability to create semantic annotations compliant with the AIM standard.

iPad permits users to describe images and image regions using a graphical interface (Figure 2) and drawing tools similar to those that radiologists currently employ (Figure 1). In addition, iPad unifies related concepts that have the same meaning by mapping textual descriptions to RadLex, a controlled terminology for Radiology.[6,7] iPad also uses an ontology to check that the semantic content of the image annotations is appropriate to the type of image being annotated. iPad allows recording of annotations as a combination of as free text, single-click phrase insertions, as well as results of a contextual search of the RadLex ontology.

Figure 2: iPad Graphical User Interface.

iPad, a plugin to Osirix, enables the user to annotate lesions, describing their anatomic locations and visual observations, while checking to make the user’s entries match standard terms in RadLex. If entries do not match terms in RadLex, or if invalid combinations of findings and anatomy are detected, the user is alerted. iPad collects image metadata from the user entries, from the DICOM header, and from graphical drawings created during the annotation process, then writes the metadata to a standard AIM file.

We evaluated the usability of iPad in a clinical research study in which two radiologists used it to annotate cases of liver lesions. The radiologists annotated 10 radiological images, acquired as part of that research study, using iPad to describe the features of abnormalities seen in each image. They were then asked to qualitatively evaluate iPad in terms of its usability in the image interpretation workflow and the utility of the feedback iPad provides, compared with their experience in unassisted collection of image metadata (the current paradigm in radiology research).

The radiologists reported that iPad enabled them to annotate images efficiently using a similar workflow to that they are accustomed, completing the 10 cases within 45 minutes. While iPad collects detailed structured information, the manner in which it accomplishes this task did not qualitatively hinder the radiologists in describing the images and recording their observations.

BIMM

iPad efficiently produced computer-accessible structured output of image measurements and observations in AIM files linked to DICOM images. The AIM files could be uploaded to the BIMM resource as a background process in iPad, while the radiologist annotated additional images. BIMM is a relational database, providing a Web portal for user queries and browsing of the image metadata (Figure 3). Users can query the repository of image metadata in flexible ways, such as using the usual information in DICOM image headers and/or the structured annotations described in the AIM files (e.g., imaging observations, anatomic entities within the images, diagnoses [Figure 3]), or the person who created the annotations (to enable inter-reader analyses).

Figure 3: Biomedical Image Metadata Manager (BIMM).

AIM files are uploaded to the BIMM, which parses the AIM files and stores their information content in a database that can be searched according to individual AIM fields. The figure illustrates the results of a query for focal nodular hyperplasia, retrieving images and their associated metadata. As seen in this figure, one can search or analyze images according to imaging observations, characteristics, anatomy, and quantitative aspects of the image.

As highlighted by the TRIP initiative, a major challenge for radiology is image/information overload and the need to develop methods for computers to help radiologists work effectively with massive image databases for new discovery, improved interpretation, and better quality.[8] Our work begins to tackle this challenge by making the rich information content in images machine-accessible and minable for research and radiological practice through two tools, iPad and BIMM, for creating semantic image annotations and for storing/searching them. In attempting this, there is a tension between making information computer-friendly while keeping the tools user-friendly. Our work addresses this balance in the following ways:

1. Hiding information complexity from the user: iPad implements the AIM schema for recording the rich quantitative and qualitative image metadata. AIM is highly detailed, with sections for recoding image ROIs, calculations on ROIs, anatomy, image observations, and information about the patient and reader of the image. iPad hides this complexity by populating fields where possible from the DICOM header or from the information objects users create transparently as they annotate images.

2. Support for controlled terminologies: Controlled terminologies, such as RadLex, are emerging as important standards for reporting imaging studies, particularly in research studies where consistent terminology will enable analysis. iPad provides an intuitive user inteface that maps the descriptions that users enter into RadLex terms as the user reports the case (Figure 2), eliminating the need for terminology “look-up” by the user.

3. Mechanism to ensure complete annotations: In general, there is a minimum set of information required to completely describe the content of an image. For example, a region of interest may need to be annotated with the anatomy to which it relates, with at least one imaging observation, and optional modifiers on those observations. iPad detects incomplete or invalid annotations (e.g., visual observation without describing its anatomic location), and alerts the user to ensure complete and valid image annotations.

4. Searchable annotation repository: Once annotions are created, they need to be stored in a manner that makes future query and analysis of these image metadata facile. BIMM provides an archive for all image metadata with a user-friendly Web portal for querying AIM documents, making the individual data fields directly searchable (Figure 3).

These tools enable radiologists to access the rich information content in images and to query large databases for patients/studies/images containing specified information. iPad, a semantic image annotation tool, enables radiologists to create a structured description of image contents and BIMM, a resource that warehouses semantic image annotations, linked to the original images, and provides the means for radiologists to search for particular image content. Making radiology image content structured and computer-accessible through methods such as these will enable radiologists and biomedical researchers to tap into the rich knowledge in large image databases for decision support and biological discovery.

1. Therasse P, Arbuck SG, Eisenhauer EA, et al. New guidelines to evaluate the response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada. J Natl Cancer Inst. 2000;92(3):205-216.
2. Rubin DL, Mongkolwat P, Kleper V, Supekar K, Channin DS. Medical Imaging on the Semantic Web: Annotation and Image Markup. 2008 AAAI Spring Symposium Series, Semantic Scientific Knowledge Integration. 2008.
3. Wolfson W. caBIG: seeking cancer cures by bits and bytes. Chem Biol. 2008;15(6):521-522.
4. Prior FW, Erickson BJ, Tarbox L. Open source software projects of the caBIG In Vivo Imaging Workspace Software special interest group. J Digit Imaging. 2007;20Suppl1:94-100.
5. Jalbert F, Paoli JR. Osirix: Free and open-source software for medical imagery. Revue De Stomatologie Et De Chirurgie Maxillo-Faciale. 2008;109(1):53-55.
6. Langlotz CP. RadLex: A new method for indexing online educational materials. Radiographics. 2006;26(6):1595-1597.
7. Rubin DL. Creating and Curating a Terminology for Radiology: Ontology Modeling and Analysis. J Digit Imaging. 2007.
8. Andriole KP, Morin RL. Transforming medical imaging: The first SCAR TRIP conference a position paper from the SCAR TRIP subcommittee of the SCAR research and development committee. J Digit Imaging. 2006;19(1):6-16.