Portal de Periódicos da CAPES

The Use of Sound for Data Exploration

2000; Association for Information Science and Technology; Volume: 26; Issue: 5 Linguagem: Inglês

10.1002/bult.173

2163-4289

Myke Gluck,

Data Management and Algorithms

When presented with the feasibility of talking movies Jack Warner, early head of Warner Brothers Studios, is quoted as saying, "Who the hell wants to hear actors talk?" Unfortunately today, many data analysts might be overheard to make similar remarks regarding the use of sound for data analysis. Our work at Florida State University has begun to exploit sound in conjunction with other multimedia support, such as visualization and cartography, to aid data analysts in performing spatial data mining, exploratory data analysis and pattern detection. Data mining has many descriptions that are useful for our purposes. These include the reuse of data, multiple uses of data and analysis of patterns in large datasets. Data is expensive to collect and often is used once and discarded or archived without review. Data mining suggests that any given dataset may contain valuable information that can be reused for reoccurring analyses or the study of effects over time. Also, the same data can occasionally be reused for purposes for which it was not initially collected yielding new insights at low cost. Data mining may also suggest methods for exploration of data for new insights to the original data related to the original purpose for which the data was collected. Examples include weather data, satellite observation data, aerial photography, U.S. transportation data, U.S. census and county data as well as sampled files of audio and video. Data mining is actually a subset of a broader range of techniques called Exploratory Data Analysis (EDA) used to seek possibilities of patterns and effects. Often EDA techniques are used to generate hypotheses that more traditional methods may then be used to confirm. EDA often purposely overlooks the known patterns and explores for novel relationships; for example, EDA might ignore the standard results of linear regression analysis examining more carefully the residuals from such an analysis. Said more accurately, EDA and data mining often care more to avoid Type II statistical errors in which useful patterns are missed than be worried about selecting or describing a pattern that on further study is not really there. A major research arena has been the use of visualization to aid in the analysis of large datasets including virtual reality simulations for large datasets. The eye can often observe minute variations or broad patterns that are difficult to detect when presented with merely large columns of numerical data. Visualization is a natural addition to EDA since both permit more serendipitous examinations of data. Recent innovations in technology, such as larger storage devices, improved display devices and much faster digital processors, have made such software visualizations much more feasible. Such visualization methods allow analysts to manipulate data by rotating, zooming, panning, slicing and dicing datasets to extract meaningful relationships and patterns not easily discerned by traditional statistical techniques. Such visualization techniques also allow analysts with certain disabilities to better examine datasets. Visualization has a range of marvelous techniques and has led to many interesting insights in large datasets. Work in research for improving visualization techniques has stressed the analysis of various visual variables such as size, shape, orientation, color, texture and position as well as dynamic variables such as duration of image, image shift, image change and image sequencing. Research in visualization has studied how each of these and other such variables affect the ability of analysts to do their work effectively and efficiently. Researchers, including my research group, have begun to show that sound alone and in combination with visualization techniques can further enhance data mining and EDA methods employed by data analysts to seek relationships and patterns in large datasets. The use of sound in this regard is named sonification. Sonification studies the effects of sound variables such as pitch and relative pitch, volume, duration of tones, timbre (or sound quality, say, cello vs. piano middle C), rate of change of tones, articulation or order of sounds and augmentation (additional tones surrounding the main tone). Sounds are easily used as an alerting tool to quickly gain an analyst's attention to a strange detail or emergency condition. Tools of this sort have been called earcons, serving the purpose of icons to indicate important events. Sound can also be used to locate an object much as we hear a train coming and going. One minor but far from insignificant use of sonification is for those with visual impairments. Some researchers are exploring such use of sonification but our efforts are directed to sonification and sonfication with visualization for general populations. Several schemes have been devised to link data to sound: • Range depiction. The range of a variable is categorized into several classes and each class assigned a tone. For example, a higher value assigned a higher pitch, louder volume, different instrument, etc. • N-Tiles. The classification of the data into classes of equal numbers of data points. Executed similarly to range depiction. • Multiple Notes. Selection of an octave for a class of data and a particular note within that octave for a data value. • Different scales. Use of diatonic, five-note scales, other western, non-Western or Oriental music schemes and scales. • Chords. Use of multiple notes simultaneously to represent the data. Currently, most research in this arena links datasets with a-tonal or non-music sounds for study. Significant findings that have been reproduced in the study of sound and data analysis indicate that • Sound and visualization work better together than either works separately. • Sound patterns can be equally well-detected and described by musically and non-musically trained people. • Analysts can detect many data observations and patterns using either sonification or visualization techniques. We have built a software tool for both visualization and sonification studies that supports the use of maps and is designed essentially for spatial data analysts called augmented seriation. The tool is based upon a rather simple concept of replacing the numerical values within the cells of a spreadsheet data matrix with icons that are proportional in size to the numerical values (see Figure 1 ). The data analyst manipulates the rows and columns of the matrix seeking patterns in the data visually or with the aid of sound. Often the goal is to get all the empty or low value cells off the diagonal and the higher value cells on the diagonal region of the spreadsheet matrix. This diagonalization places the data in a series, and so the technique is called seriation. These manipulations permit making judgments about the relationship of the data to a single indicator. For example, this technique (without all our additional features) can be used to analyze and date objects from an archeological dig. The rows might represent various locations, the columns represent various attributes of pottery shards and the diagonalization process indicates the chronology of the sites or of the pottery styles from oldest to youngest. Further, our tool allows the analyst to select a cell or row and have it highlighted in various maps or select a region or place on the maps and highlight the row in the data matrix. This selection of data in one mode or medium and its display in another is called brushing. In addition our tool permits analysts to select several variables for display in map form, making manifest the spatial correlations among the variables. One feature not yet fully implemented allows analysts to display several data matrices of related data in a multidimensional structure that imitates multiple spreadsheets. Such matrices might represent the same data at different times and hence permits analysis of changes over time. Our tool's additional features of maps and sound augment the basic seriation process; hence the software name of augmented seriation. Bulletin of the American Society for Information Science The truly unique features of our spatial EDA and data mining software are the ability to do several novel sound explorations of the data and spatial distributions. Most traditional attempts at data representation inside cells use variations in pitch and/or volume based on data ranges and classes as described above. Our software allows for music-driven analysis in which a pre-selected piece of music is varied by tempo, key, augmentation and articulation. The analyst hears the same tune played in various ways that allow for detection of patterns. Our major insight is driven by the fact that most people more easily recognize and recall tunes and their variations much more readily than mere collections of sounds in some non-musical or atonal pattern. Future work in this project expects to study the usability of the sound features as well as the multidimensional descriptions and their efficacy in aiding spatial and non-spatial analysts to explore and find meaningful patterns in data. Figure 1 displays a typical augmented seriation screen. The data used is derived from the perceptions of risk managers of all the counties of New York State about the severity of eight different environmental risks to their county. Bivar map indicates the bivariate or combined map of these managers' perceptions of toxic spill and nuclear event risks to their county. Acknowledgments I wish to acknowledge the contributions to this work of Dr. Lixin Yu of FSU's School of Information Studies, Dr. Jack Taylor of the Center of Music Research and Dr. Jonathan Raper of the City University of London. Their efforts and intellectual support have been invaluable in development of this research project. This paper is based on a presentation given at the 1999 ASIS Annual Meeting.