Image and Video Processing for Cultural Heritage

The preservation, archival, and study of cultural heritage is of the utmost importance at local, national, and international levels. Not only global organizations like UNESCO but also museums, libraries, cultural institutions, and private initiatives are working in these directions. During the last three decades, researchers in the field of imaging science have started to contribute a growing set of tools for cultural heritage, thereby providing indispensable support to these efforts.

Indeed, signal, image, and video processing, computer vision, 3D modelling, and graphics technologies are nowadays widely employed to capture, analyze, conserve, virtually or physically restore, document, classify, recognize, and render cultural artefacts. These include historic buildings and monuments, archaeological sites and finds, works of art (paintings, frescoes, sculptures, decorative items, etc.), manuscripts, music score manuscripts, photographs or photographic negatives, films, and other entities of artistic, historical, or archaeological importance. This research can be grouped into two main strands. On the one hand, tools aim to provide easy access to cultural heritage by both the general public and scholars. On the other hand, a substantial body of work wants to ensure its preservation for future generations. An important factor that expedited and boosted the use of image and video processing techniques in cultural heritage applications was the initiation of extensive digitization campaigns by public institutions, museums, libraries, and archives during the last years. As a result, a huge amount of culture-related information is now digitally stored, and thus amenable to digital processing.

The field of image and video processing for cultural heritage encompasses a large variety of topics, such as high-resolution 2D and 3D digital capture and rendering of artworks, digital restoration, enhancement, recognition, and classification of features, structures and content in cultural heritage visual data, creation of large-scale multimedia databases of artworks, and user-centered heritage-related visual or multimedia applications. Indeed, image and video processing techniques can significantly improve and make more efficient many aspects of traditional preservation, archival, study, and fruition of our cultural heritage and, very interestingly, can also provide answers to emerging needs. Moreover, they have made feasible the creation of new applications and tools, which would otherwise be impossible to realize.

For instance, multispectral, multisensory, multiresolution, and multiframe imaging, complemented with quality evaluation and correction of degradations caused during acquisition, data registration, integration, and mosaicing, can help with the qualitative and quantitative analysis of cultural artefacts. These high-resolution digital representations, combined with various processing techniques and 3D modelling methods, can then be used to assist with the physical restoration, condition assessment, and examination of artworks. At the same time, digital restoration or reassembly of damaged artworks (films, photographs, paintings, frescoes, pottery, manuscripts, etc.) is becoming standard practice.

Recovery and enhancement of hidden, damaged, or masked information, for example, the detection of erased texts in palimpsests or the detection of preliminary drawings or hidden layers on paintings, as well as the extraction of information about the type of material used (e.g., quality of the support, types of colors and ink, etc.), can provide precious knowledge about an artwork and its creator, assist in its dating, and even facilitate the detection of forgery or fakes.

For the creation of multimedia artwork repositories, methods for the efficient storage, transmission, processing, and visualization of large data sets (3D models, high-resolution images, etc.) are required. In addition, techniques for the efficient indexing and intuitive retrieval and browsing of stored items, adapted to the particularities of cultural heritage, are also actively investigated.

Finally, computer graphics and virtual and augmented reality have found interesting, novel applications in the field of cultural heritage, including the 3D reconstruction of ancient monuments and cities with stereo viewing, 3D navigation and manipulation capabilities, and haptics-enabled 3D virtual museums. User-centered visual or multimedia applications for museums, digital art repositories, and edutainment are also becoming increasingly popular.

Processing and analysis of visual cultural heritage data is a very challenging field that does not merely exploit and apply standard techniques, already developed in other application domains, but often entails original research that is specific to this domain. A case in point are the methods for modelling the color degradation of paintings and the methodologies for their virtual or physical restoration. In addition, it usually requires multidisciplinary efforts, bringing together engineers, computer scientists, restoration experts, archaeologists, historians, and art curators.

The eight papers that have been selected for publication in this special issue (out of the twenty one submitted) present interesting new ideas on a number of topics related to visual data processing and analysis for cultural heritage applications. Although not all areas mentioned above are represented, we do hope that the issue will give readers the opportunity to sample some state-of-the-art approaches and appreciate the diverse methodologies, research directions, and challenges in this extremely broad and important field of research.

The first two papers of the issue deal with two drastically different imaging techniques and their application in cultural heritage preservation. In a paper entitled "Multispectral Acquisition of Large-sized Pictorial Surfaces," A. Paviotti et al. examine the challenges posed by the acquisition of large-sized pictorial surfaces, such as frescoed rooms or large paintings, as compared with the acquisition of regular paintings. Special attention is paid to the critical aspect of the illumination sources that must be placed far from the scene to be acquired. Four illumination setups have been tested versus the acquisition of the spectral reflectance of a set of calibrated colored tiles. The error has been defined as a function of wavelength, using a metrological procedure to infer the uncertainty of the computed error from the statistics of the measured variables. The illumination setup that combines metallic iodide lamps with incandescence lamps is the one giving the best results and has been found suitable for acquisitions in a controlled environment. The illumination setup combining metallic iodide lamps with halogen lamps also yields a good performance and has been found to be preferable for applications in-the-field.

The paper "Resistivity Probability Tomography Imaging at the Castle of Zena, Italy,'' authored by V. Compare et al., describes an interesting case-study, where the recently developed 3D probability-based electrical resistivity tomography (ERT) is applied to locate buried structural remains of the medieval Castle of Zena (an Italian 13th century fortress, destroyed in the 18th century). Data analysis has been performed by using a low cost, fast approximate inversion method, suitable to deal with the huge quantities of data generated from this kind of archaeological surveys. The 3D tomography has allowed three interesting anomaly source areas to be identified in the 1-2 m depth range below ground. Subsequent excavations have brought to light a brickwork room for food maintenance, a furnace, and the basement of a wing of the castle, exactly in correspondence with the anomaly sources detected by the geoelectric inspection.

The next two papers focus on the analysis and archival of manuscripts. In the paper "Optical music recognition for scores written in white mensural notation,'' L. J. Tardòn et al. propose a complete system for optical music recognition of ancient scores written in white mensural notation. The focus is on manuscripts dating back to the 17th and 18th centuries, and two different notation styles have been considered. The system consists of the segmentation of the music symbols, their classification, and their transcription into a suitable electronic format, for the creation of a playable MIDI file. The crucial step of music symbol segmentation is performed through several, adapted image processing techniques. Similarly, multiple methods for the extraction of features of the music symbols have been implemented, and the resulting vectors have been employed in various classification strategies. The combination of feature vectors based on angular-radial transform coefficients with a k-NN classifier has been found the best for the dataset considered.

In "Multimedia in cultural heritage manuscripts: integrating description, transcription and image content,'' C. Calistru et al. propose an indexing and retrieval environment for the integrated management of the vast and heterogeneous information often associated with collections of historical documents. In this system, the documents are processed in their descriptive, textual, and image content, the MetaMedia multimedia database platform is used to account for both metadata and content, and a browsing and searching interface is developed where all the various and diverse descriptors are fully accessible and linked together. The collection used as a case study is a set of Portuguese medieval documents available on-line. A hierarchical structure for the documents, their descriptions, and content annotations, including the transcriptions, has been integrated in the MetaMedia platform. The document repository can be enriched with new content descriptors by its curators, and generic users can access and browse the document center, searching the structured information in the descriptors and annotations, or finding documents by similarity in their textual or visual features.

The fifth and sixth papers in this collection present two interesting approaches for the restoration of old photographic prints and negatives, respectively. The first of those papers, authored by V. Bruni et al. and entitled "Context-based defading of archive photographs,'' presents a multistage approach for the defading, that is, the enhancement of the contrast, in antique photographic prints. Conventional enhancement algorithms often produce artifacts that affect the visual quality of the restored images. The authors focus on resolving some of these drawbacks, by exploiting the local context of each pixel. In particular, they propose to exploit the relation between the variation of contrast at different resolutions and the local Lipschitz regularity of the image, in order to classify each pixel as noise pixel, edge pixel, or pixel belonging to a smooth region. The pixel intensity is thus adaptively corrected according to its nature. An optimal, global gamma correction is then performed to complete the restoration process. The experimental results showed that this strategy allows for a gradual enhancement of the image while avoiding typical drawbacks like halos and noise amplification.

Old acetate-based safety negatives are important cultural assets whose digital preservation and restoration have received relatively little attention. The paper "A new technique for the digitization and restoration of deteriorated photographic negatives," by G. Landon et al., details the development and analysis of a novel image-based photo-negative restoration system. The system uses a combination of structured-light and high-dynamic range imaging to acquire the data that allows for both photometric and geometric correction of the negatives. The intrinsic intensity information and shape distortion of the film are modeled using a single scatter diffuse transmission model. In terms of hardware, the proposed approach employs a simple to deploy and low-cost camera and LCD system. Analysis results show the good accuracy of the proposed method. Results from the application of the system on actual degraded negatives are also provided.

The last two papers of this issue deal with two fields that are already having a large impact on art and cultural heritage, namely, graphics and virtual/augmented reality. The paper "Augmented reality for art, design & cultural heritage; system design and evaluation," by J. Caarls et al., describes the design of a see-through head-mounted display (HMD) system, which is accompanied by a head pose tracker, for augmented reality (AR). The evaluation and testing of the proposed design was performed by using it in art productions and exhibitions in museums. The goal was to make virtual objects indistinguishable from real objects and to find out to which extent imperfections are hindering its application especially in art and cultural heritage. Since for AR the fast and accurate measurement of head motions is crucial, the authors designed a head tracker for the HMD that combines an optical and an inertial tracker using error state Kalman filters. The use of the system in applications proposed by artists and designers, including applications in museums, led to very interesting results. They showed that AR is a powerful tool for bringing the general public closer to art and cultural heritage.

S. Haegler et al. deal in their paper "Procedural modelling for digital cultural heritage" with the use of procedural modelling approaches based on shape grammars for the 3D visualization of archaeological sites. An important issue here is that, since computer graphics nowadays provides the archaeologist with several tools to realistically model and visualize archaeological sites in 3D, visually compelling models may lead people to falsely believe that there exists very precise knowledge about the past appearance of a site. One way to visualize the underlying uncertainty is to modulate the respective levels of transparency. The authors argue that procedural modelling provides an interesting alternative to such methods, which tend to spoil the viewing experience. Its efficiency and compactness make procedural modelling a tool capable of producing multiple models, which together sample the space of possibilities. Thus, variations between the different models implicitly express the levels of uncertainty, while letting each individual model be visualized realistically. Several recent examples, like the Rome Reborn 2.0 project, are used to demonstrate the proposed procedural modelling implementation.

The guest editors of this issue wish to thank the numerous reviewers who have volunteered their precious time to provide valuable feedback to the authors. They would also like to express their gratitude to the contributors, for making this issue a hopefully important source of information within the existing body of knowledge in the field of image and video processing for cultural heritage. Finally, they would like to acknowledge the assistance of the editorial support team of the EURASIP Journal on Image and Video Processing throughout the preparation of this issue.

Vincent CharvillatAnna TonazziniLuc Van GoolNikos Nikolaidis