Contributions published at Visualization and MultiMedia Lab (Renato Pajarola)

inergia: Future Cinema is a Swiss National Science Foundation funded project led by an interdisciplinary team of scientists in machine learning, visual analytics, digital museology and archival science. Sinergia strives to offer the public access to large-scale audiovisual archives of recordings from the twentieth and twenty-first century. The archive of recordings is edited with computational techniques to provide an impressive and memorable museological experience through beautiful visualisations the users can interact with. In this context, the University of Zurich acts as a partner of Sinergia and is responsible for visual analysis. This master thesis aims to do visual analysis on the International Olympic Committee (IOC) data set. In specific, it focuses on the comparison and analysis of Visual Patterns for Runners of One of a Kind. The goal of the thesis is to develop a tool to support the comparison of sprinters. The results of the comparison should be visually presented to the user. The experience gained from the development of such a technical solution should help to understand what challenges need to be accounted for in order to develop a running pattern analysis tool. The visualisation tool should take audiovisual files as input and output visualisations in the form of a virtual track to let runners virtually compete against each other. Displayable charts within the application further support the analysis and comparison task. A special focus while developing the sprint comparison tool was placed on the definition and implementation of a user-friendly User Interface (UI), as the end-users might not be computer science affine. The UI has been developed and designed using proven visual guidelines. A data preparation pipeline is provided to extract skeleton data out of audiovisual files using image recognition software. Furthermore, the preparation pipeline includes post-processing the data, which results in better data quality. It additionally puts the data into a format that can be used with the frontend technology to eventually visualise it in the UI. Depending on the outcome of the application development, the thesis gives insights whether and under which circumstances it is possible to generate a meaningful visualisation tool for sprinter comparison with the current state-of-the-art technology. Besides the application development, this master thesis evaluates, based on literature review, sprint-specific features that must be examined to allow a sprinter comparison. Such features include movements that have major impact on a sprint event and are decisive for winning, as for example the way a sprinter positions his front leg's knee during each step. Literature research on sprint-specific features also include a sprint segmentation to divide the race into different sprint phases (start, acceleration, maximum speed, finish). For each sprint phase, movement patterns are sought to be discovered, as they are crucial for the fastest possible sprint. This master thesis concludes that comparisons between sprinters based on video files are possible. However, this comes with certain limitations: During the development of the tool, difficulties related to the image recognition software and its skeleton data extraction capabilities were encountered. Additionally, video files with different viewing angles and image focuses posed a challenge that could not be resolved in this master thesis. Therefore, sprint comparison analyses are still limited since for many limitations no general solutions exist yet.

The increasing size and dimensionality of datasets in the humanities pose new challenges to scholars working with them, in- cluding establishing an overview over the dataset, connecting concepts, developing new hypotheses, and testing them. Material, pattern, and texture aesthetics in moving images is an attractive example of such multi-dimensional datasets in film studies, as an almost infinite number of combinations thereof are possible. Clustering techniques such as t-SNE are popular automated methods to organize these complex datasets, but they bring little or no-semantic meaning to their grouping strategies. We pro- pose a novel interactive visualization technique for multi-level hierarchical exploration of clustered features, named Sankey- Bridges. Our technique allows the users to (1) abstract local and global semantics from the automated methods, (2) extract relevant relationships, and (3) quantify them. Our technique is embedded in a system with other interactive visual components combined with exhaustive computational methods. The proposed solution is able to convey the global and local structure of high-dimensional clustered data sets and the relationship between different groups of features. The resulting visualization tool is embedded in the well-established VIAN [HBRFP19] research framework. We illustrate the benefits of our approach in the context of typical film researchers’ investigation of relationships in high-dimensional spaces, and a wide range of qualitative analysis labels, with examples from an extensive film database.

Rendering large amounts of terrain data is fundamental to a lot of different applications in academic, professional, and personal contexts. The technological advancement of computers over the last decades made handling the large amounts of data required to present real world terrains feasible for stand-alone applications. However, providing visual fidelity with a performance that allows intuitive navigation through the terrain over the internet running in a browser is still an open problem. This work investigates how already known approaches to terrain rendering can be implemented in a web application running in the browser. The produced application allows users to navigate through a high-resolution terrain on typical PCs, smart phones, and tablet computers. Further the application allows the user to draw and safe lines that are pixel perfect rendered on top of the terrain providing a use case for the application.

We present the implementation, and evaluation of a software library for efficient and parametrizable edge-collapses and vertex-splits. The library consists of a fast mesh data structure, decimation and refinement components and includes a set of modules that guide the decimation and refinement processes. The library is free of external dependencies and is modular in design. We investigate the choices we made during development, and give a detailed description of our implementation. We evaluate our implementation in terms of decimation quality and computational performance and compare it to other existing tools and libraries. Our evaluation demonstrates that both in terms of quality and performance our implementation can match or beat existing solutions.

In this work, we present several interactive visual designs for mobile visualization of severe weather events for the communication of weather hazards, their risks, uncertainty, and recommended actions. Our approach is based on previous work on uncertainty visualization [5], cognitive science [6], and decision sciences for risk management [3, 4]. We propose six configurations that vary the ratio of text vs graphics used in the visual display, and the interaction workflow needed for a non-expert user to make an informed decision and effective actions. Our goal is to test how efficient these configurations are and to what degree they are suitable to communicate weather hazards, associated uncertainty, risk, and recommended actions to non-experts. Future steps include two cycle of evaluations, consisting of a first pilot to rapidly test the prototype with a small number of participants, collect actionable insights, and incorporate potential improvements. In a second user study, we will perform a crowd-sourced extensive evaluation of the visualization prototypes.

In real-time computer graphics applications such as video games, simulators, or increasingly geographic information systems, vegetation is present in almost every scene imaginable. In this context, it is an indispensable ingredient to achieve a high degree of realism. In the case of a forest covering an entire landscape, the vegetation to be represented may be millions of individual trees. Furthermore, to obtain a convincing result, these trees must not only be rendered, but also provided with an illumination model. Having said that, realistic lighting and rendering of vegetation on a large scale remains a difficult task. Due to the immense complexity of a single tree, which can have thousands of individual leaves, such a scene contains an unimaginable amount of geometry. Therefore, the goal and challenge is to reduce this complexity to achieve feasibility while still providing satisfactory results with realistic lighting. Against this background, this thesis delivers two contributions. First, a method is implemented to reconcile the above-mentioned objectives, namely a z-buffer shape reconstruction method. This method reconstructs a 3D model from precomputed 2D images. Moreover, to demonstrate the capabilities of such an approach, it is applied to real GIS data of Switzerland. Thereby a large spatial image data set is used, which fits neither in the main memory nor in the graphics memory. As a second contribution, a virtual texturing system is implemented that can handle such immense amounts of texture data. Finally, the developed methods are combined in an application that allows the rendering of the deciduous and coniferous forests of the whole of Switzerland in real-time and with realistic lighting and shadows.

Rendering engines like appleseed use camera simulations for image formation. Oftentimes however, such camera models are simplifications of realistic cameras that compromise true realism for rendering efficiency. In this thesis, appleseed's camera models are extended by a physically-based multi-lens camera based on scientific literature that features realistic lens phenomena. The implementation is then tested and compared to simpler camera models. The results show that the implemented model is capable of capturing a multitude of optical phenomena of real cameras and that these effects achieve realism better than existing camera models. While a certain rendering deficiency is found, it is shown that this overhead solely depends on the used lens's complexity and, under the right conditions, accounts for a fraction of the total rendering time.

Direct volume rendering (DVR) is a technique to visualize 3D volume data without an intermediate geometric representation. Thereby, GPU-accelerated ray casting is used to parallelize volume rendering with fragment shaders. Rays are sent from the eye through the volume for every pixel in the image plane. The volume data is sampled along the ray, and an output color is computed for every pixel through alpha blending. A transfer function maps optical properties such as opacity and color (RGBA) to the voxel data during composition. Studies have shown that immersive, virtual environments are well suited for the visual exploration of spatial data. Furthermore, Virtual Reality (VR) hardware is much more readily available than a few years ago. Headsets are sold at reasonable prices and with considerable power, providing enough frames per second to be considered for DVR. In this thesis, volume rendering in the context of virtual reality is investigated. To that end, an interactive volume rendering application for virtual reality is implemented using the HTC Vive, OpenVR, and OpenGL. The application offers an interactive user interface allowing users to design and adjust transfer functions and see the changes in real-time. With hand-held controllers, the users can place and move cutting planes to look inside the volume. Acceleration techniques such as early ray termination are employed to increase performance. The feasibility of the application is evaluated by measuring interactive frame rates in different scenarios.

Effective communication of weather hazards to the general public is critical to prevent and mitigate adverse outcomes from weather hazards. With the emergence of technology, there are many approaches to weather forecast visualization, such as contour and thematic maps, that communicate forecast information about specific geographic regions. For example, in Switzerland, MeteoSwiss App provides accessible and accurate weather forecasts, including natural hazards. However, sometimes the visual encodings are challenging to grasp and not informed by best practices in visualization research, such as utilizing visual encodings difficult or time-consuming for non-experts to interpret. In this master thesis, we propose different visual metaphors for the communication of Hailstorms as a weather hazard to improve the decision-making process in such severe conditions. The proposed interactive mobile prototype contains visual encodings to develop hazard maps for risk perception, temporal visualization to know the time of the hazard, and behavioral recommendations for actions to prevent and mitigate adverse outcomes in such severe weather events. These behavioral recommendations are tailored to meet the decision needs of different user groups in different scenarios. Finally, the mobile prototype is created in Figma with fluid interactions and smooth transitions to increase effective communication and reduce cognitive resources. Our evaluation shows that the recipient’s perception of risk does not follow the behavioral recommendations for actions in a low-risk scenario. In contrast, in a medium and high-risk scenario, the recipient’s perception of risk chooses to follow the behavioral recommended actions. The preferences for behavioral recommendations were represented on three levels, and in particular, the mixed(Textual + Icon) approach stands out as the best approach to communicate behavioral recommendations for actions.

Analyzing body movement as a means of expression is of interest in diverse areas, such as dance, sports, films, as well as anthropology or archaeology. In particular, in choreography, body movements are at the core of artistic expression. Dance moves are composed of spatial and temporal structures that are difficult to address without interactive visual data analysis tools. We present a visual analytics solution that allows the user to get an overview of, compare, and visually search dance move features in video archives. With the help of similarity measures, a user can compare dance moves and assess dance poses. We illustrate our approach through three use cases and an analysis of the performance of our similarity measures. The expert feedback and the experimental results show that 75% to 80% of dance moves can correctly be categorized. Domain experts recognize great potential in this standardized analysis. Comparative and motion analysis allows them to get detailed insights into temporal and spatial development of motion patterns and poses.

Virtual reality enables a far more immersive experience than a visualization on a 2D screen. Moreover, the representation of data distributed over the three spatial dimensions can be perceived in a very intuitive way. The goal of this work is to illustrate the representation of segmented point clouds as well as to present possibilities for interactions with point clouds in VR. Furthermore, the process which integrates the raw data of a 3D scan into a VR application shall be automated. In particular, the immersive experience is enhanced by having the user physically located in the corresponding space from which their scanned data is viewed. The final result is achieved by training a neural network with a dataset that includes scans of office spaces. After preprocessing the data from a point cloud scan of an office, it can be segmented using the trained network. Finally, the segmented data is rendered in a game engine so that it can be viewed and interacted with in virtual reality. Thus, the segmentation can be visualized and serves as the basis for interactive elements that creates an inspiring or game-like experience.

We propose a novel educational approach for teaching visualization, using a community-driven and participatory methodology that extends the traditional course boundaries from the classroom to the broader visualization community.We use a visualization community project, VisGuides, as the main platform to support our educational approach. We evaluate our new methodology by means of three use cases from two different universities. Our contributions include the proposed methodology, the discussion on the outcome of the use cases, the benefits and limitations of our current approach, and a reflection on the open problems and noteworthy gaps to improve the current pedagogical techniques to teach visualization and promote critical thinking. Our findings show extensive benefits from the use of our approach in terms of the number of transferable skills to students, educational resources for educators, and additional feedback for research opportunities to the visualization community.

Visualization guidelines provide impressions and guidance on how a visualization should be designed to ensure effective interpretation by people and systems. However, there is no integrated framework in this area that provides all guidelines for visualization design and evaluation, as the sources of such guidelines are spread across a variety of platforms and are accordingly difficult to compare and evaluate with each other. In this master thesis, visualization guidelines are collected and categorized according to three different methods. The classification methods are based on: Thematic topics, the open coding principle and an automatic topic modeling method based on non-negative matrix factorization (NMF). Also, further analysis will be performed to discover textual relationships and similarities between the different guidelines. The main results of the thesis are finally visualized through a visual interface called "VisGuidesExplorer". This could help decision makers to follow the right visualization guidelines to make the right decision in their visualization design. Keywords: Visualization Guidelines, Guideline Categorization, Open Coding, Topic Modeling, Semantic Similarity Analysis

Analyzing the evolution of thunderstorms is critical in determining the potential for the development of severe weather events. Existing visualization systems for short-term weather forecasting (nowcasting) allow for basic analysis and prediction of storm developments. However, they lack advanced visual features for efficient decision-making. We developed a visual analytics tool for the detection of hazardous thunderstorms and their characterization, using a visual design centered on a reformulated expert task workflow that includes visual features to overview storms and quickly identify high-impact weather events, a novel storm graph visualization to inspect and analyze the storm structure, as well as a set of interactive views for efficient identification of similar storm cells (known as analogs) in historical data and their use for nowcasting. Our tool was designed with and evaluated by meteorologists and expert forecasters working in short-term operational weather forecasting of severe weather events. Results show that our solution suits the forecasters' workflow. Our visual design is expressive, easy to use, and effective for prompt analysis and quick decision-making in the context of short-range operational weather forecasting.

Recent years have seen a proliferation of new digital products for the efficient management of indoor spaces, with important applications like emergency management, virtual property showcasing and interior design. While highly innovative and effective, these products rely on accurate 3D models of the environments considered, including information on both architectural and non-permanent elements. These models must be created from measured data such as RGB-D images or 3D point clouds, whose capture and consolidation involves lengthy data workflows. This strongly limits the rate at which 3D models can be produced, preventing the adoption of many digital services for indoor space management. We provide a radical alternative to such data-intensive procedures by presentingWalk2Map, a data-driven approach to generate floor plans only from trajectories of a person walking inside the rooms. Thanks to recent advances in data-driven inertial odometry, such minimalistic input data can be acquired from the IMU readings of consumer-level smartphones, which allows for an effortless and scalable mapping of real-world indoor spaces. Our work is based on learning the latent relation between an indoor walk trajectory and the information represented in a floor plan: interior space footprint, portals, and furniture. We distinguish between recovering area-related (interior footprint, furniture) and wall-related (doors) information and use two different neural architectures for the two tasks: an image-based Encoder-Decoder and a Graph Convolutional Network, respectively. We train our networks using scanned 3D indoor models and apply them in a cascaded fashion on an indoor walk trajectory at inference time. We perform a qualitative and quantitative evaluation using both trajectories simulated from scanned models of interiors and measured, real-world trajectories, and compare against a baseline method for image-to-image translation. The experiments confirm that our technique is viable and allows recovering reliable floor plans from minimal walk trajectory data.

Tensor decomposition methods and multilinear algebra are powerful tools to cope with challenges around multidimensional and multivariate data in computer graphics, image processing and data visualization, in particular with respect to compact representation and processing of increasingly large-scale data sets. Initially proposed as an extension of the concept of matrix rank for 3 and more dimensions, tensor decomposition methods have found applications in a remarkably wide range of disciplines. We briefly review the main concepts of tensor decompositions and their application to multidimensional visual data. Furthermore, we will include a first outlook on porting these techniques to multivariate data such as vector and tensor fields.

Sobol's method is one of the most powerful and widely used frameworks for global sensitivity analysis, and it maps every possible combination of input variables to an associated Sobol index. However, these indices are often challenging to analyze in depth, due in part to the lack of suitable, flexible enough, and fast-to-query data access structures as well as visualization techniques. We propose a visualization tool that leverages tensor decomposition, a compressed data format that can quickly and approximately answer sophisticated queries over exponential-sized sets of Sobol indices. This way, we are able to capture the complete global sensitivity information of high-dimensional scalar models. Our application is based on a three-stage visualization, to which variables to be analyzed can be added or removed interactively. It includes a novel hourglass-like diagram presenting the relative importance for any single variable or combination of input variables with respect to any composition of the rest of the input variables. We showcase our visualization with a range of example models, whereby we demonstrate the high expressive power and analytical capability made possible with the proposed method.

The visualization of real-world winds on tangible surfaces is a process that can provide better understanding of a phenomena that is, at its core, hard to predict. The ability to simulate, compare, and visualize wind velocities can help to determine their effects over space and time. With respect to fluid simulations, computer graphics research typically concentrates on small-scale scenarios using water. This work aims to broaden the current spectrum in two ways. First, by enlarging the simulation domain to a medium-scale geographical area, and second, by using an implementation of SPH in order to simulate the winds above this region. This was achieved through the integration of a coarse real-world wind dataset, which is used as initial conditions for the winds. The results are then interactively simulated and visualized either on a regular screen or over a 3Dprinted terrain of the region using a projector. The final simulation, coupled with information about the terrain topography, is able to show wind movements with higher detail than the original input data.

Creating a lot of 3D assets for visualisations, computer games, or animated movies can take many working hours. The need for creating such high-quality assets in significant numbers is higher than ever. However, these assets generally take a lot of disk space to store, are hard to create, and not much of a created asset can be reused for another without affecting the end product's quality. With the advent of procedural techniques and software, this is changing rapidly. This work introduces proc-rock, a program specialising in creating realistic-looking rocks by mapping visual and geological features to procedural techniques. Based on the geological categorisation of rocks, having three default mappings gives the user the option to create various realistic-looking rocks without needing expertise in procedural techniques or even technical art. These mappings were made possible due to a highly customisable pipeline approach, node-based noise function graphs, boolean mesh geometry, and skin surfaces. Proc-rock also creates reproducible results between operating systems and computers via simple text files, which store the parameters used for the creation.

The number of applications requiring large point cloud data sets is increasing. Sensors, such as Light Detection and Ranging (LiDAR), are able to produce large point clouds as raw data. Therefore, this data must undergo a process of annotation in order to obtain semantic meaning. Many data science applications exploit deep-learning techniques which require large amounts of annotated data to train the underlying neuronal network. Annotation must generally be performed manually. As data sets increase in complexity, time and effort needed for the annotation also increase. Therefore, general-purpose tools to simplify and accelerate the process of annotation can bring benefit to the entire process of data analysis. In this thesis such a toolkit is developed, the PointCloudAnnotator. The PointCloudAnnotator is able to load point clouds as unstructured lists of points or as a collection of registered panoramic depth maps. The points can then be visualised as a 3D model. Additionally, if the data is loaded as a collection of maps, each map can be visualised as an image. The PointCloudAnnotator includes functionalities such as grouping points either by selecting individual points, selecting regions of points, or by boolean union of existing groups. Depending on the user’s needs, the selection of points can be done on the 3D-model, or on the map images. In addition to manual selection, the user is provided with tools for global or local segmentation of the point cloud, featuring various degrees of automation. The resulting groups are associated with a geometric primitive best approximating the shape of each group. Additionally, groups can be labelled by the user according to their semantic meaning. For a toolkit to fully accomplish the goals of improving the overall annotation process, the provided tools must not only be efficient but also easy to use, as the PointCloudAnnotator aims to achieve.