Contributions published at Visualization and MultiMedia Lab (Renato Pajarola)

This thesis aims to analyze the impact of occlusion on the quality of semantic segmentation methods for point cloud data. Occlusion is a prevalent phenomenon in 3D scenes, where objects often overlap or obstruct each other. This can significantly compromise the quality and integrity of data, leading to inaccuracies in semantic segmentation. While the issue of occlusion has garnered attention in 3D data processing, current research on how different occlusion levels impact the quality of semantic segmentation is rare. Specifically, there is a palpable gap in understanding how to quantify occlusion in the scene and how this characteristic influence the performance of advanced semantic segmentation software like the Minkowski Engine. To bridge the research gap, we proposed a novel metric to quantify the occlusion level of a scene. We then applied this metric to analyze the impact of occlusion on the quality of semantic segmentation methods for point cloud data. Our results show that the occlusion level of a scene has limited impact to the quality of semantic segmentation.

Non-uniform rational B-Spline surfaces (NURBS surface), a kind of parametric surface, are widely used in 3D modeling. This work explores NURBS surface reconstruction via the NURBS-Diff module. The NURBS-Diff module enables NURBS surfaces differentiable using the PyTorch framework. With supervised parameters, the module reconstructs the NURBS-based point cloud efficiently. This work introduces several pipelines by utilizing the NURBS-Diff module in unsupervised cases. The unsupervised pipelines make use of supersampling methods to obtain unstructured input and propose various metrics for point cloud and surface evaluation. The baseline unsupervised method is adapted from the original supervised pipeline. An extension of the NURBS-Diff module is also presented. The unsupervised pipelines are evaluated against the baseline. The pipelines serve as a stepping stone to further investigation into NURBS surface reconstruction based on unstructured input.

This thesis presents an in-depth exploration of optimization algorithms aimed at addressing the challenging problem of unfolding 3D meshes by removing overlaps from initial unfoldings. Four distinct algorithms were selected for investigation: iterated local search (ILS), stochastic hill climbing (SHC), adaptive step size random search (ASSRS), and adaptive stochastic hill climbing (ASHC). Through implementation and experimentation, the performance of each algorithm was analyzed across varying mesh sizes and complexities. In the course of investigation, it became apparent that ILS struggled to deliver effective and efficient solutions, primarily due to its simplistic approach. ASSRS, a promising concept, faced challenges in its execution, with significant fail rates and a dependence on basic local search strategies. SHC, incorporating randomness to overcome local optima, demonstrated solid performance with success rates exceeding 93\% and competitive runtimes. Notably, ASHC emerged as the standout algorithm, enhancing SHC through adaptive probabilities of making unfavorable moves as overlap counts decrease. ASHC consistently outperformed the other algorithms, showcasing the potential of adaptiveness in computational unfolding. Comparison with related works revealed ASHC's competitive edge, outperforming simulated annealing and performing on par with a genetic algorithm. As a result, this thesis contributes valuable insights into the realm of 3D mesh unfolding optimization, paving the way for future refinements of ASHC and potential advancements in the unfolding of complex 3D structures.

Histograms are widely used because they are easy to implement and provide a simple overview of the underlying data. However, histograms are limited to two dimensions and thus not suited for multi-dimensional data. To resolve this, several models have been designed in the existing literature. These typically combine parallel coordinates plot (PCP) with histograms, so that they can represent multidimensional data. However, these existing models typically do not enable clustering of multivariate data or user interaction. To fill this gap, this thesis introduces a new "clustering PHP application" which offers a visual explorative framework with user interaction for the purpose of clustering. This application integrates PHP, Principal Component analysis (PCA), and scatter plots to merge their respective advantages. First, the PCA part offers ideas about variables such as how important they are and how they are related. Variables of interest can then be plotted on the PHP, which was adjusted for clustering (clustering PHP), to visually find relationships between variables. Axes on the clustering PHP can be reordered to focus on specific variables. Finally, a scatter plot helps users to observe local features and allows for the selection of principal components or variables. Interactions are immediately synchronized on the scatter plot and clustering PHP to detect data points sharing similarities on subspaces effortlessly. Overall this "clustering PHP application" thus helps users to determine clustering groups and improve clustering accuracy. In summary, "clustering PHP application" can help a user to explore data and make subspace clustering with complex multi-dimensional data more easy and efficient.

Visualization guidelines are one of the core elements of the visualization field. Guidelines pose an essential role for practitioners, researchers and students alike and therefore they should not just be accepted as they are proposed, but discussed, evolved and verified as thoroughly as possible. Currently, only a small amount of the proposed visualization guidelines go through this in-depth process, with the majority of them being the more prominent guidelines, such as the Data-Ink-Ratio or the Chart-Junk debate. While there are tools showcasing categorizations of visualization guidelines and forums for discussing them, there is no tool that acts as an accessible and easily navigable central source of knowledge for visualization guidelines. In the scope of this thesis, a tool is developed that provides guidance and context on visualization guidelines by providing additional information such as taxonomies and related scientific resources. To showcase the functionality of the tool, additional data required to provide guidance on visualization guidelines will be collected. The collected data is added to an existing collection of visualization guideline related data. The complete data will then be displayed on the VisKnowsBest tool to showcase a tool with the potential of acting as a central repository for visualization guidelines.

Point clouds are used in various algorithms in related fields, such as Visual Computing. Some of these algorithms require high-quality normals as input. Computing these surface normals is often the job of a different algorithm. Researchers have suggested vastly different algorithms over the years. Recent work focuses on using deep learning to estimate surface normals. It is important that these normal estimation algorithms perform well, as the other algorithms rely on their quality. This means the normal estimation algorithms need to be robust against various defects, including noise, outliers and differences in sampling density. Researches have proposed different measures to evaluate these normal estimation algorithms as well as how to produce synthetic test data to test against. Synthetic test data is especially useful as it provides ground truth normals to test against and can be constructed to contain a single or multiple defects with various strengths. Because of this, synthetic models are a valuable addition to real-world test data. In this paper, we evaluate four different normal estimation algorithms. This includes three regression based and one deep learning based method. We develop a tool to execute and compare the selected normal estimation algorithms. Our goal is to use good measures and cover multiple defects with our test data. In order to do this, we test the methods against various levels of noise and outliers. We use five different synthetic point cloud models and a real-world point cloud to test against. This allows us to identify the strengths and weaknesses of the tested methods.

Although influencer marketing has been experiencing tremendous growth, and many firms have been trying to find the right influencers for their marketing campaigns, there has been little research on identifying the timing of the high-impact work or analyzing the hot streak periods of influencers on social media. My thesis is among the first works in this area on the TikTok platform where I applied some methods used in previous literature. All the analyses were made using the video-level popularity metrics of TikTok: “number of likes”, “number of shares”, “number of plays”, and “number of comments”. In addition, unlike other studies, I created the visualization of the hot streak durations of TikTok authors to analyze information such as the start, and end time of hot streaks, or the length of hot streaks for each author. All plots for the analyses were built with bokeh- a Python library for interactive data visualizations. In conclusion, with the dataset in my thesis study, first I found that the timing of the most popular video is random in TikTok users’ lifecycles. Second, I discovered that the timing of the biggest hit and the second hit are close to each other, so TikTok authors may experience average success close to their most popular videos. Third, I detected a pattern indicating that TikTok authors use more diverse hashtags during hot streak periods than before them. Lastly, I observed that the relative hot streak length -success duration- of an author is usually between 12% and 50%. As a result, marketers can use these observations in my thesis to identify successful influencers on the TikTok platform. Finally, the formulas and visualization methods used in my thesis can be applied to a larger TikTok dataset or other datasets from other social media platforms such as Instagram, or YouTube.

Folding a discrete geometry from a flat sheet of material is one way to construct a 3D object. While nowadays for this purpose a lot of attention lays on 3D printing, folding can be a considerable alternative, complementing the possibilities 3D printing provides. A typical creation pipeline first designs the 3D object, unfolds it, prints and cuts the unfold pattern from a 2D material, and then refolds the object. Within this work we focus on the unfold part of this pipeline. Most current unfolding approaches segment the input, which has structural downsides for the refolded result. Therefore, we are aiming to unfold the input into a single-patched pattern. Our algorithm applies tabu search to the topic of unfolding. We show empirically that our algorithm is faster and more reliable than other methods unfolding into single-patched unfold-patterns. Moreover, our algorithm can handle any sort of flat polygon as faces, while comparable methods are bound to triangles.

Terrain rendering is a fundamental requirement when visualizing 3D geographic data in various research, commercial or personal ap- plications such as geographic information systems (GIS), 3D maps, simulators, and games. It entails handling large amounts of data for height and color as well as high-performance algorithms that can benefit from the parallel rendering power of GPUs. The main challenge is (1) to create a detailed renderable mesh using a fraction of the data that is most relevant to a specific camera position and orientation, and (2) to update this mesh in real time as the camera moves while keeping the transition artifacts low. Many algorithms have been proposed for adaptive adjustment of the level of detail (LOD) of large terrains. However, the existing web-based terrain rendering frameworks do not use state-of-the-art algorithms. As a result, these frameworks are prone to classic shortcomings of sim- pler terrain rendering algorithms such as discontinuities and limited visibility. We introduce a novel open-source web-based framework for rendering high quality terrains with adaptive LOD: Terrender. Terrender employs RASTeR, a modern LOD-based terrain rendering algorithm, while running smoothly with a limited bandwidth on all common web browsers, even on mobile devices. Finally, we present a thorough analysis of our system’s performance when the camera moves on a predefined trajectory. We also compare its performance and visual quality to another well-known framework.

Isolation forest is a popular anomaly detector due to its model-free structure and validity in detecting outliers across different types of datasets. This thesis presents solutions for two main issues in terms of isolation-based outlier detection algorithms. First, there is an adjustment to the isolation-based outlier detection model to improve the detection accuracy of Isolation Forest (iForest) and Extended Isolation Forest (EIF). The EIF is an extension of iForest, which addresses the block artifacts issue of iForest. Motivated by the failures of outlier detection on some real-world benchmark datasets by EIF, an adjusted EIF regarding the problem arising from the randomness of split hyperplane is presented. The outlier detection accuracy and precision of the adjusted EIF show that it is capable of enhancing the performance of both iForest and EIF. However, the drawback of the adjustment is that it is not time efficient. Second, this thesis proposes methods to generate a credible image presentation with outliers scattered in the relative areas based on isolation-based detection for multi-variate datasets. Inspired by the fact that neither iForest nor EIF can detect local outliers for each class in multi-class datasets, and few related works have been done in this direction, several class-wise detectors based on EIF and adjusted EIF are proposed in this thesis. By comparing the graphs, one of the methods achieves the best performance in providing insights for identifying potential outliers in clustering datasets.

Human digital traces (text, image, video) about weather events are a unique and novel resource that could potentially help for early detection and tracking of high-impact weather events. This thesis proposes the first step toward this ambitious goal by visually analyzing the Online citizens’ reports of the app MeteoSwiss of the Federal Office of Meteorology and Climatology MeteoSwiss. In October 2021, MeteoSwiss launched in this app a new feature, the Meteo reports. In this, users can report the weather around them and add pictures of the weather if they want. Based on these reports, we created the visualization analysis tool CitizenWeatherVA (Citizen report on Weather Condition Visual Analysis) to analyze them. With the help of the developed tool, based on selected use case events we then analyzed how the crowd reports describe weather events.

Atrial fibrillation (AF) is the most common type of arrhythmia occurring as an irregular excitation from the ventricles that affects the function of the heart and increases the risk of stroke and heart attack, leading to an extremely high mortality rate worldwide. Nearly half of all patients with AF are those with paroxysmal atrial fibrillation (PAF), and the chances of cure with medical intervention at this stage are very high compared to later stages. Therefore, the early detection and prediction of PAF are clinically important. However, due to the asymptomatic and interim episodic nature of PAF, its early detection and onset prediction have been challenging topics. Recent advances in the field of artificial intelligence, particularly machine learning techniques based on electrocardiogram(ECG) data, including deep learning, have enabled the development of PAF prediction. The goal of this work is to review the published studies related to PAF prediction, and predict the onset of PAF in advance with the shortened ECG signals and a high degree of accuracy. We systematically review the publications over the past 10 years, focusing on the prediction of PAF enabled by ECG-based machine learning models. Totally 15 studies of both traditional machine learning and deep learning models proposed over the past decade are covered and reviewed in this work. We propose a novel neural network framework PAFNet based on a convolution neural networks (CNN) with residual structure, a bi-directional Long-short-term memory network (LSTM) and the attention mechanism to predict the onset of PAF in advance. The PAFNet model is evaluated on public datasets and compared with previous studies with good performance. We describe the ideas behind the model design and analyze and discuss the results of the experiments. Finally we also discuss the potential future progress and challenges in the field.

Graffiti are omnipresent in today’s urban areas. A special form of graffiti are stencils which are mostly two-tone and easy to replicate. The research objective of the present thesis is to evaluate whether it is possible to automatically create stencils from an arbitrary image. While research on image abstraction is present in relevant literature, automatic stencil creation is understudied in scholarly work. In order to automatically create a stencil, a known algorithm is implemented and extended to create artistically pleasing stencils. The created stencils are then assessed using guidelines driven by research and expert input. Consequently, this work shows that the combination of already existing algorithms with carefully chosen parameters leads to the production of objectively wellmade stencils.

Displaying polygonal vector data is essential in various application scenarios such as geometry visualization, vector graphics rendering, CAD drawing and in particular geographic, or cartographic visualization. Dealing with static polygonal datasets that has a large scale and are highly detailed poses several challenges to the efficient and adaptive display of polygons in interactive geographic visualization applications. For linear vector data, only recently a GPU-based level-of-detail (LOD) polyline simplification and rendering approach has been presented which can perform locally-adaptive LOD visualization of large-scale line datasets interactively. However, locally optimized LOD simplification and interactive display of large-scale polygon data, consisting of filled vector line loops, remains still a challenge, specifically in 3D geographic visualizations where varying LOD over a scene is necessary. Our solution to this challenge is a novel technique for locally-optimized simplification and visualization of 2D polygons over a 3D terrain which features a parallelized point-inside-polygon testing mechanism. Our approach is capable of employing any simplification algorithm that sequentially removes vertices such as Douglas-Peucker and Wang-Müller. Moreover, we generalized our technique to also visualizing polylines in order to have a unified method for displaying both data types. The results and performance analysis show that our new algorithm can handle large datasets containing polygons composed of millions of segments in real time, and has a lower memory demand and higher performance in comparison to prior methods of line simplification and visualization.

Preterm new-borns have high chances to get long-term disabilities due to brain lesions. Near infrared tomography (NIROT) is a promising technique to diagnose brain lesions and detect abnormal brain oxygenation for those preterm infants. The traditional model-based methods applied on the data collected from NIROT is very time-consuming. However, neural networks can help to move the computational load to offline training, which reduces the computational time and output tissue property predictions on the infants' brain data. Our aim is to reduce the reconstruction time and achieve semi real-time imaging. We propose a sparse-input method which just uses 4 slices of the 2-D reconstructed image and assign them to the volume instead of using whole volume as input data. We trained the 3-D Unet with this input data and the corresponding ground truth and evaluated the performance of the segmentation. We used adapted random error as our evaluation metric. With several training trials with different data sets, it did reach a stable and good evaluation score. However, the segmentation performance was not satisfying. The 3-D Unet failed to classify the inclusions from the bulk.

The main goal of this work is to analyze the Montreux Jazz Festival video archive which was made accessible to the University of Zurich, Visualization and Multimedia Lab by ”Narratives from the long tail: Transforming access to audiovisual archives” project partners. The project has been funded by the Sinergia Grant of the Swiss National Foundation. Montreux Jazz Festival video archive dates back to 1967, keeps recordings of live performances of Montreux Jazz Festival, and is listed as a Memory of the World register by UNESCO. The problem that this thesis focuses on is the need for an online tool that will help music enthusiasts explore the archive and implement high-dimensional data analysis and visualization techniques to present different aspects of the existing recordings. The main proposed work is a web application with interactive visualizations to help the audience explore the Montreux Jazz Festival video archive. The web application is implemented by extending VIAN, a film analysis and visualization web application. To provide a high-dimensional data visualization tool, possible video and audio analysis methods were explored and a requirement analysis was performed to determine the important features. The selected features were extracted using neural networks, signal processing techniques, and audio analysis tools. A similarity metric was defined and implemented to compare different performance videos. Using this measure it is possible to see the outstanding songs which are significantly different than the others or to detect song clusters. To present the extracted data, attractive interactive visualizations were designed and implemented as a tool to analyze the visual and audio patterns of videos. Both an overview visualization and detailed performance analysis views are provided. The overview visualization presents the data clusters and similarities between different recordings and the detail view visualizes the extracted visual and audio features of a video. The final product is presented as multiple scripts to perform the feature extraction and the similarity analyses and a web application that includes interactive visualizations to help music enthusiasts explore the Montreux Jazz Festival archive.

The digital world and our economy rely on data, and blockchain is ideally suited for supplying this information. It delivers real-time, transparent, and fully sharable data stored on an immutable ledger that only authorized network users can access. At the same time, solutions are necessary due to the growing environmental threat caused by specific blockchain platforms, such as Bitcoin and Ethereum. The energy-intensive Proof-of-Work consensus algorithm is responsible for their high energy consumption and carbon footprint. This bachelor thesis examines the consensus mechanisms Proof-of-Work and Proof-of-Stake. This paper aims to investigate the transition from Proof-of-Work to Proof-of-Stake, emphasizing environmental sustainability. In addition, this thesis seeks to determine if there are any implications for developing and implementing blockchain platforms and oracles. The paper is written under the framework of the Blockchain Presence Project and Informatics Sustainability Research Group. A systematic literature review has been conducted to answer the research questions and understand why blockchain platforms intend to move to energy-efficient consensus protocols.

”What I do is the opposite of building walls. I build bridges. A bridge is something that connects instead of separating.” This statement by the world-famous architect Santiago Calatrava is to be turned on its head in this thesis. Even if the quotation is to be understood in a figurative sense, it is to be shown that walls not only separate but also create spaces and connect zones. Walls are to be reproduced in this project, creating a bridge between two technologies (laser scanner and Computer Aided Design (CAD) software). The captured point clouds are to be separated so that the spaces can be connected and provided for further processing. This thesis focuses on the vision of an automatic reconstruction of the built environment based on a point cloud captured by a laser scanner. Architectural plans that document the as-is state of a built environment are in great demand. Often they have to be re-recorded in the view of the fact that, for example, only old hand drawings are available in which later modifications were not documented or for the reason that blueprints have been lost. It is likely that digital plans have not been kept up to date or that old CAD data are no longer compatible with new versions. In such cases, it should be possible to reproduce new or updated plans as swiftly and precisely as possible. For this purpose, an autonomous tool was developed that offers an optimal interface between data acquisition and the usage in Building Information Modeling (BIM) applications. The presented pipeline aims to recognise and extract the most important structural elements and to model them for export. The process is characterised by the fact that no diversion is made by the creation of 2D planes, however, a direct 3D modelling is aimed at. The point clouds are acquired with the Leica BLK360 laser scanner and processed in C++ using the Point Cloud Library (PCL). In order to guarantee the integration and the connection to common CAD programs, an Industry Foundation Classes (IFC) file is created and provided.

Semantic information of 3D scans is a fundamental step for many scene understanding applications such as autonomous driving and 3D reconstruction. The goal of this work is to obtain the semantic segmentation of indoor point clouds using deep learning techniques. While semantic segmentation with deep learning of 3D point clouds still faces several challenges. In this work, instead of directly using deep learning on 3D point clouds, we first extract the panoramic images from 3D point clouds. These panoramic images are then used as input to a CNN model for semantic segmentation. Finally, we map the obtained segmentation panoramas to the 3D point cloud and develop a viewer interface to visualize the point cloud and its corresponding panoramic images. Therefore, the segmented point cloud can be analyzed in the interface visually.

3D reconstruction is a problem that has occupied the minds of computer scientists for decades, and working with 3D data is notoriously difficult because of the computational complexity and lack of data compared to other areas. With the rise of machine learning and especially deep learning, new ideas have been developed to tackle 3D reconstruction. One direction that has recently gained traction is called "implicit representations". The idea of implicit methods is to represent a signal, like 3D data, as an implicit function that is being approximated by a neuronal net. This thesis explores three selected papers about neural implicit representations, IM-NET, NDF, and SIREN. These methods are analyzed and compared theoretically, and experiments are conducted to gain insight into implicit representations for 3D reconstruction. In addition, a tool is created for analyzing the vector representations of shapes some methods generate.