Contributions published at Department of Informatics (Burkhard Stiller)

This thesis delves into the convergence of the Internet of Things (IoT) and Blockchain technologies, focusing on the innovative application of these technologies within artwork transportation. The main goal is to introduce a system that capitalizes on IoT and blockchain to enhance the tracking and management of artwork during transportation processes. In pursuit of this goal, the study adopts a dual-pronged methodology. A comprehensive literature review provides a foundational understanding of the underlying principles. Subsequently, an applied research approach is employed, culminating in designing, implementing, and evaluating a prototype tailored to the intricacies of artwork transportation. The outcome of this thesis is ARTIS, a real-world prototype that effectively supports the targeted artwork tracking use case. However, it is acknowledged that further strides are needed to refine the prototype, particularly in safeguarding sensitive data and optimizing sensor accuracy. The significance of this work lies in its innovative amalgamation of IoT and blockchain technologies, presenting a novel avenue for addressing challenges in the artwork transportation domain. By demonstrating the feasibility of such a system, this thesis lays the groundwork for future endeavors to advance this concept into a production-ready solution.

This thesis aims to analyze the impact of occlusion on the quality of semantic segmentation methods for point cloud data. Occlusion is a prevalent phenomenon in 3D scenes, where objects often overlap or obstruct each other. This can significantly compromise the quality and integrity of data, leading to inaccuracies in semantic segmentation. While the issue of occlusion has garnered attention in 3D data processing, current research on how different occlusion levels impact the quality of semantic segmentation is rare. Specifically, there is a palpable gap in understanding how to quantify occlusion in the scene and how this characteristic influence the performance of advanced semantic segmentation software like the Minkowski Engine. To bridge the research gap, we proposed a novel metric to quantify the occlusion level of a scene. We then applied this metric to analyze the impact of occlusion on the quality of semantic segmentation methods for point cloud data. Our results show that the occlusion level of a scene has limited impact to the quality of semantic segmentation.

Non-uniform rational B-Spline surfaces (NURBS surface), a kind of parametric surface, are widely used in 3D modeling. This work explores NURBS surface reconstruction via the NURBS-Diff module. The NURBS-Diff module enables NURBS surfaces differentiable using the PyTorch framework. With supervised parameters, the module reconstructs the NURBS-based point cloud efficiently. This work introduces several pipelines by utilizing the NURBS-Diff module in unsupervised cases. The unsupervised pipelines make use of supersampling methods to obtain unstructured input and propose various metrics for point cloud and surface evaluation. The baseline unsupervised method is adapted from the original supervised pipeline. An extension of the NURBS-Diff module is also presented. The unsupervised pipelines are evaluated against the baseline. The pipelines serve as a stepping stone to further investigation into NURBS surface reconstruction based on unstructured input.

The Metaverse has become a buzz-phrase among tech businesses. Facebook's rebranding to Meta is symptomatic of this. Many firms and other actors are trying to shape visions of the Metaverse, leading to confusion about the term's meaning. We use social construction of technology (SCOT) theory to disentangle the conflicting notions proposing that what the Metaverse is and will become relies on the collective sensemaking processes. We point out similarities and differences between various concepts presented in the public media and link them to individual actors' monetary, political, or social motives. We describe the tensions that occur because of the conflicting interests. As the Metaverse is an emerging phenomenon, opportunities exist to reorient it toward humanist values rather than singular interests. However, the complexity of the social processes that shape the Metaverse requires a considerate approach rather than premature conclusions about the Metaverse’s characteristics. The analysis presents the Metaverse as a new, continually evolving sociotechnical phenomenon, and calls for research that explores it as a dynamic, moving target.

Information systems development (ISD) requires dynamic and flexible ways of working, particularly when developing requirements in collaboration with customers. Although prior research has acknowledged the importance of objects to support ISD practices, there has been a lack of frameworks to help discern the multiple overlapping roles objects play to support requirements development in a variety of ways throughout an ISD project. This paper explores and theorizes this phenomenon by leveraging multiplexity as a theoretical lens to analyze an extensive qualitative data set from a case study at a Swiss multinational banking software provider. Results show how diagrams and prototypes both play the roles of epistemic, activity, boundary, and infrastructure objects as a reflection of how they are used in requirements development. Our analysis articulates how two classical requirements specifications play multiple overlapping roles to support dynamic and flexible ISD practices. Based on these findings, we advance a framework for discerning the multiplex role of objects in practice.

We consider the “block withholding attack” as introduced by Eyal, where mining pools may infiltrate others to decrease their revenues. However, when two mining pools attack each other and neither controls a strict majority, the so-called miner’s dilemma arises. Both pools are worse off than without an attack. Knowing this, pools may make implicit non-attack agreements. Having said this, the miner’s dilemma is known to emerge only if no pool controls the majority of the mining power. In this work, we allow for miner migration and show that the miner’s dilemma emerges even for pools whose mining power exceeds 50%. We construct a game, where two mining pools attack each other and use simulation analysis methods to analyze the evolution the pools’ mining power, infiltration preferences and revenue densities under the influence of different mining pool sizes and miner migration preferences. The results show that underlying game experiences a phase transition fueled by miners’ migration preference. Without migration, it is profitable for a large mining pool to attack the other pool. The higher the migration preference of the miners, the more the game transitions into the miner’s dilemma and attacking makes both pools worse off. In a second step, we introduce solo-mining into the system. Introducing solo-mining cannot prevent the miner’s dilemma, however, it improves the efficiency of the mining process as the infiltration preferences of the mining pools are lowered. Thus, solo-mining has a control effect on the miner’s dilemma by keeping the infiltration preference below a certain threshold.

The relation between digitalization and environmental sustainability is ambiguous. There is potential of various digital technologies to slow down the transgression of planetary boundaries. Yet resource and energy demand for digital hardware production and use of data-intensive applications is of substantial size. The world over, there is no comprehensive regulation that addresses opportunities and risks of digital technology for sustainability. In this perspective article, we call for a Digital Green Deal that includes strong, cross-sectoral green digitalization policies on all levels of governance. We argue that a Digital Green Deal should first and foremost aim at greater policy coherence: Current digital policy initiatives should include measures that service environmental goals, and environmental policies must address risks and advance opportunities of digital technologies to spur sustainability transformations.

First-person view (FPV) drone racing is a televised sport in which professional competitors pilot high-speed aircraft through a 3D circuit. Each pilot sees the environment from the perspective of their drone by means of video streamed from an onboard camera. Reaching the level of professional pilots with an autonomous drone is challenging because the robot needs to fly at its physical limits while estimating its speed and location in the circuit exclusively from onboard sensors. Here we introduce Swift, an autonomous system that can race physical vehicles at the level of the human world champions. The system combines deep reinforcement learning (RL) in simulation with data collected in the physical world. Swift competed against three human champions, including the world champions of two international leagues, in real-world head-to-head races. Swift won several races against each of the human champions and demonstrated the fastest recorded race time. This work represents a milestone for mobile robotics and machine intelligence, which may inspire the deployment of hybrid learning-based solutions in other physical systems.

Algorithmic management can potentially improve healthcare delivery, for example, in community-based healthcare in low-and middle-income countries. However, most research neglects the user perspective and focuses on health-related outcomes. Consequently, we know little about the effects of algorithmic management on the user: community health workers. This paper reports on a 12-week pilot study in ComBaCaL, a community-based healthcare project tackling the increasing burden of non-communicable diseases (NCDs). We evaluate the Community Health Toolkit (CHT), a digital tool designed to support CHWs in community-based NCD care. We find that CHT is generally suitable for this purpose and can help CHWs to assume broader responsibilities. However, its design creates a tension between control and autonomy when confronted with reality. This tension could lead to disempowerment and attrition among CHWs. We propose design adaptations for CHT’s task scheduling, balancing the socio-technical system to resolve the tension between control and autonomy.

null

Current HCI research often focuses on mitigating algorithmic biases. While such algorithmic fairness during model training is worthwhile, we see fit to mitigate human cognitive biases earlier, namely during data entry. We developed a conversational agent with voice-based data entry and visualization to support financial consultations, which are human-human settings with information asymmetries. In a pre-study, we reveal data-entry biases in advisors by a quantitative analysis of 5 advisors consulting 15 clients in total. Our main study evaluates the conversational agent with 12 advisors and 24 clients. A thematic analysis of interviews shows that advisors introduce biases by “feeling” and “forgetting” data. Additionally, the conversational agent makes financial consultations more transparent and automates data entry. These findings may be transferred to various dyads, such as doctor visits. Finally, we stress that AI not only poses a risk of becoming a mirror of human biases but also has the potential to intervene in the early stages of data entry.

This paper presents Fedstellar, a platform for training decentralized Federated Learning (FL) models in heterogeneous topologies in terms of the number of federation participants and their connections. Fedstellar allows users to build custom topologies, enabling them to control the aggregation of model parameters in a decentralized manner. The platform offers a Web application for creating, managing, and connecting nodes to ensure data privacy and provides tools to measure, monitor, and analyze the performance of the nodes. The paper describes the functionalities of Fedstellar and its potential applications. To demonstrate the applicability of the platform, different use cases are presented in which decentralized, semi-decentralized, and centralized architectures are compared in terms of model performance, convergence time, and network overhead when collaboratively classifying hand-written digits using the MNIST dataset.

Artificial intelligence (AI) is becoming increasingly important as the amount of digital data grows. However, AI systems are often opaque and perceived as black boxes, which has a negative impact on user acceptance and trust. We see this in healthcare, where despite the great potential of AI, a lack of understanding and trust has held back physicians from adopting it. One way to address these issues is through Explainable AI (XAI), which focuses on understanding and interpreting AI behavior. In this thesis, we want to contribute to XAI by developing a visual analytics system called VisAIExplorer. We want to find out how an interactive visual analytics system can be designed to explain machine learning models to novice machine learning users, and what types of visualizations within the system can help to build understanding. The goal of VisAIExplorer is to explain the two models, logistic regression and hierarchical clustering, to novice machine learning users by providing various visualizations and support throughout the work process. The machine learning models are trained on a medical dataset about strokes, as healthcare professionals could benefit from a better understanding and increased trust in AI systems. By improving transparency and user support, VisAIExplorer aims to overcome the limitations of existing AI tools and promote more explainability in AI. The thesis includes a literature review, system development, evaluation, and suggestions for future improvements.

The Digital Humanities are increasingly employing computational methods for the curation of their research artifacts. The digitization of historical documents and their subsequent curation and annotation is common practice. The resulting digital editions often utilize a semi-structured data format to enhance the digitized research objects with annotations. Despite the presence of established annotation standards, annotation practices can still differ significantly within and across editions, resulting in considerable heterogeneity. This hampers the interoperability and reusability of digital editions. We contribute a visual analytics (VA) approach for the exploration of annotation practices within and across digital editions. We worked closely with the digital edition community to develop Tag Explorer, a VA tool tailored to their needs. Multiple coordinated views visualize annotation practices on various granularity levels, enabling users to better understand common practices and differences of editions stemming from heterogeneous sources. The users can adapt the visualizations to their information needs by delineating the exploration space and switching between different viewpoints. Tag Explorer fills a gap in the existing landscape of VA tools for the Digital Humanities, allowing the exploration of annotation strategies within and across heterogeneous digital editions. We evaluated our approach by two case studies with domain experts. Tag Explorer enabled the domain experts to check existing hypotheses, inspired potential improvements in their own editions, and uncovered unexpected findings regarding the annotation practices within and across digital editions. These insights help domain experts making more informed decisions during the annotation process, leading to more interoperable and reusable digital editions.

Deep neural networks have found great success in various recognition tasks. While their performances speak for themselves, they are still not fully understood. In particular, deep neural networks are susceptible to adversarial attacks. Research has found ways to defend against these attacks, one of the strategies being adversarial training, where networks are introduced to adversarial samples during training time. Another field where deep neural networks face problems are open-set recognition tasks, where the neural network has to address samples that do not belong to any known class. Some of the approaches addressing this problem incorporate samples, not belonging to any known class, whilst using a specific loss functions like the entropic open-set loss. The question remains if these two problems are somehow related to each other. Prior work suggested that open-set performance can be achieved by utilizing adversarial training. In this thesis we perform adversarial training on different types of loss functions, research these networks for adversarial stability, and evaluate their open-set recognition performances.

The use of convolutional neural networks (CNNs) in image classification tasks is a rapidly progressing field of research, including the classification of facial attributes. However, it is not yet completely understood how CNNs make decisions. To improve the transparency of the decision-making process and thus enhance interpretability and trustworthiness of CNNs, methods have been developed to visualize this process. In this thesis, we use the Gradient-weighted Class Activation Mapping (Grad-CAM) technique proposed by Selvaraju et al. (2017) to identify the regions of an image that the CNN uses for classification. This technique produces class-specific heatmaps that are intuitively interpretable. In order to evaluate the class activation maps, we define a set of masks, one for each of the 40 facial attributes that we examine. By using an approach called Acceptable Mask Ratio (AMR) we quantify how much of the activated area lies within the masked area. The higher the value of the AMR the more active is the CNN within the area that we expect, which usually corresponds to the location of the attribute being classified. We compare two different CNNs, one considers the class imbalance inherent to the data set (balanced CNN), and the other does not (unbalanced CNN). Our results show that overall the balanced CNN more often uses image regions that lie within the masked area. Furthermore, the results show an unexpected pattern for the unbalanced CNN namely for highly biased attributes the Grad-CAMs for the majority class show no activity at all.

Deep convolutional neural networks (CNNs) have become the state-of-the-art approach for image classification. While these networks are very effective at identifying the class to which an image belongs, they often do not properly learn the semantic relationship between classes. This means that models treat all misclassifications equally during training, regardless of the semantic distance between the predicted and actual class. This approach does not reflect the complexity of the real world, where some entities are more similar to each other, making mistakes between related classes less severe than those of unrelated classes. An architecture suited for hierarchical classification is presented as a potential solution to this problem. Rather than just predicting a single class, networks predict a simplified hierarchy consisting of higher-level concepts. This thesis explores how the architecture of CNNs can be adapted to incorporate hierarchical information to increase performance and the semantical conditioning of CNNs. The ultimate goal is to enhance the accuracy and robustness of image classification models by improving their understanding of the semantic relationships between classes, which could potentially lead to fewer and less severe misclassifications. To achieve this, several architectures are explored -- all using a ResNet backbone with classifiers for each hierarchical level -- that are compared with a baseline model that does not utilize the hierarchy for predictions. Most importantly, this thesis proposes an attention mechanism that does not contain any extra trainable parameters. This attention mechanism transforms the deep features given to a lower-level classifier based on the weight matrix from the higher-level classifier. This transformation aims to highlight features relevant to the classification of the higher-level concept, thus enabling the model to learn the decision boundary between classes of different higher-level concepts. This attention mechanism can effectively increase the classification accuracy for the ImageNet classes compared to a baseline architecture. Furthermore, when provided ground-truth information about the hierarchies from classes during training, it effectively learns the decision boundaries between classes from different higher-level concepts. This thesis also explores whether these architectures can be used for open-set classification. While showing some potential, the attention mechanism could likely be adapted for open-set classification, representing a promising possibility for future research.

Open-Set classification (OSC) addresses one of the core issues of traditional classification techniques, namely, the underlying closed-world assumption. The goal of OSC methods is to classify known classes correctly while also rejecting unknown classes. We propose two novel generic loss functions, Margin-OS and Margin-EOS, which combine the Entropic Open-Set and Objectosphere loss with margin-based loss functions used in face recognition tasks, CosFace and ArcFace, to learn discriminative features. We find that the margin has a positive effect on the closed-set accuracy but a mixed effect on the open-set performance. For applications that can tolerate high false positive rates, our losses improve the classification of known classes, but for low false positive rates the margin negatively impacts the training which leads to subpar classification of known samples.

Query optimisation remains an open problem in the field of database research. Inspired by the recent successes of reinforcement learning in various domains, adaptive approaches have emerged for addressing the problem. This thesis introduces a novel system called FRANTIC, which builds on recent advances and extends the adaptive approach to encompass factorised databases. By combining adaptivity and factorisation, FRANTIC outperforms competitors. Unlike previous research on factorised databases, which often assumed knowledge of a good factorised query plan, FRANTIC leverages reinforcement learning to efficiently explore the vast space of potential query plans, seeking effective execution strategies for queries. In addition to providing a performant implementation of FRANTIC, this thesis explores the system's inner workings in detail. Experiments reveal that design choices around the data partitioning, which is required for parallel processing, significantly impact the system's performance and even influence the effectiveness of different execution strategies. By shedding light on the interplay between the used join algorithm and data partitioning, robust heuristics are derived, enabling the reduction of the optimisation problem's search space. Consequently, this approach reduces the number of potential execution plans, making the optimisation problem more tractable.