Contributions published at Department of Informatics (Burkhard Stiller)

The Internet of Things (IoT) platform is one of the key drivers of the smart home market, having revolutionized the advancement of smart home technology. Besides the many benefits for convenience and efficiency, there are also concerns about security and privacy in such environments. The increasing complexity of smart homes and hardware limitations of individual devices necessitate the storage and processing of data in remote cloud environments. This raises privacy issues due to potential misuse or disclosure of sensitive information about residents. To the author's knowledge, no existing Privacy Enhancing Technology (PET) offers a lightweight approach to enforce privacy in smart home environments by combining existing tools into a unifying framework. The goal of this thesis is to take a first step towards an extensible open source software system that integrates into the smart home environment with the purpose of monitoring smart home device communications and controlling their communication behavior through user-defined policies. To this end, a prototype application is developed, which monitors smart home devices' Domain Name System (DNS) requests and enforces policies via a DNS sinkhole mechanism. The prototype system is deployed to a system-on-chip platform and evaluated in a live smart home environment to gain insight into the viability of the prototype. The aim is to examine the performance, effectiveness, and limitations of the prototype with the intention of validating the general approach. The results of these experiments indicate that the prototype successfully achieves the goals outlined in this thesis. The application prototype is capable of monitoring the network activity of smart home devices. The collected data are processed to gain insights and make this information transparent to the users. Furthermore, the prototype allows users to define simple allow/block policies which are subsequently enforced by the system.

This thesis presents an in-depth exploration of optimization algorithms aimed at addressing the challenging problem of unfolding 3D meshes by removing overlaps from initial unfoldings. Four distinct algorithms were selected for investigation: iterated local search (ILS), stochastic hill climbing (SHC), adaptive step size random search (ASSRS), and adaptive stochastic hill climbing (ASHC). Through implementation and experimentation, the performance of each algorithm was analyzed across varying mesh sizes and complexities. In the course of investigation, it became apparent that ILS struggled to deliver effective and efficient solutions, primarily due to its simplistic approach. ASSRS, a promising concept, faced challenges in its execution, with significant fail rates and a dependence on basic local search strategies. SHC, incorporating randomness to overcome local optima, demonstrated solid performance with success rates exceeding 93\% and competitive runtimes. Notably, ASHC emerged as the standout algorithm, enhancing SHC through adaptive probabilities of making unfavorable moves as overlap counts decrease. ASHC consistently outperformed the other algorithms, showcasing the potential of adaptiveness in computational unfolding. Comparison with related works revealed ASHC's competitive edge, outperforming simulated annealing and performing on par with a genetic algorithm. As a result, this thesis contributes valuable insights into the realm of 3D mesh unfolding optimization, paving the way for future refinements of ASHC and potential advancements in the unfolding of complex 3D structures.

Histograms are widely used because they are easy to implement and provide a simple overview of the underlying data. However, histograms are limited to two dimensions and thus not suited for multi-dimensional data. To resolve this, several models have been designed in the existing literature. These typically combine parallel coordinates plot (PCP) with histograms, so that they can represent multidimensional data. However, these existing models typically do not enable clustering of multivariate data or user interaction. To fill this gap, this thesis introduces a new "clustering PHP application" which offers a visual explorative framework with user interaction for the purpose of clustering. This application integrates PHP, Principal Component analysis (PCA), and scatter plots to merge their respective advantages. First, the PCA part offers ideas about variables such as how important they are and how they are related. Variables of interest can then be plotted on the PHP, which was adjusted for clustering (clustering PHP), to visually find relationships between variables. Axes on the clustering PHP can be reordered to focus on specific variables. Finally, a scatter plot helps users to observe local features and allows for the selection of principal components or variables. Interactions are immediately synchronized on the scatter plot and clustering PHP to detect data points sharing similarities on subspaces effortlessly. Overall this "clustering PHP application" thus helps users to determine clustering groups and improve clustering accuracy. In summary, "clustering PHP application" can help a user to explore data and make subspace clustering with complex multi-dimensional data more easy and efficient.

The purpose of this thesis is the development and evaluation of a spoken pedagogical conversational agent (PCA) which can carry out complete telephone calls in the context of a corporate identity (CI) training. The term CI refers the overall self-image of an organization regarding its corporate personality and reflects how an organization is perceived by both internal and external stakeholders. One aspect of the overarching study about CI in public administrations of two German communes is service on the telephone, which aims at giving employees an opportunity to train and improve their skills in different areas such as active listening, issue solving, or de-escalation on the telephone. Besides the implementation of effective telephone exercises, the focus is on the personalization abilities of the PCA such as complete simulated conversations and feedback generation, which is accomplished by making use of recent language models such as the state-of-the-art ChatGPT model by OpenAI. How these models can be used in this organizational context is a relatively new area of study and not much research exists about this yet. Furthermore, the effects of personalization on the users such as social presence and interaction quality is examined. To accomplish this, requirements are established based on an iterative process considering stakeholder input and related work surrounding institutional talk, workplace learning, and conversational agents. Then, the technologies are selected consisting of different components such as speech-to-text and text-to-speech engines. During development a major focus is on prompt engineering which aims at providing optimal instructions for the language model to generate an optimal response. Finally, a functioning PCA is implemented and deployed in a CI training with real employees of two public administrations. Following the training, an evaluation is carried out consisting of different surveys and interviews, which are used in the analysis of the PCA. The results show that a PCA in this context can be personalized in various ways by utilizing novel language models such as ChatGPT. Furthermore, the evidence indicates that the personalization can potentially lead to an increased social presence and higher interaction quality. In a final step, the personalization of the PCA, problematic aspects, and challenges are discussed, resulting in derived design principles. The further development of the PCA can make use of these insights as a foundation regarding different technical and sociotechnical aspects.

Patient-centered care enables physicians to understand their patients as people with individual needs. It helps the patient to be informed, respected, and involved in decisions about their treatment. Thus, it should be the standard approach in the medical practice. However, with the current increase in people suffering from chronic diseases, the healthcare system reaches its limits and new forms of treatment have to be explored. Technology can pose a relief both for the physician and the patient. During a consultation, technological means can help recall knowledge and assist in the decision-making process. At home, it can support the patient to adhere to the treatment and provide motivation. Yet currently those two aspects are often disconnected from each other, and no exchange of data happens between the technologies used at home and the ones in the medical practice. Additionally, the provision of patient-centered care might suffer from the involvement of technology in the consultation and lead to the further scattering of information about the patient. The Digital Companion Project aims to close the loop between obesity consultations and improve the connection between physician and patient. The relevant data gathered by the patient at home can be accessed by the physician and the patient receives the information discussed during the consultation on their device. This bachelor thesis analyzes the use of the Digital Companion in a field study with twenty-seven patients and six physicians to figure out if proper patient-centered care was provided. Furthermore, emerging practices regarding patient-centered care and the influence of the device on the consultation were observed. The results show that the Digital Companion could improve the provision of patient-centered care in all aspects. It helped involve the patients in the decision-making and led to a formulation of a realistic treatment plan. Trust was established quickly, and the patients were openly sharing personal details about their lives. During the consultation, the Digital Companion worked as calm technology, did not disrupt the conversation, and did not attract unnecessary attention.

We present ManuKnowVis, the result of a design study, in which we contextualize data from multiple knowledge repositories of a manufacturing process for battery modules used in electric vehicles. In data-driven analyses of manufacturing data, we observed a discrepancy between two stakeholder groups involved in serial manufacturing processes: Knowledge providers (e.g., engineers) have domain knowledge about the manufacturing process but have difficulties in implementing data-driven analyses. Knowledge consumers (e.g., data scientists) have no first-hand domain knowledge but are highly skilled in performing data-driven analyses. ManuKnowVis bridges the gap between providers and consumers and enables the creation and completion of manufacturing knowledge. We contribute a multi-stakeholder design study, where we developed ManuKnowVis in three main iterations with consumers and providers from an automotive company. The iterative development led us to a multiple linked view tool, in which, on the one hand, providers can describe and connect individual entities (e.g., stations or produced parts) of the manufacturing process based on their domain knowledge. On the other hand, consumers can leverage this enhanced data to better understand complex domain problems, thus, performing data analyses more efficiently. As such, our approach directly impacts the success of data-driven analyses from manufacturing data. To demonstrate the usefulness of our approach, we carried out a case study with seven domain experts, which demonstrates how providers can externalize their knowledge and consumers can implement data-driven analyses more efficiently.

Item rankings are useful when a decision needs to be made, especially if there are multiple attributes to be considered. However, existing tools do not support both categorical and numerical attributes, require programming expertise for expressing preferences on attributes, do not offer instant feedback, lack flexibility in expressing various types of user preferences, or do not support all mandatory steps in the ranking-creation workflow. In this work, we present RankASco: a human-centered visual analytics approach that supports the interactive and visual creation of rankings. The iterative design process resulted in different visual interfaces that enable users to formalize their preferences based on a taxonomy of attribute scoring functions. RankASco enables broad user groups to (a) select attributes of interest, (b) express preferences on attributes through interactively tailored scoring functions, and (c) analyze and refine item ranking results. We validate RankASco in a user study with 24 participants in comparison to a general purpose tool. We report on commonalities and differences with respect to usefulness and usability and ultimately present three personas that characterize common user behavior in ranking-creation. On the human factors side, we have also identified a series of interesting behavioral variables that have an influence on the task performance and may shape the design of human-centered ranking solutions in the future.

Internet-of-Things (IoT) devices have become critical assets to be protected due to increased adoption for emerging use cases. As such, these devices are confronted with a myriad of malware-based threats. To combat malware in IoT, Moving Target Defense (MTD) is a viable defense layer, since MTD does not rely on a low breach probability - aiming to increase security in a dynamic way. Although evidence supports the usefulness of MTD for IoT, the current state of the art suffers from unrealistic deployments, including the problem of operating multiple MTD techniques. Especially, there is a commonly observed gap in determining and deploying one of a set of locally available MTD techniques. This paper addresses this gap by relying on a rule-based selection mechanism. For that, a risk-driven methodology to establish this selection agent with a well-defined architecture is followed. As an input, the device's behavior, as expressed through its resource consumption, serves as a selection criterion. This architecture was implemented for a Raspberry Pi and evaluated against seven malware, given four existing MTD techniques. The resulting prototype highlights that a rule-based system can efficiently mitigate the malware samples.

Motivation Social media platforms centered around content creators (CCs) faced rapid growth in the past decade. Currently, millions of CCs make livable incomes through platforms such as YouTube, TikTok, and Instagram. As such, similarly to the job market, it is important to ensure the success and income (usually related to the follower counts) of CCs reflect the quality of their work. Since quality cannot be observed directly, two other factors govern the network-formation process: (a) the visibility of CCs (resulted from, e.g., recommender systems and moderation processes) and (b) the decision-making process of seekers (i.e., of users focused on finding CCs). Prior virtual experiments and empirical work seem contradictory regarding fairness: While the first suggests the expected number of followers of CCs reflects their quality, the second says that quality does not perfectly predict success. Results Our paper extends prior models in order to bridge this gap between theoretical and empirical work. We (a) define a parameterized recommendation process which allocates visibility based on popularity biases, (b) define two metrics of individual fairness (ex-ante and ex-post), and (c) define a metric for seeker satisfaction. Through an analytical approach we show our process is an absorbing Markov Chain where exploring only the most popular CCs leads to lower expected times to absorption but higher chances of unfairness for CCs. While increasing the exploration helps, doing so only guarantees fair outcomes for the highest (and lowest) quality CC. Simulations revealed that CCs and seekers prefer different algorithmic designs: CCs generally have higher chances of fairness with anti-popularity biased recommendation processes, while seekers are more satisfied with popularity-biased recommendations. Altogether, our results suggest that while the exploration of low-popularity CCs is needed to improve fairness, platforms might not have the incentive to do so and such interventions do not entirely prevent unfair outcomes.

Comprehending communication is dependent on analyzing the different modalities of conversation, including audio, visual, and others. This is a natural process for humans, but in digital libraries, where preservation and dissemination of digital information are crucial, it is a complex task. A rich conversational model, encompassing all modalities and their co-occurrences, is required to effectively analyze and interact with digital information. Currently, the analysis of co-speech gestures in videos is done through manual annotation by linguistic experts based on textual searches. However, this approach is limited and does not fully utilize the visual modality of gestures. This paper proposes a visual gesture retrieval method using a deep learning architecture to extend current research in this area. The method is based on body keypoints and uses an attention mechanism to focus on specific groups. Experiments were conducted on a subset of the NewsScape dataset, which presents challenges such as multiple people, camera perspective changes, and occlusions. A user study was conducted to assess the usability of the results, establishing a baseline for future gesture retrieval methods in real-world video collections. The results of the experiment demonstrate the high potential of the proposed method in multimodal communication research and highlight the significance of visual gesture retrieval in enhancing interaction with video content. The integration of visual similarity search for gestures in the open-source multimedia retrieval stack, vitrivr, can greatly contribute to the field of computational linguistics. This research advances the understanding of the role of the visual modality in co-speech gestures and highlights the need for further development in this area.

Community detection is widely used in social networks to uncover groups of related vertices (nodes). In cryptocurrency transaction networks, community detection can help identify users that are most related to known illegal users. However, there are challenges in applying community detection in cryptocurrency transaction networks: (1) the use of pseudonymous addresses that are not directly linked to personal information make it difficult to interpret the detected communities; (2) on Bitcoin, a user usually owns multiple Bitcoin addresses, and nodes in transaction networks do not always represent users. Existing works on cluster analysis on Bitcoin transaction networks focus on addressing the later using different heuristics to cluster addresses that are controlled by the same user. This research focuses on illegal community detection containing one or more illegal Bitcoin addresses. We first investigate the structure of Bitcoin transaction networks and suitable community detection methods, then collect a set of illegal addresses and use them to label the detected communities. The results show that 0.06% of communities from daily transaction networks contain one or more illegal addresses when 2,313,344 illegal addresses are used to label the communities. The results also show that distance-based clustering methods and other methods depending on them, such as network representation learning, are not suitable for Bitcoin transaction networks while community quality optimization and label-propagation-based methods are the most suitable.

The problem of identifying the source of an epidemic (also called patient zero) given a network of contacts and a set of infected individuals has attracted interest from a broad range of research communities. The successful and timely identification of the source can prevent a lot of harm as the number of possible infection routes can be narrowed down and potentially infected individuals can be isolated. Previous research on this topic often assumes that it is possible to observe the state of a substantial fraction of individuals in the network before attempting to identify the source. We, on the contrary, assume that observing the state of individuals in the network is costly or difficult and, hence, only the state of one or few individuals is initially observed. Moreover, we presume that not only the source is unknown, but also the duration for which the epidemic has evolved. From this more general problem setting a need to query the state of other (so far unobserved) individuals arises. In analogy with active learning, this leads us to formulate the active querying problem. In the active querying problem, we alternate between a source inference step and a querying step. For the source inference step, we rely on existing work but take a Bayesian perspective by putting a prior on the duration of the epidemic. In the querying step, we aim to query the states of individuals that provide the most information about the source of the epidemic, and to this end, we propose strategies inspired by the active learning literature. Our results are strongly in favor of a querying strategy that selects individuals for whom the disagreement between individual predictions, made by all possible sources separately, and a consensus prediction is maximal. Our approach is flexible and, in particular, can be applied to static as well as temporal networks. To demonstrate our approach’s practical importance, we experiment with three empirical (temporal) contact networks: a network of pig movements, a network of sexual contacts, and a network of face-to-face contacts between residents of a village in Malawi. The results show that active querying strategies can lead to substantially improved source inference results as compared to baseline heuristics. In fact, querying only a small fraction of nodes in a network is often enough to achieve a source inference performance comparable to a situation where the infection states of all nodes are known.

As part of the 6th Lifelog Search Challenge, this paper presents an approach to arrange Lifelog data in a multi-modal knowledge graph based on cluster hierarchies. We use multiple sequence clustering approaches to address the multi-modal nature of Lifelogs in relation to temporal, spatial, and visual factors. The resulting clusters, along with semantic metadata captions and augmentations based on OpenCLIP, provide for the semantic structure of a graph including all Lifelogs as entries. Textual queries on this hierarchical graph can be expressed to retrieve individual Lifelogs, as well as clusters of Lifelogs.

Personal lifelog data collections are becoming more common as a memory aid, as well as for analytical tasks, such as health and fitness analysis. Due to the multimodal and personal nature of lifelog data, interactive multimedia retrieval approaches are required to facilitate flexible and iterative query formulation and result exploration for retrieval and analysis. In recent years, novel user interface modalities have emerged, that allow new ways for users to interact with a retrieval system. Virtual reality, one such new modality, provides advantages as well as challenges for interactive multimedia retrieval in comparison to conventional desktop-based interfaces. This paper describes a novel desktop-virtual reality hybrid system participating in the Lifelog Search Challenge 2023. The system, which is based on the components of the vitrivr stack, is described with a focus on query formulation in the web-based desktop user interface vitrivr-ng, and result exploration in the virtual reality-based vitrivr-VR.

The war in Ukraine and the following refugee crisis have recently again highlighted the need for effective refugee management across European countries. Refugee management contemporarily mostly relies on top-down management approaches by governments. These often lead to suboptimal policies for refugees and highlight a need to better identify and integrate refugee needs into management. Here, we show that modern applications of Natural Language Processing (NLP) allow for the effective analysis of large text corpora linked to refugee needs, making it possible to complement top-down approaches with bottom-up knowledge centered around the current needs of the refugee population. By following a Design Science Research Methodology, we utilize 58 semi-structured stakeholder interviews within Switzerland to develop design requirements for NLP applications for refugee management. Based on the design requirements, we developed R2G – “Refugees to Government”, an application based on state-of-the-art topic modeling to identify refugee needs bottom-up through Telegram data. We evaluate R2G with a dedicated workshop held with stakeholders from the public sector and civil society. Thus, we contribute to the ongoing discourse on how to design refugee management applications and showcase how topic modeling can be utilized for data-driven governance during refugee crises.

Visual-inertial odometry (VIO) is the most common approach for estimating the state of autonomous micro aerial vehicles using only onboard sensors. Existing methods improve VIO performance by including a dynamics model in the estimation pipeline. However, such methods degrade in the presence of low-fidelity vehicle models and continuous external disturbances, such as wind. Our proposed method, HDVIO, overcomes these limitations by using a hybrid dynamics model that combines a point-mass vehicle model with a learning-based componentthat captures complex aerodynamic effects. HDVIO estimates the external force and the full robot state by leveraging the discrepancy between the actual motion and the predicted motion of the hybrid dynamics model. Our hybrid dynamics model uses a history of thrust and IMU measurements to predict the vehicle dynamics. To demonstrate the performance of our method, we present results on both public and novel drone dynamics datasets and show real-world experiments of a quadrotor flying in strong winds up to 25 km/h. The results show that our approach improves the motion and external force estimation compared to the state-of-the-art by up to 33% and 40%, respectively. Furthermore, differently from existing methods, we show that it is possible to predict the vehicle dynamics accurately while having no explicit knowledge of its full state.

This paper explores the public value outcomes generated by collaborative platforms in public services through a comparative case study of 25 privately-owned digital platforms across various public service areas in Italy. The study confirms that collaborative digitally-enabled endeavors can have both positive and negative effects on public value, highlighting the challenges associated with multi-actor dynamics and the delicate balance between public interest and private gain in public services. The findings reveal implications related to user-centric platforms, including concerns about inclusivity, autonomy, decision-making abilities, and privacy infringement. Additionally, the study suggests that collaborative platforms alone do not enhance collaboration in public services unless accompanied by strong public governance that promotes interoperability through standardized frameworks, common templates, and data reuse.

Traditional explanatory resources, such as user manuals and textbooks, often contain content that may not cater to the diverse backgrounds and information needs of users. Yet, developing intuitive, user-centered methods to effectively explain complex or large amounts of information is still an open research challenge. In this paper we present ExplanatoryGPT, an approach we devised and implemented to transform textual documents into interactive, intelligent resources, capable of offering dynamic, personalized explanations. Our approach uses state-of-the-art question-answering technology to generate on-demand, expandable explanations, with the aim of allowing readers to efficiently navigate and comprehend static materials. ExplanatoryGPT integrates ChatGPT, a state-of-the-art language model, with Achinstein’s philosophical theory of explanations. By combining question generation and answer retrieval algorithms with ChatGPT, our method generates interactive, user-centered explanations, while mitigating common issues associated with ChatGPT, such as hallucinations and memory shortcomings. To showcase the effectiveness of our Explanatory AI, we conducted tests using a variety of sources, including a legal textbook and documentation of some health and financial software. Specifically, we provide several examples that illustrate how ExplanatoryGPT excels over ChatGPT in generating more precise explanations, accomplished through thoughtful macro-planning of explanation content. Notably, our approach also avoids the need to provide the entire context of the explanation as a prompt to ChatGPT, a process that is often not feasible due to common memory constraints.

Detecting and solving sleep disorders can significantly impact society and the economy in general. The polysomnogram is the gold standard exam for diagnosing sleep disorders. Manually annotating the patient's sleep has limitations, including its time-consuming and tedious nature, lack of reliability, sensitivity to the setup of different clinics, and motion noise. This work tests the ability of neural network models to be faster and more reliable than manual scoring by incorporating temporal information in the training setting and changing the model architecture. The study concentrates on algorithms that are robust to the setup of different clinics and fair to diverse populations, using an intelligent combination of the most used datasets in experimental settings: the Sleep-EDF and the MASS datasets. We first analyze the ability of the automated classifier to handle data from different sleep centers and patient groups by experimentally testing loss functions and other crucial model parameters across datasets. Then, we incorporate temporal context in the data samples by concatenating previous sleep epochs to the current sample. We show that our model trained on longer temporal context performs equally to many of the analyzed manual sleep stage scoring conducted by expert technicians and is superior to some state-of-the-art models we analyzed.

Maintaining multiple complex queries on large, dynamic datasets in real time poses a major challenge. Thus, this thesis introduces CAscading VIEw tRees (CaVieR), an extension to Factorized Incremental View Maintenance (F-IVM) that addresses this challenge. CaVieR adds a method to F-IVM that joins the view trees of multiple conjunctive queries into a directed acyclic graph (DAG). This can efficiently handle updating and enumerating a set of Cascading Q-Hierarchical Queries. Reducing redundant query maintenance significantly reduces computational workload and can even reduce the theoretical asymptotic complexity of maintaining Cascading Q-Hierarchical Queries. The algorithm was tested against F-IVM on synthetic and real-world datasets with different sets of Cascading Q-Hierarchical Queries. In experiments, the two main performance indicators in IVM were measured: update time and enumeration delay. While in most scenarios, a significant improvement in both measurements was observed, it was found that F-IVM, when using batch updates and ordered input relation streams, can outperform CaVieR. To verify that this is the only scenario where individually maintaining the view trees is better than joining them, various parameters were tested, and their influence on update time and enumeration delay was recorded. While parameters like the cardinality of the input relations and the number of free vari- ables significantly influenced the update time and enumeration delay, CaVieR outperformed F-IVM consistently. Thus this thesis not only presents a new method of optimization to F-IVM that can decrease the asymptotic complexity of the problem, including theoret- ical proofs of correctness and completeness. It also includes an implementation and experiments to estimate its performance impact.