Contributions published at Department of Informatics (Burkhard Stiller)

Quantifying the security of the consensus mechanism is usually complicated and performed in either a theoretical or computational fashion. The advantage of the Proof of Work (PoW) consensus mechanism is that its Sybil resistance mechanism makes it computationally difficult for an attacker to perform an attack. The attacker needs enough total hashing power to break the security of PoW. whereas the security of the Proof of Stake (PoS) consensus mechanism is associated with measurable rewards or penalties on the initial deposit and is mostly theoretically proven. Ethereum’s consensus transition from PoW to PoS brought many attack vectors to light, as described in the papers (Neuder et al. 2021) (D’Amato et al. 2022) (Schwarz-Schilling, Neu, et al. 2021). Most of these attack vectors are mitigated before the transition. However, many new attack vectors might be seen in upcoming upgrades. We employed agent-based modeling to model network behavior computationally to help with modeling the attacks and their mitigations. Our experiment imitates network behavior using a sleek and sublime representation of a participant in the Ethereum PoS consensus. We performed experiments to test the network behavior under various average information propagation delays and to test the network against an ex-ante reorg attack and its mitigation. The agent in our experiment can model network effects under different stake distributions between honest and byzantine, numerous network topologies, and a broad variety of information transmission latencies. This report goes into detail about some fascinating observations regarding the average block transmission, the ex-ante reorg attack mitigation impact on the block tree evolution under Gasper consensus, as well as the rate of adversary block finality and the success rate of the ex-ante reorg attack based on the block timeliness.

Unmanned Aerial Vehicles (UAVs), commonly known as drones, have revolutionized various industries, including agriculture, photography, delivery, and security. The UAV's ability to fly autonomously and perform various missions with ease is largely attributed to the advancement in vision algorithms. However, as these UAVs become more prevalent in civilian airspace, their reliability and security become crucial concerns. One of the key components of a UAV is its onboard stereo camera, which enables the UAV to navigate through its environment. However, stereo cameras are vulnerable to vision-based security attacks, which can cause the UAV to crash or malfunction. The safety and reliability of UAVs heavily depend on the performance of their vision-based navigation systems. To ensure that these systems are robust and secure, it is essential to evaluate their resilience to different types of attacks and identify potential vulnerabilities. In order to address this issue, a platform was developed that can inject vision-based adversarial attacks into the UAV system to determine its vulnerability. This platform, called AerialShield, is an extension of Aerialist and is capable of carrying out different kinds of vision-based adversarial attacks on a UAV platform. AerialShield generates several adversarial test cases by mutating important parameters to attack the system. Through experiments, it was found that the PX4 Avoidance system, which is used for obstacle avoidance, is prone to adversarial attacks. The experiments conducted by AerialShield showed that the PX4 Avoidance system is very sensitive to even a little noise in the stereo camera in a real-world-like simulated environment, leading to crashes. Moreover, the UAV was found to be less resilient to noise at a lower altitude as compared to a higher altitude. This highlights the importance of testing UAVs in various environments and altitudes to ensure their reliability and security. Our experiments have shown that several factors, such as UAV altitude, environmental complexity, and the level of noise injected into the camera, can significantly impact the system's performance. While UAVs offer numerous benefits, their reliability and security are critical for their safe integration into civilian airspace. AerialShield's ability to carry out vision-based adversarial attacks on UAVs provides valuable insight into their vulnerability, allowing for improvements to be made to ensure their safe and secure operation.

The rapid development of technology requires architectures that can adapt to this high demand. Using mesh networks can provide a dynamic and cost-efficient way for horizontal scalability. Although mesh networks have many advantages, these networks lose the ability to monitor traffic reliably. Traffic monitoring on mesh networks would be beneficial for network operators. It would reduce the opportunity cost of adapting mesh networks compared to traditional network topologies. In this thesis, a new protocol will try to provide a new approach to traffic accounting. The Cascade Encryption Protocol (CEC) will couple monitoring- and data traffic utilizing encryption to encapsulate routing information in the packets. The CEC protocol was evaluated using simulations on a larger scale on the Abilene topology for a real-life-like environment and line topology on a smaller scale to determine created overhead. The analysis showed a correlation between the overhead and the number of flows on the small-scale tests. Furthermore, the protocol’s performance stays steady on a fully utilized network and degrades slowly as the number of flows increases.

Online reviews have become a ubiquitous source of information for consumers, and response authors play a critical role in addressing customer feedback. However, the vast amount and complexity of online reviews pose a significant challenge for those who respond to them. By implementing artificial intelligence (AI), the possibility of enhancing both the efficiency and quality of online review responses arises. In this thesis, it is examined how the use of various degrees of AI assistance affects the performance of response authors across a variety of efficiency and quality metrics. In order to address this question, a survey was conducted to gather data on the perceived quality of review responses composed by novice and professional authors in four distinct settings: without any AI support (B setting), with partial AI support (I setting), with AI-powered response generator support (G setting), and with fully automated AI-generated responses (G-AI setting). In addition, data on the efficiency of the response authors was recorded during writing sessions in each setting. The results show that the use of AI significantly improves the efficiency of response authors. In particular, the G setting greatly reduced the writing time for both professional and novice authors. Furthermore, the use of GPT-3, an advanced AI language model, resulted in significantly higher quality responses than those which a competing AI system and most of the other work configurations were able to produce. Based on these findings, a work configuration is proposed which combines the strengths of AI systems with human authors to optimize the online review response process. This proposed configuration aims to maximize efficiency and response quality while minimizing the workload on human authors. In conclusion, this thesis provides valuable insights into the potential of AI support to enhance the performance of response authors in the context of online reviews. The proposed optimal work configuration provides a practical solution for businesses and individuals looking to optimize their response authoring process.

Model Predictive Control (MPC) has become a popular framework in embedded control for high-performance autonomous systems. However, to achieve good control performance using MPC, an accurate dynamics model is key. To maintain real-time operation, the dynamics models used on embedded systems have been limited to simple first-principle models, which substantially limits their representative power. In contrast to such simple models, machine learning approaches, specifically neural networks, have been shown to accurately model even complex dynamic effects, but their large computational complexity hindered combination with fast real-time iteration loops. With this work, we present Real-time Neural MPC , a framework to efficiently integrate large, complex neural network architectures as dynamics models within a model-predictive control pipeline. Our experiments, performed in simulation and the real world onboard a highly agile quadrotor platform, demonstrate the capabilities of the described system to run learned models with, previously infeasible, large modeling capacity using gradient-based online optimization MPC. Compared to prior implementations of neural networks in online optimization MPC we can leverage models of over 4000 times larger parametric capacity in a 50 Hz real-time window on an embedded platform. Further, we show the feasibility of our framework on real-world problems by reducing the positional tracking error by up to 82% when compared to state-of-the-art MPC approaches without neural network dynamics.

Over the last decade, transparency reports have been adopted by most large information technology companies. These reports provide important information on the requests tech companies receive from state actors around the world and the ways they respond to these requests, including what content the companies remove from platforms they own. In theory, such reports shall make inner workings of companies more transparent, in particular with respect to their collaboration with state actors. They shall also allow users and external entities (e.g., researchers or watchdogs) to assess to what extent companies adhere to their own policies on user privacy and content moderation as well as to the principles formulated by global entities that advocate for the freedom of expression and privacy online such as the Global Network Initiative or Santa Clara Principles. However, whether the current state of transparency reports actually is conducive to meaningful transparency remains an open question. In this paper, we aim to address this through a critical comparative analysis of transparency reports using Santa Clara Principles 2.0 (SCP 2.0) as the main analytical framework. Specifically, we aim to make three contributions: first, we conduct a comparative analysis of the types of data disclosed by major tech companies and social media platforms in their transparency reports. The companies and platforms analyzed include Google (incl. YouTube), Microsoft (incl. its subsidiaries Github and LinkedIn), Apple, Meta (prev. Facebook), TikTok, Twitter, Snapchat, Pinterest, Reddit and Amazon (incl. subsidiary Twitch). Second, we evaluate to what degree the released information complies with SCP 2.0 and how it aligns with different purposes of transparency. Finally, we outline recommendations that could improve the level of transparency within the reports and beyond, and contextualize our recommendations with regard to the Digital Services Act (DSA) that received the final approval of the European Council in October 2022.

Digital agents are considered a general-purpose technology. They spread quickly in private and organizational contexts, including education. Yet, research lacks a conceptual framing to describe interaction with such agents in a holistic manner. While focusing on the interaction with a pedagogical agent, that is, a digital agent capable of natural-language interaction with a learner, we propose a model of learning activity based on activity theory. We use this model and a review of prior research on digital agents in education to analyze how various characteristics of the activity, including features of a pedagogical agent or learner, influence learning outcomes. The analysis leads to identification of information systems research directions and guidance for developers of pedagogical agents and digital agents in general. We conclude by extending the activity theory-based model beyond the context of education and show how it helps designers and researchers ask the right questions when creating a digital agent.

vitrivr is a general purpose retrieval system that supports a wide range of query modalities. In this paper, we briefly introduce the system and describe the changes and adjustments made for the 2023 iteration of the video browser showdown. These focus primarily on text-based retrieval schemes and corresponding user-feedback mechanisms.

Scientific datasets often have hierarchical structure: for example, in surveys, individual participants (samples) might be grouped at a higher level (units) such as their geographical region. In these settings, the interest is often in exploring the structure on the unit level rather than on the sample level. Units can be compared based on the distance between their means, however this ignores the within-unit distribution of samples. Here we develop an approach for exploratory analysis of hierarchical datasets using the Wasserstein distance metric that takes into account the shapes of within-unit distributions. We use t-SNE to construct 2D embeddings of the units, based on the matrix of pairwise Wasserstein distances between them. The distance matrix can be efficiently computed by approximating each unit with a Gaussian distribution, but we also provide a scalable method to compute exact Wasserstein distances. We use synthetic data to demonstrate the effectiveness of our Wasserstein t-SNE, and apply it to data from the 2017 German parliamentary election, considering polling stations as samples and voting districts as units. The resulting embedding uncovers meaningful structure in the data.

The number of collisions in the upcoming runs of the Large Hadron Collider at CERN will increase significantly. The increasing amount of data and a higher granularity of the newly developed calorimetric detectors pose a substantial data volume and complexity challenge to the current particle shower reconstruction algorithms. This thesis aims to explore the feasibility of machine-learned models scalable to large data volumes for improving the reconstruction quality of calorimetric particle showers via calorimetric clustering. The goal of calorimetric clustering is to recognize and reconnect fragmented energetic components of particle showers described by three-dimensional spatial structures called tracksters. We show that machine-learned models are viable methods for calorimetric clustering and provide a significant reconstruction performance benefit over classical clustering approaches. Furthermore, we investigate the feasibility of node classification and link prediction problem formulations for training graph neural networks. Experimentally, we show that graph-based models provide a better reconstruction performance, more compact data representation, and better scalability on the tested datasets than feed-forward neural networks.

This thesis conducts an explorative data analysis of People also asked (PAA) questions and answers on Google. The study uses web scraping techniques to collect PAA data for various search terms in the domain of complementary and alternative medicine (CAM). By performing an algorithmic audit, we show that inside the US, neither the location nor the search history influences the set of questions and answers a user is presented by Google for a given search term. The collection of PAA data for an array of real-world search terms in the domain of CAM reveals that many of the answers provided by Google are not from independently fact-checked sources, but instead biased websites such as retail businesses or special interest advocacies. Our results further suggest that the question and answer pairs in PAA might lead to confirmation bias.

With the recent emergence of Vision Language models in the cross-domains of Computer Vision and Natural Language Processing, novel capabilities are being presented to a wide variety of tasks in different domains. Tasks such as Visual Question Answering and Visual Question generation are increasingly being studied in both the general domain and medical domain. However, such Vision Language tasks are still in the early adoption phases in the medical domain. Thus, recent studies are starting to focus more on the Visual Question Answering and Visual Question Generation tasks in the radiology domain, mainly due to the potential benefits for the radiology domain while utilizing the capabilities of Vision Language models. The main focus of this thesis is the Visual Question Generation task in the radiology domain, which we aim to explore how it can be implemented and what multimodal considerations are required. We investigate the differences and capabilities of model architectures by first implementing a baseline model with a CNN-RNN architecture and then to our knowledge the first Transformer-based model architecture focused on the VQG task in radiology. Lastly, we also contribute to future work involved in this domain by providing comprehensive reasoning of model architectures with respect to the textual and visual data modalities and their implications on performance. We show that Visual Question Generation of Radiology images is a complex task with many factors influencing the performance of the model, ranging from the quality and size of the dataset to model architecture decisions.

This master thesis explores domain adaptation methods for pre-trained Large Language Models (LLMs) to map natural language mentions from a text genre onto a target domain ontology based on cosine similarity in a semantic vector space. For the thesis, the input mentions are skill requirement mentions extracted from Swiss job ad postings written in German or English, and the target domain onto which these terms have to be mapped is the European Skills, Competences, Qualifications and Occupations (ESCO) ontology. The objective of this task is to track changes in the labor market and help recruiters fill positions based on skill requirements fulfilled by candidates. The thesis explores three methods: Masked Language Modelling, Multiple Negative Ranking Loss, and binary classification method for further pre-training in order to adapt LLMs to a target domain ontology. Experiments were conducted on 15 model variants using different input data and starting models. Two gold standard datasets, one consisting of randomly selected skill requirement mentions, and the other specifically crafted from challenging cases, were used for evaluating model performance. The evaluations were created by annotating the top suggestions made by our model variants. Mean Average Precision (MAP) scores were computed based on human annotations of the suggestions, made by each model variant for each term in the gold standard datasets. MAP is used as an evaluation metric since more than one mapping might be correct or acceptable, and a good ranking of the appropriate ontology concepts can be measured via this metric. The MNR models with the hard negative sampling strategy, wherein the negative samples are taken with lexical and semantic similarities to the anchor term, and domain adaptation on both the job-ads data and the ESCO ontology data were found to be the best-performing model variants for both the English and German languages. The thesis concludes that domain adaptation on both the input texts and the target domain is beneficial for mapping mentions from the input genre onto the target domain. It also suggests that using a hard negative sampling method for creating the MNR data is beneficial compared to a random negative sampling method.

Code review is a process in which developers assess code changes submitted by their peers. Despite its numerous benefits, code review is a time-consuming and costly endeavor for both the reviewers and the code author. Reviewers are tasked with meticulously scrutinizing the author’s code and offering natural language comments to identify functional or non-functional issues. Meanwhile, the author must comprehend the review feedback and revise the submitted changes accordingly, a task referred to as ‘Code Revision After Review’ (CRA). Existing research has explored methods to automate the CRA task, by pre-training large language models (LLMs), such as CodeBERT and CodeT5 on source code data and fine-tuning them to generate revised code. Although these models utilize distinct pre-training strategies, the impact of these strategies on the CRA task has yet to be investigated. In this paper, we present an empirical study aimed at investigating the effects and efficacy of various pre-training strategies on the CRA task. In this context, we also introduce and evaluate CodeRef—a novel ensemble of pre-training strategies that substantially surpasses baseline performance, achieving at least four times greater likelihood of producing perfectly revised code. Our findings underscore the significance of pre-training in achieving optimal performance and offer insights into various pre-training strategies that may be applicable to other code refinement tasks.

Ethereum in its current version reaches a maximum transaction throughput of 15 transactions a second and thus suffers from a scalability problem. The Polygon proof-of-stake blockchain presents itself as an already active solution to this problem. Previous research focuses on the fairness in other proof-of-stake blockchains and the scalability issue in general. Our contribution is to provide a careful investigation of incentives and decentralisation in Polygon PoS. To this end, we analyse the scalability potential by having a look at transactions, usage and distribution of rewards to participants in the network. Our results indicate that Polygon PoS, as a cheap solution, can enhance the transaction throughput. Furthermore, the blockchain has a fairly good user adoption paired with climate-friendliness. However, in order to be the ultimate scaling solution, there is the need to double-down on incentives and increase performance by a lot. It can also be shown that certain participants get disproportionately more rewards than others, as seen by applying measures like the Gini and Nakamoto index to the data. Centralisation seems to be a problem throughout the network. In other words, we find that Polygon PoS at the current stage is lacking incentives and decentralisation and only early adopters of the blockchain can profit from it.

Vision Transformers (ViTs) are powerful deep learning models and have recently made impressive strides in the computer vision field. However, vision transformers are not data efficient, and their high computational cost, quadratic in the number of tokens, currently limits their adoption in power- and computation-constrained applications. To improve the data and inference efficiency of ViTs, we explore two different paths. First, we notice that the tokens in ViTs do not take any inductive bias. We extract more fine-grained tokens (dubbed subtokens) from each token by expanding its channel dimension to spatial dimensions, and introduce convolutions or shifting on the subtokens to insert intra-token spatial priors. The subtoken convolution improves the classification accuracy for ViTs training from scratch by 2.21% on small datasets (Cifar100) and 1.14% on larger datasets (ImageNet-1K), and also shows faster convergence speed. Secondly, recent studies have shown that not all tokens are helpful for the final task, and ViTs can be made more efficient by pruning redundant tokens. However, active research is mostly focusing on high-level tasks like image classification. To extend the token pruning methods to more complex downstream tasks, we revisit the designs of token pruning and find three key components that lead to better performance: (1) the token selection should not be based on the class token, (2) a dynamic pruning rate is better than a static pruning rate, (3) preserving the feature map of all tokens is better than dropping tokens for all later layers. To this end, we propose SViT, a simple yet effective dynamic token selection scheme that selects and processes highly informative tokens while preserving a structured feature map, thus maintaining compatibility with downstream tasks. On the image classification task (ImageNet-1K), we improve the throughput of DeiT-S by 49% with only 0.4% accuracy drop. On object detection and instance segmentation tasks(COCO), we improve the inference speed by 32.5% with -0.3 box AP and no drop in mask AP.

Knowledge workers generally have only a limited amount of personal resources, including energy, attention and physical capacity, to achieve tasks throughout their day. When these resources are depleted, the person can feel stressed and emotionally exhausted. Knowing how and being able to recharge personal resources is, therefore, essential. As knowledge workers often spend a large part of their day at work, it can be helpful to use time spent at work to establish tiny positive habits, which help to recharge personal resources. Several studies demonstrated that regular breaks can significantly reduce stress and physical discomfort. However, while many studies focus on identifying opportune moments to suggest breaks, they rarely consider the activities that knowledge workers pursue during the break. By the definition of resource depletion, each activity can recharge or deplete resources, depending on personal preferences, making break activities crucial for achieving beneficial breaks. This thesis explores how the Break Scheduler approach may increase users' awareness on their personal resources and break habits and how it supports them in identifying beneficial break activities to improve their personal resources. This approach focuses on self-experimentation to improve awareness through self-reporting and nudging. Additionally, a rule-based system suggests a break schedule which is personalized by the user and will be adjusted by the Break Scheduler over the use period based on the user's self-reports. The investigation included 13 participants who used the software over one to two weeks. A total of 154 breaks were reported, as well as 143 daily reports. Each participant also answered a pre- and post-intervention questionnaire, giving valuable insights into their demographics, previous break habits, and their experience with the Break Scheduler. Overall, the findings suggest that self-reporting and nudging, such as scheduling the breaks in advance and notifications, can improve the awareness of the participant's personal resources and break habits. Additionally, the personalisation aspect of the Break Scheduler is crucial to help users to identify break activities that were successfully supporting them to recharge their personal resources. The results of this thesis offer insights into the potential of the Break Scheduler approach in supporting knowledge workers to increase their awareness on their personal resources and break habits by self-reporting and nudging and in helping them find beneficial activities to improve their personal resources.

Various studies have assessed the GHG footprint of the ICT sector (ICT end-user devices, data centers, telecommunication networks) and the potential of ICT use cases (e.g. smart homes, ride sharing) to avoid GHG emissions in other sectors (e.g buildings, transport). We systematically compare relevant studies from the last ten years and discuss the robustness of results in view of the methods used. The results show that the ICT sector causes between 1.5% and 4% of global GHG emissions, a major share of which is due to the production of ICT end-user devices. Estimating GHG impacts of device production is the main source of uncertainty. Results of studies on ICT's GHG abatement potential are less robust, in particular due to uncertainty with regard to use case impacts in a real-life setting, types and sizes of economy-wide rebound effects. Thus the existing studies do not provide a reliable basis for estimating the actually realized GHG abatements. To improve the assessment results and provide a more reliable basis for deriving GHG reduction measures future research should empirically investigate which solution design and accompanying policies are suitable to exploit GHG reduction potentials in real-life. Results of these studies would also increase the robustness of assessments of GHG abatement potentials.

AI has the potential to revolutionize industries and improve daily life through the development of advanced machine learning (ML) and deep learning (DL) models. These models, such as chatbots and language models, use algorithms or artificial neural networks to recognize patterns and make decisions. ML involves training algorithms on large datasets to identify patterns and make decisions, while DL uses artificial neural networks composed of interconnected nodes called artificial neurons to process and transmit information. Neural networks can learn and make decisions by adjusting the connections between neurons based on input data. There are two types of ML and DL: unsupervised and supervised. Unsupervised learning involves using algorithms or neural networks to learn from data without labeled outcomes, while supervised learning involves training algorithms or neural networks on labeled data to make predictions or decisions. As AI becomes more advanced and widespread, it is important to have confidence in the decisions and actions of these systems. Trusted AI refers to the reliability and ethical behavior of AI systems. It is crucial to have a framework for evaluating the trustworthiness of different AI models to ensure their safe and responsible deployment. A taxonomy of pillars and metrics can be used to quantify the trustworthiness of AI models, allowing for a structured and comprehensive evaluation of their strengths and limitations. The following bachelor thesis aims to survey existing platforms, define requirements and develop a web app that allows the computation of the trustscore, pillarscores, metricscores of supervised and unsupervised and DL platform is extended to allow for user management, and the return of the trustworthiness levels via API endpoints.

This work aims to investigate the feasibility of exploiting reinforcement learning (RL) to improve the impact of ransomware on a target device while evading dynamic detection methods such as behavioral fingerprinting-based anomaly detection (AD). Given the constantly growing number of connected resource-constrained devices, such as Internet of Things (IoT) devices, and the significant rise in ransomware attacks over the past years, the importance of investigating ransomware attacks and corresponding defense approaches is evident. So far, most related research has been confined to exploring unethical artificial intelligence (AI) systems instead of analyzing the possibilities of using AI for launching optimized malware attacks. This work covers the mentioned limitations and introduces Ransomware Optimized with AI for Resource-constrained devices (ROAR), an RL framework to hide ransomware from dynamic detection mechanisms and optimize its impact on the target device. ROAR has been deployed in a real-world IoT crowdsensing scenario, including a Raspberry Pi 4 as a spectrum sensor. The Raspberry Pi was infected with ROAR, and behavioral data were collected from the target device to facilitate environment simulation. The results obtained by executing prototypes of the RL agent have been aggregated, and the corresponding plots are discussed and compared. These findings suggest that no relation exists between individual actions within an episode and that discounting future rewards does not improve performance in this particular RL problem. Overall, this work demonstrates the feasibility of optimizing ransomware attacks with RL and the effectiveness of the resulting evasion capabilities. The findings derived from the collected results hold in a simulated environment and when the agent is deployed in a real scenario. To our knowledge, this work is the first to investigate the possibilities of supporting malware attacks with RL during the attack phase. Further studies are needed to investigate additional optimizations of the RL model, efficiency improvements to the underlying ransomware implementation, and the feasibility of attacking more powerful devices.