Contributions published at Dynamic and Distributed Information Systems (Abraham Bernstein)

This thesis describes SSE - a system for automatically retrieving the sample size from an empirical study. SSE employs a three-stage pipelined architecture. The first stage utilizes Pattern Matching in order to extract potentially relevant sentence fragements from a document. The second stage is responsible for rule-based filtering of the matches returned in the first level. The last and most important stage is accountable for the application of case-specific heuristics in order to return the correct sample size for the document. The strengths of SSE can be seen in the fact that it is applicable to a variety of research publications.

As the Semantic Web receives increasing attention, there is a challenge in managing large RDF datasets efficiently. In this thesis, we address the problem of predicting SPARQL query performance using machine learning. We build a feature vector describing the query structurally and train different machine learning models with it. We explore ways to optimize our model's performance and analyze TensorFlow deployed on the IFI cluster. While we adopt known feature modeling, we can reduce the vector size and save computation time. Our approach can significantly outperform existing approaches in a more efficient way.

Appropriate task routing and assignment is an important, but often overlooked, element in crowdsourcing research and practice. In this paper, we explore and evaluate a mechanism that can enable matching crowdsourcing tasks to suitable crowd-workers based on their cognitive abilities. We measure participants’ visual and fluency cognitive abilities with the well-established Kit of Factor- Referenced Cognitive Test, and measure crowdsourcing performance with our own set of developed tasks. Our results indicate that participants’ cognitive abilities correlate well with their crowdsourcing performance. We also built two predictive models (beta and linear regression) for crowdsourcing task performance based on the performance on cognitive tests as explanatory variables. The model results suggest that it is feasible to predict crowdsourcing performance based on cognitive abilities. Finally, we discuss the benefits and challenges of leveraging workers’ cognitive abilities to improve task routing and assignment in crowdsourcing environments.

Crowdsourcing has raised interest in both the scientific and industrial community as an online distributed problem-solving model. TopCoder is one of the biggest crowdsourcing platform, regularly hosting different types of competition. This paper analyzes ten different data science competitions on TopCoder. Following the grounded theory method, we identified key factors leading to different results. We found low diversity of high quality results and try to find the reason for it. We further discuss the influence of the competition structure on the results and suggest a less limiting format to improve the quality of results.

With the increasing popularity of crowdsourcing and crowd computing, the question of how to select a well-performing crowd process for a problem at hand is growing ever more important. Prior work casted crowd process selection to an optimization problem, whose solution is the crowd process performing best for a user’s problem. However, existing approaches require users to probabilistically model aspects of the problem, which may entail a substantial investment of time and may be error-prone. We propose to use black- box optimization instead, a family of techniques that do not require probabilistic modelling by the end user. Specifically, we adopt Bayesian Optimization to approximate the maximum of a utility function quantifying the user’s (business-) objectives while minimizing search cost. Our approach is validated in a simulation and three real-world experiments. The black-box nature of our approach may enable us to reduce the entry barrier for efficiently building crowdsourcing solutions.

A key issue, whenever people work together to solve a complex problem, is how to divide the problem into parts done by different people and combine the parts into a solution for the whole problem. This paper presents a novel way of doing this with groups of contests called contest webs. Based on the analogy of supply chains for physical products, the method provides incentives for people to (a) reuse work done by themselves and others, (b) simultaneously explore multiple ways of combining interchangeable parts, and (c) work on parts of the problem where they can contribute the most. The paper also describes a field test of this method in an online community of over 50,000 people who are developing proposals for what to do about global climate change. The early results suggest that the method can, indeed, work at scale as intended.

Before building a classifier to make predictions about a target variable, one must decide what input data to use. Most scientific publications about feature selection deal with methods that can be used once training data has been collected. Yet, in the real world, one has to collect, clean and transform data before it can be used to create predictive models. Collecting data is a very expensive and time consuming process. Going through this process for data not relevant to the target variable is very inefficient. A common approach to minimize the effort for feature selection is asking domain experts for their opinion. However, experts have been shown to perform worse at this task than one might expect. In this paper, I present a tool, KrowDD, that is able to identify relevant features among a number of feature ideas before obtaining data. An evaluation using three datasets shows that KrowDD performs significantly better than human experts. KrowDD is the first step on the way to more efficient feature selection: feature selection before obtaining training data.

As the importance of data from the social web increases, research and attempted analysis of such data, gain more and more relevance in science. In a time where the digital and the real world are connected like never in history, such data provide important insight on social developments on a local and a global scale. The ability to handle this kind of data is hindered not only by humongous size and complexity, but also by the researchers’ ability to handle it in a programmatically way. This despite the fact, that the most needed operations include very basic and generically reusable scripts. The following work provides a package, that includes mechanisms to process, transform, enrich and visualize data gathered from the Twitter-API. It can be useful to data scientists, social scientists or journalists. We present a description of the package and its abilities through some examples.

null

null

null

Background: Value sensitivity – the ability to recognize value-related issues when they arise in practice – is an indispensable competence for medical practitioners to enter decision-making processes related to ethical questions. However, the psychological competence of value sensitivity is seldom an explicit subject in the training of medical professionals. In this contribution, we outline the traditional concept of moral sensitivity in medicine and its revised form conceptualized as value sensitivity and we propose an instrument that measures value sensitivity. Methods: We developed an instrument for assessing the sensitivity for three value groups (moral-related values, values related to the principles of biomedical ethics, strategy-related values) in a four step procedure: 1) value identification (n = 317); 2) value representation (n = 317); 3) vignette construction and quality evaluation (n = 37); and 4) instrument validation by comparing nursing professionals with hospital managers (n = 48). Results: We find that nursing professionals recognize and ascribe importance to principle-related issues more than professionals from hospital management. The latter are more likely to recognize and ascribe importance to strategy-related issues. Conclusions: These hypothesis-driven results demonstrate the discriminatory power of our newly developed instrument, which makes it useful not only for health care professionals in practice but for students and people working in the clinical context as well.

In the last two decades, we have experienced a tremendous growth of the digital infrastructure, leading to an emerging web ecosystem that involves a variety of new types of services. A characteristic element of this web ecosystem is the massive increase of the amount, availability and interpretability of digitalized information—a development for which the buzzword “big data” has been coined. For research, this offers opportunities that just 20 years ago were believed to be impossible. Researchers now can access large participant pools directly using services like Amazon Mechanical Turk, they can collaborate with companies like Facebook to analyze their massive data sets, they can create own research infrastructures by, e.g., providing data-collecting Apps for smartphones, or they can enter new types of collaborations with citizens that donate personal data. Traditional research ethics with its focus of informed consent is challenged by such developments: How can informed consent be given when big data research seeks for unknown patterns? How can people control their data? How can unintended effects (e.g., discrimination) be prevented when a person donates personal data? In this contribution, we will discuss the ethical justification for big data research and we will argue that a focus on informed consent is insufficient for providing its moral basis. We propose that the ethical issues cluster along three core values—autonomy, fairness and responsibility—that need to be addressed. Finally, we outline how a possible research infrastructure could look like that would allow for ethical big data research.

Researchers, governments, and funding agencies are calling on research disciplines to embrace open data - data that is publicly accessible and usable beyond the original authors. The premise is that research efforts can draw and generate several benefits from open data, as such data might provide further insight, enabling the replication and extension of current knowledge in different contexts. These potential benefits, coupled with a global push towards open data policies, brings open data into the agenda of research disciplines – including Information Systems (IS). This paper responds to these developments as follows. We outline themes in the ongoing discussion around open data in the IS discipline. The themes fall into two clusters: (1) The motivation for open data includes themes of mandated sharing, benefits to the research process, extending the life of research data, and career impact; (2) The implementation of open data includes themes of governance, socio-technical system, standards, data quality, and ethical considerations. In this paper, we outline the findings from a pre-ICIS 2016 workshop on the topic of open data. The workshop discussion confirmed themes and identified issues that require attention in terms of the approaches that are currently utilized by IS researchers. The IS discipline offers a unique knowledge base, tools, and methods that can advance open data across disciplines. Based on our findings, we provide suggestions on how IS researchers can drive the open data conversation. Further, we provide advice for the adoption and establishment of procedures and guidelines for the archival, evaluation, and use of open data.