Contributions published at Artificial Intelligence and Machine Learning (Manuel Günther)

In recent years, image classification with neural networks have shown promising results and hence garnered interest of researchers. These networks have been able to perform classification with near human efficiency. However it was later discovered that it is easy to fool neural networks and they are not robust to small non random perturbations. Addition of small non random pertubations which is imperceptible to human eye can cause neural network to misclassify an image which was previously correctly classified. Such a perturbed image is called adversarial image and an approach to generate such image is called adversarial attack. Morever these perturbations are portable to other neural network architecture which means an adversarial image that can fool one network can also be used to fool other networks. This poses a huge security risk for system that rely on image classification such medical analysis and autonomous vehicle. An attacker can modify the input image which could lead to failure of the system. This suggests that there is a need to build systems which are robust against such small perturbations. Portability is another import aspect, since if an adversarial example is only portable to few of the neural networks, an ensemble of network can be used to prevent against such attacks. In this work we propose experiments to evaluate portability of adversarial images across popular neural networks. We perform an in depth portability study by looking into how far the predictions are from true class. Since there has been very few research for targeted attack and it’s portability we include them in our study. We generate adversarial images on 13 neural network model with 3 adversarial attack and different value of perturbation constant epsilon on the subset of ImageNet dataset (ILSVRC 2012). A distance metric was proposed to calculate the distance between any two classes of the dataset. The experiments shows that with untargeted attack and a larger epsilon the portability is high between networks of similar architecture but it drops significantly as we reduce the epsilon. With targeted attack, 3 different approach to choosing target was proposed but we observe low portability with all the targeted attacks and different values of target and perturbation constant.

The aim of this study is to devise deep learning (DL) approaches trained on paired and unpaired data that are able to generate realistic synthetic computed tomography (CT) images for magnetic resonance-guided radiotherapy (MRgRT) in the area of the abdomen and to assess its clinical applicability. The imaging data of 76 patients with a tumour in the abdomen who were treated with MRgRT at USZ was collected retrospectively and divided in training (59) and test sets (17). To improve the current state-of-the-art of DL technologies by studying different architectures and ensembles of configurations, the following experiments were conducted: (a) evaluating the influence of the different GAN architectures trained on paired (Pix2pix) and unpaired data (CycleGAN and CUT, which firstly applied for the purpose of sCT generation); (b) investigating the footprint of different preprocessing methods (Nyul, novel N-peaks); (c) improving spatial consistency of results by adjusting the network configuration (2D, pseudo3D); (d) testing different GAN training objectives (LSGAN, WGAN-GP); (e) estimating the influence of the loss function on the generated results (per-pixel L1 loss, VGG19 perceptual loss). The quality of sCT generation was assessed using both, image similarity and dosimetric accuracy metrics. The dosimetric accuracy of the best performing models was estimated by comparing the dose distribution of MRgRT treatment plans calculated from synthetic CT and original CT images using dose-volume histogram (DVH) parameters to allow assessment of the clinical applicability of the DL methods. Our results suggest that DL models trained with unpaired data achieve similar performance as models requiring perfectly aligned image pairs, and even perform better in the bone and air pocket areas. The mean absolute errors (mean ± SD) calculated within the body contour are 71.0±20, 73.4±21 and 84.5 ±19 HU when using the best performing configuration of pix2pix, CycleGAN and CUT, respectively. The proposed DL-based synthetic CT generation methods may be considered clinically applicable for treatment planning in the abdominal area with the mean DVH indicator discrepancies with the original plan of less than 1% for all models, and less than 0.5% for all tumour DVH indicators for the best performing model, CycleGAN. The study confirmed that generation of synthetic CT using a DL approach from low field magnetic resonance images in the abdomen is feasible and allows a reliable calculation of irradiation plans in MRgRT.

Tables are one of the most convenient ways to present complex, correlated, and structured information. While rule and heuristic based approaches have long dominated the table detection and structure recognition, their usefulness has been confined to a subset of tables that follow these rules. Significant research has been conducted to localize table structure, the majority of which focuses on using heuristics and rules with the assistance of optical character recognition (OCR) to manually select layout characteristics of the tables. With the rise of Deep Learning, new models have shown to be applicable across multiple unseen domains by incorporating transfer learning. This thesis presents an end-to-end object detection approach to detect tables and recognize their structures in a document and thus, help in table data extraction with the use of a deep learning model namely, Faster R-CNN. This work will also introduce a new metric based on Intersection over Union (IoU) for the task of table detection which does not penalize large bounding box predictions up to a defined extent and reduces the dependency of the F1 score on the chosen IoU threshold. A significant amount of experiments will be discussed on many popular publicly available datasets like ICDAR 2013, ICDAR 2019, ISRI-OCR, Marmot, TableBank, and PubTables-1M to carefully adapt and design the parameters of the Faster R-CNN model and demonstrate the robustness of the model across unseen datasets. The model present in this thesis outperforms other models including a transformer-based model to establish the state-of-the-art results on these datasets proving once again the superiority of the Faster R-CNN architecture.

Open Set Recognition (OSR) is a promising field that aims to adapt deep neural models to a real-world scenario, i.e., the presence of unknown data distributions at inference time. Recent approaches address OSR by including into the training set negative classes that a classifier learns to reject, expecting that these data increase the robustness of the classifier on unknown classes. Furthermore, a critical question on OSR is how to select representative negative classes. In this work, we study the performance an OSR approach, the Entropic open-set and Objectosphere, two recently proposed loss functions to deal with unknowns. We show that these losses can be used in complex custom scenarios that use natural images and several levels of similarity between the negative classes. Additionally, we investigate a method to adapt adversarial training into OSR scenarios. Our experimental results show that it is possible to use synthetic adversarial examples to increase the robustness of a classifier in OSR.

Face recognition has been popular in the video recently. As the development of deep learning, various CNNs models are implemented into face recognition such as ResNet, MobileNet, Mo- bileFaceNet. During this experiment, we verified that the light CNN model – Stacked2D, and 3D MobileFaceNet can extract features from several frames at the same time on the video dataset (YoutubeFaces). First, the baseline model – the original 2D MobileFaceNet combined ArcFace loss function model is trained from the face recognition task. Then, this model is implemented as the feature extractor in bob framework, which can construct a face recognition pipeline easily. Using the same process, the Stacked2D and 3D MobileFaceNet models with Arcface are trained using YTF dataset. In the end, we run the video recognition pipeline in bob framework and com- pare the results using different models. In this experiment, we verify that it is feasible to use 2D, Stacked2D, and 3D MobileFaceNet models in video face recognition, and the model with larger frames input can perform better because it can capture more spatial and temporal information from video data.

The goal for the open-set face recognition is to identify the unseen subjects and do not assign them to any known subject with high confidence. There are two types of subjects involved in the task: the ones that we are interested in and have labels, i.e. known subjects; the ones that we do not care about and have no labels (we use −1 in the experiment instead), i.e. unknown subjects. We build a complete face recognition pipeline through Bob. ArcFace R100 network, as a feature extractor, has a good performance on the IJB-C dataset. Our goal is to add an extra network after ArcFace to enhance its power on open-set face recognition tasks. We attempt three cases: first, the unknown subjects have never appeared in the training; second, the unknown subjects appear in both training and testing; third, the unknown subjects only appear in the testing, and they are replaced by the adversarial samples generated from the knowns in the training. The training unknowns have no overlap with the testing unknowns in case three. Plain softmax loss and entropic open-set loss are applied to the first two cases, respectively, and objectosphere loss is used for the second and third cases. We prove that those models create a high True Positive Identification Rate especially when the False Positive Identification Rate is small. Replacing the unknown subjects in case two to the adversarial samples as in case three is successful without performance degradation. One flaw is that the magnitude separation property of the entropic open-set loss and objectosphere loss is not apparent. When working with the adversarial samples, the situation is worse.

Face alignment is considered a crucial process in the pipeline of face recognition. However, to perform face alignment precisely is not an easy job. Günther et al. (2017) proposed AFFACT technique to train a network that is robust to facial misalignment, while the original work is for attribute classification problem. In this thesis, I apply the AFFACT technique to train a target face recognition model on AFFACT-perturbed images, and a baseline model on aligned images. The results of the two models are compared to show how AFFACT technique works for face recognition model. Furthermore, I put the emphasis on studying the image perturbation in the test stage. Single perturbations are experimented and combination of perturbations are searched by different optimization algorithms. To combine different perturbations, majority voting for the final predicted labels and averaging the predictions of probability are often used in different works. Combining the image similarities is also mentioned in previous research. In this thesis, I test the combination of perturbations by averaging the (weighted) features extracted from the perturbed images.

Current work on reinforcement learning algorithms for the reset-free setting focuses heavily on model-based agents because of their strong planning capabilities, especially for previously unseen tasks. Model-based reinforcement learning relies heavily on internal models of the world surrounding the agent, but not much attention is paid to shifting environment dynamics, which is likely to happen in various real-world scenarios. In this work, LiSP (Lifelong Skill Planning) and LSR (Learning Skillful Resets) are evaluated and compared against SAC (Soft Actor-Critic) in situations with different environment behaviour. Additionally, methods of environment shaping and environment dynamism are examined to facilitate the adaptation to new environment dynamics or as an alternative way to slowly introduce complex environment dynamics during training. The results suggest that both LiSP's and LSR's exploration mechanisms fail to explore both environments sufficiently to achieve good performance. SAC needs some help in the form of environment shaping or the help of occasional resets to achieve near-optimal performance in one environment, but fails to learn in the second. The results demonstrate what kinds of environment modifications are most useful to improve performance and which ones are better avoided.

A real-world test situation not only comprises of classes that a supervised model is trained with, but also consists of unknown classes that the model is unaware of. Such a test situation forms part of an open space, and an open-set classifier, trained using open-set algorithms is used to handle it. These algorithms are developed using artificially generated open space, created using available datasets and their classes. Open-set algorithms that are currently in place are either developed using small datasets or, address only those unknown samples that are distinctly different from known classes. In this thesis, I propose three ImageNet based open-set protocols that closely align with the real-world open space and also, address unknown samples that are similar to known classes. I use these protocols to compare the performance of two easy-to-implement open-set algorithms, SoftMax with Garbage Class and Entropic Open-set Loss, that were developed using small datasets, and compare their performance to that of a baseline Traditional SoftMax.

The predictions of object and face detectors suffer from unstable bounding boxes. The main reason for this problem is the post-processing algorithm Non-Maximum Suppression. During the suppression of redundant bounding boxes, only the most confident box does not get removed. This behavior can be observed when inspecting video sequences. Even marginal change in pixel values causes the detected bounding boxes to jitter. In this work, a method is proposed to further inspect this problem by fabricating sequences of augmented static images to simulate moving objects and faces. Combining this approach with an evaluation metric from video detection leverages the quantification of temporal and spatial stability of detected bounding boxes compared to their associated ground truth annotations. Simultaneously, two alternative Non-Maximum Suppression algorithms are proposed to solve the problem of jittering bounding boxes. The algorithms are called Average and Average IoU Non-Maximum Suppression. Both alternatives consider aggregating the overlapping bounding boxes and their detection scores using a weighted average of the individual coordinates and class probabilities. An increase in stability can be evidenced by implementing Average and Average IoU NMS into post-processing the multi-stage detectors Faster R-CNN and MTCNN and comparing it to their default NMS function. By evaluating the object and face detectors on the MS COCO, PASCAL VOC, and WIDER Face datasets, even an improvement in accuracy can be observed.

Computational vision has become more and more relevant in the last few years. Analysis of image and video input is relevant to robotics and self-driving cars to name just a few examples. Because of this relevance, algorithms for stereo matching need to get more precise and faster. This thesis considers the use of Gabor Jets for disparity estimation, evaluates its efficiency, and compares it to other algorithms. For this comparison, three different algorithms have been chosen: Horn Schunk Optical Flow, Lius Optical Flow, and StereoBM. To be able to compare different types of images, a sample from the DrivingStereo as well as the Middlebury dataset has been selected. For the evaluation, the bad matched pixels, mean relative error, and mean absolute error of all four algorithms were calculated. Gabor Jets worked better on images taken in a real-life setting than in a studio setting. Especially in images with a lot of traffic, Gabor Jets work better than the other algorithms. Based on these evaluations, Gabor Jets can keep up with the aforementioned algorithms and even surpass them in certain aspects.

Modern day machine learning models are becoming omnipresent and are required to handle progressively more complex environments in their tasks. In classification problems, an increasingly popular scenario is called Open Set Recognition, which does not require the model to have complete knowledge of the world and during which unknown classes can be submitted to the algorithm while testing. This thesis tackles the challenge to correctly handle and reject these unknown inputs by performing adversarial training on our classification model. Furthermore, we analyze the difference in performance of several state-of-the-art adversarial attacks used in our adversarial training. The experiments show that our approach effectively deals with unknown inputs and delivers very promising results. To our knowledge, there has been no prior work that used adversarial training for Open Set Recognition like in our approach.

In recent years, research has made tremendous progress in the classification of facial attributes through convolutional neural networks (CNNs). Such neural networks specialize in extracting various facial attributes from images. In this thesis, we investigate how these attributes correlate with identity and whether such correlations can be used to improve the classification accuracy of extractions from neural networks. We introduce a method for calculating correction terms based on statistical metrics of each attribute of ran identity. These correction terms are then applied to the extractions of neural networks, resulting in reweighted values which possess a lower average error rate than the original extractions. We offer approaches for both unbalanced and balanced CNNs. For the balanced networks, we adapt our correction terms to the domain distribution in order to consider the imbalance of classes. We show that this approach achieves similar results for manually annotated identity labels and identity labels that are inferred from the clustering of features that are extracted with neural networks. Our results show that it is indeed possible to lower the average attribute classification error rates for both neural networks based on the correlation of these attributes with identity.

Face recognition has become an indispensable part of life in today's world. In addition to unlocking mobile devices, it can also be used in public safety applications. Earlier, face recognition was accomplished using traditional algorithms. Today, face recognition has been dominated by deep learning, and deep convolutional neural networks can achieve impressive results. Unfortunately, these results can rarely be reproduced due to missing experimental details. The goal of this work is to compare state-of-the-art deep neural networks with respect to different aspects of face variations. For this purpose, four open-source networks from ArcFace and one from VGGFace2 were used. Experiments are performed on different databases to evaluate the influence of face variations. The results show that deep learning methods clearly outperform traditional face recognition algorithms, and the training database plays a crucial role in their performance. Most of the networks can handle occlusion and illumination well, but poses and facial expressions may still cause problems. Finally, recognizing faces at longer distances requires further improvement.

Applying data augmentation on test images can improve facial attribute classifications. The Alignment-Free Facial Attribute Classification Technique (AFFAT) shows that performing 162 transformations on test images can improve the prediction of facial attributes on the CelebA dataset. However, this process costs a substantial amount of time, and the effectiveness of each transformation is not considered. This research aims to find the best combination of transformations that will be applied as test-time data augmentation, such that the prediction results of the AFFACT model can be improved. Genetic algorithms are employed for the optimization task. As a result, even though the overall prediction accuracy does not improve, the number of transformations applied as test-time data augmentation has been decreased dramatically.

Deep neural networks are capable of solving various tasks very well and in certain cases even outperform humans. However, they are vulnerable to inputs designed to cause the network to give faulty output, so-called adversarial examples. There does not yet exist a solution for the problem that they pose with respect to the robustness of neural networks. However, there are various techniques to alleviate the vulnerability of neural networks towards adversarial examples. Adversarial training, that is, incorporating adversarial examples into the training data, is amongst the most effective currently existing defense techniques. Layerwise Origin-Target Synthesis (LOTS) is a method to create adversarial examples. This work inspects how well adversarial training increases the robustness of neural networks against adversarial examples created using LOTS.

In this thesis we do face verification, where the main task is to decide if two images belong to the same person or to different people. To do that we use the images from the Labeled Faces in the Wild (LFW) database. The images are taken in an uncontrolled environment and thus have natural variation in pose, lighting and background. We approach face verification through comparing facial attributes of two images from LFW. These facial attributes are extracted from our AFFACT network. To compare the facial attributes, we develop algorithms that perform different distance-based calculations between them. Some of our algorithms train a Support Vector Machine (SVM). After the comparison of two images, a score value is computed which deduces whether they belong to the same person and thus solve the face verification problem.

Image classification models are often trained on additional transformations of the original images to increase the number and the variance of the training set. These transformations can include, for example, horizontal flips, rotations, and color transformations. Most of the recent data augmentation methods focus on improving data augmentation techniques during the training phase. There is very little research on test-time augmentation, which contains data augmentation during the testing phase. Often, only very simple transformations are used during testing, and the predictions are frequently aggregated through simple averaging or majority voting. I propose new aggregation methods to improve the predictive performance of image classification models during the testing phase. I show on four different datasets, namely CIFAR-10, MNIST, ImageNet, and a dataset containing images for melanoma classification, that other aggregation methods, besides the commonly applied averaging and majority voting, obtain better results.

Accuracy in face recognition, especially with machine learning and artificial neural networks, has reached new records in recent years. However, the direct verification of two face images with a large difference in pose is still a big challenge even for these methods. An earlier developed method avoids the direct comparison of two images by ranking similarities to reference images and then determining the similarity of two faces by the similarity of their rank lists. This method is combined in this thesis with state-of-the-art neural networks to evaluate the value of rank lists and reference images, especially in the recent context. The results are demonstrated on the two datasets "Celebrities in Frontal-Profile" (CFP) and "Cross-Pose Labeled Faces in the Wild" (CPLFW) and with four different convolutional neural networks. In this bachelor thesis it could be shown that this combination, applied on CPLFW, is not significantly different than the most accurate neural network by its own. Additionally, two new methods were developed that can calculate the similarity of two rank lists in this context more accurately than all known methods from the literature. Furthermore, it was shown that in addition to the low ranks, i.e. similar reference persons, the high ranks, i.e. dissimilar reference persons, are also extremely informative about a person's identity.

Recognition of colour is an essential part of our visual system and impressive neural resources are devoted to colour vision. With the emergence of Convolutional Neural Networks (CNNs) from Artificial Intelligence (AI) and AI’s goal to simulate vision, the influence of colour on artificial image classification and face recognition gained further attraction in recent years and remains subject to much debate until today. In this thesis, we compare the performance of CNNs using four datasets, six network architectures, and two image chromaticities to investigate on the influence of colour on image classification and face recognition. To extend our analysis beyond the application of pre-existing network architectures, we implement a novel CNN architecture, R-G-B, that learns convolutional filters on the colour channels of an RGB image separately and fuses those layers later in the network architecture, allowing us to investigate on the influence of processing colour within CNNs. Our results suggest that colour does improve performance on both, image classification and face recognition at the expense of computational costs. Furthermore, R-G-B networks are applicable but do not improve performance and are outperformed by regular networks trained on either RGB or greyscale images.