IPAB Workshop-05/03/2020

Boyan Gao

Topic: loss function learning

Loss function plays an essential role in machine learning, most of the learning problems can be phrased as an optimization problem where the loss function works as an objective function. Unfortunately, to make a learning model achieve high performance, usually, every specific machine learning task involves some handcraft of loss function, such as adding penalty terms, which not only requires some domain knowledge and increases the difficulties of hyper-parameter tuning. Inspired by the learning-to-learn concept in the recent re-activate research field, meta-learning, in our research, we explore whether a learning system is able to learn a loss function parameterized by a neural network, meta-network. Like most gradient-based meta-learning methods, we study a meta-learner under a bilevel optimization setting which can learn a loss function to supervise the training of another task-specific learning model, task-specific-net, in the inner loop. In our work, we design the meta-networks from both feature and label perspectives and learn them from scratch jointly with task-specific-net. We also consider a task that makes meta-network generate hyperparameters, namely the label smoothing parameter for classification. 

Alexandros Keros

Title: Jittering samples using a kd-tree stratification

Abstract:

Monte Carlo sampling techniques are used to estimate high-dimensional integrals that model the physics of light transport in virtual scenes for computer graphics applications. Appropriate sample selection guaranteeing uniformity of samples on the integration domain is shown to improve error convergence speed and alleviate undesirable visual artifacts, such as grain and repeating patterns. We propose an efficient and easily parallelizable domain stratification and sampling scheme for d-dimensional hypercubes using the kd-tree data structure, for which equidistribution bounds are derived. The method exhibits performance comparable to state of the art Monte Carlo and Quasi Monte Carlo methods for image synthesis.

Taha Kocyigit

Title: Alternating Batch Normalization

Abstract:

Batch Normalization is a widely used tool in neural networks to improve the generalization and convergence of training. However, on small datasets due to the difficulty in obtaining unbiased batch statistics it cannot be applied effectively. We propose Alternating Batch Normalization (ABN) a generalization of BN which disentangles the batch statistics calculation from the model updates and enables the use of unlabeled examples to calculate batch normalization statistics. We show that when unlabeled examples are used for batch statistic calculations the bias of batch statistics is reduced and a regularization that utilizes the data manifold is applied. ABN is easy to implement and computationally inexpensive semi supervised learning method that can be applied to a variety problems without any change to the algorithm.  We report results on various vision problems where obtaining dense labeled examples is difficult like, depth estimation, optical flow estimation, semantic segmentation.