Turing AI Postdocs Workshop #2
A 7 December 2022 workshop for ELIAI postdoctoral researchers to present and discuss their research
The second Turing AI Postdocs Workshop, held on 7 December 2022, was hosted by ELIAI and organized by Director Mirella Lapata. The workshop provided an opportunity for the ELIAI postdoctoral researchers Xue Li (project title "Causal Knowledge Graphs for Counterfactual Claims Reasoning"), Davide Moltisanti (project title "Grounding Actions and Action Modifiers in Instructional Videos"), Victor Prokhorov (project title "Multimodal Interpretability from Partial Sight"), and Cheng Wang (project title "Asking your Self-Driving Car to Explain its Decisions") to present and discuss their most recent work on their respective projects. These workshops will be held periodically as the researchers progress on their project objectives. Three newly funded ELIAI projects were introduced as follows: Antonio Vergari introduced "Gradient-based Learning of Complex Latent Structures"; Jacques Fleuriot and Ram Ramamoorthy introduced "Constrained Neural Network Training via Theorem Proving, with Applications to Safe Human-Robot Interaction"; and Amos Storkey and Trevor McInroe introduced "Multimodal Integration for Sample-Efficient Deep Reinforcement Learning."
Project: Causal Knowledge Graphs for Counterfactual Claims Reasoning, Led by Björn Ross and Vaishak Belle
Talk Title: Knowledge Graphs based Misinformation Detection on Counterfactual Claims about Climate Change
Abstract: Climate change is a crisis that requires global action. As social media significantly impacts people's opinions, it is essential to reduce the harmful misinformation about climate change spreading there. However, a single AI approach is not enough for misinformation detection (MD), especially for counterfactual claims. In this project, we aim to develop a system that will apply NLP techniques to parse a given counterfactual claim and then query knowledge graphs (KGs) to get evidence for the MD in the climate change domain. In addition, probability will be computed to represent how much the conclusion is trusted. In this talk, we will present the logic for determining the truth value of a given counterfactual claim, the current pipeline, and the prepared data for annotation and then summarize future work.
Project: Grounding Actions and Action Modifiers in Instructional Videos, Led by Hakan Bilen, Frank Keller, and Laura Sevilla
Talk Title: Learning Action Changes by Measuring Verb-Adverb Textual Relationships
Abstract: The goal of this work is to understand the way actions are performed in videos. That is, given a video, we aim to predict an adverb indicating a modification applied to the action (e.g. cut “finely”). We cast this problem as a regression task. We measure textual relationships between verbs and adverbs to generate a regression target representing the action change we aim to learn. We test our approach on a range of datasets and achieve state-of-the-art results on both adverb prediction and antonym classification. Furthermore, we outperform previous work when we lift two commonly assumed conditions: the availability of action labels during testing and the pairing of adverbs as antonyms. Existing datasets for adverb recognition are either noisy, which makes learning difficult, or contain actions whose appearance is not influenced by adverbs, which makes evaluation less reliable. To address this, we collect a new high quality dataset: Adverbs in Recipes (AIR). We focus on instructional recipes videos, curating a set of actions that exhibit meaningful visual changes when performed differently. Videos in AIR are more tightly trimmed and were manually reviewed by multiple annotators to ensure high labeling quality. Results show that models learn better from AIR given its cleaner videos. At the same time, adverb prediction on AIR is challenging for models, demonstrating that there is considerable room for improvement.
Project: Multimodal Interpretability from Partial Sight, Led by Siddharth Narayanaswamy and Ivan Titov
Talk Title: Multimodal Interpretability from Partial Sight
Abstract: We seek to build DGMs that capture the joint distribution over co-observed visual and language data (e.g. abstract scenes, COCO, VQA), while faithfully capturing the conceptual mapping between the observations in an interpretable manner. This relies on two key observations: (a) perceptual domains (e.g. images) are inherently interpretable, and (b) a key characteristic of useful abstractions are that they are low(er) dimensional (than the data) and correspond to some conceptually meaningful component of the observation. We will seek to leverage recent work on conditional neural processes (Garnelo et al, 2018) to develop partial-image representations to mediate effectively, and in an interpretable manner, between vision and language data. Evaluation of this framework will involve both the ability to generate multimodal data against state-of-the-art approaches, as well as on human-measured interpretability of the learnt representations. Our project image represents multi-modal data(images, text) as a "partial specification" that allows effective encoding and reconstruction of data.
Project: Asking Your Self-Driving Car to Explain its Decisions, Led by Stefano V. Albrecht, Chris Lucas, and Shay Cohen
Talk Title: Towards a Safe and Trustworthy Autonomous Vehicle: a Human-centric Reasoning System to Explain its Decisions
Abstract: Artificial Intelligence (AI) has shown considerable success in autonomous vehicle (AV) perception, localization and decision-making. For such a safety-critical system, safe AI is a prerequisite before bringing AVs to market. Thus, investigating effective methods to improve AI safety is critical. One step towards safe AI is to make AI explainable and transparent. By introducing explainable AI into AV decision-making, we can understand the causality behind AV decisions, which will help us troubleshoot, discover faults and ultimately improve AI-involved AVs. As a result, we develop a reasoning system to explain autonomous vehicle decisions based on the existing interpretable motion planning called IGP2, which applies Monte Carlo tree search for AV decision-making. By running a parallel Monte Carlo tree search in the background, we can generate past, present and future explanations for users' why, why-not, what, and what-if queries. Preliminary results demonstrate the proposed reasoning system's capability.