Anticipate, Adapt, Act: A Hybrid Framework for Task Planning

Nabanita Dash1, Ayush Kaura1, Shivam Singh1, Ramandeep Singh1,
Snehasis Banerjee2, Mohan Sridharan3, Madhava Krishna1
1Robotics Research Center, IIIT Hyderabad, India, 2TCS Research, Tata Consultancy Services, India, 3School of Informatics, University of Edinburgh, UK
Paper Code
Teaser Image

Consider a robot assisting an elderly human in a kitchen, say with getting a glass of water from the sink to the kitchen counter. Due to mobility and stability limitations, there is uncertainty about whether the human can complete the task successfully; they may end up dropping the water glass. In such situations, we would expect the robot to anticipate the potential for negative outcomes, e.g., the glass being dropped, and either prevent the potential negative outcome, e.g., by fetching the water glass, or prepare to deal with the outcome, e.g., by making sure it has access to the mop needed to clear the water spill. Our framework enables the robot to anticipate such failures; the robot then either prevents failure by completing the task, or prepares to recover from the failure by preparing to clean the potential water spill and complete the task.

Abstract

Anticipating and adapting to potential failures is a key capability that robots require to collaborate effectively with humans in complex domains. This continues to be a challenge despite the impressive performance of state of the art AI planning systems and Large Language Models (LLMs) because of the uncertainty associated with the tasks and their outcomes. Toward addressing this challenge, we present a hybrid framework that integrates the generic prediction capabilities of an LLM with the relational probabilistic sequential decision-making capability of Relational Dynamic Influence Diagram Language (RDDL). For any given task, the robot reasons about the task and the capabilities of the human attempting to complete it; predicts potential failures due to lack of ability (in the human) or relevant domain objects; and executes actions that prevent such failures or help recover from them. Experimental evaluation in the VirtualHome 3D simulation environment demonstrates substantial improvement in task completion, execution time, and collaboration.

Video Explanation

Our Pipeline

Framework's pipeline: (a) LLM takes a prompt of task lists, user preferences, scene description, and user input, to predict a task sequence; (b) RDDL description of domain model and joint goal comprising current and predicted tasks is fed to PROST planner; (c) Plan of actions to be executed by robot (and human) to achieve the goal, including steps to be taken by the robot to prevent or recover from potential failures (e.g., broken objects, spills, item unavailability) in human's action execution.