Mobile

  1. Rico: A Mobile App Dataset for Building Data-Driven Design Applications 2017

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, Ranjitha Kumar pdf, 2017

  2. Mapping Natural Language Instructions to Mobile UI Action Sequences. ACL 2020

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge pdf, 2020.5

  3. AndroidEnv: A Reinforcement Learning Platform for Android. ViGIL at NAACL 2021

    Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, Doina Precup pdf, 2021.5

  4. Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments. ViGIL at NAACL 2021

    Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer pdf, 2021.4

  5. META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI. Arxiv

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, Kai Yu pdf, 2022.5

  6. Enabling Conversational Interaction with Mobile UI using Large Language Models. CHI 2023

    Bryan Wang, Gang Li, Yang Li pdf, 2022.9

  7. UGIF: UI Grounded Instruction Following. Arxiv

    Sagar Gubbi Venkatesh, Partha Talukdar, Srini Narayanan pdf, 2022.11

  8. From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces. Arxiv

    Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, Kristina Toutanova pdf, 2023.6

  9. Empowering LLM to use Smartphone for Intelligent Task Automation Arxiv

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu pdf, 2023.8

  10. Empowering LLM to use Smartphone for Intelligent Task Automation Arxiv Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, Yunxin Liu pdf, 2023.8

  11. Android in the Wild: A Large-Scale Dataset for Android Device Control Arxiv

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, Timothy Lillicrap pdf, 2023.7

  12. You Only Look at Screens: Multimodal Chain-of-Action Agents Arxiv Zhuosheng Zhang, Aston Zhang pdf, 2023.9

  13. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning Arxiv Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar pdf, 2024.6