Biomechanical User Models for Mobile Device Control based on Reinforcement Learning

Smartphones are everywhere, but not all users have the same experience. While mobile apps are often tested for usability, they don’t always consider the unique hand biomechanics of individual users, such as hand size or age-related dexterity. This oversight can result in apps that work well for some, but are challenging for others.

The primary goal of this work is to create biomechanical user models that interact with interactive smartphone applications within human physiological constraints. We aim to build on existing biomechanical hand models and teach them to interact with physical but interactive devices (i.e., smartphone or tablet). These user models will be based on the theory of computational rationality. The core assumption is that users act according to what is best for them, given the constraints imposed by their limitations (cognitive limitations, motor limitations) and their experience of the task environment. Through RL, these agents will be trained to interact with touchscreens in a way that mimics human behavior. The biomechanical models will serve as a foundation, allowing us to adjust physiological characteristics of these agents, such as finger length or joint flexibility. By changing these characteristics, we can evaluate the subsequent changes in interaction behavior and usability. This approach will provide more profound insights into how different biomechanical factors influence user experience, ultimately leading to more inclusive application designs.

Please contact Patrick Ebel if you are interested in this topic or for any clarifications. Below you can find some related work that might be of interest.

Related Literature:

• Computational Rationality as a Theory of Interaction: https://dl.acm.org/doi/abs/10.1145/3491102.3517739
• Breathing Life into Biomechnical User Models: https://dl.acm.org/doi/abs/10.1145/3526113.3545689
• Adapting User Interfaces with Model-based Reinforcement Learning: https://dl.acm.org/doi/10.1145/3411764.3445497
• Touchscreen Typing As Optimal Supervisory Control: https://dl.acm.org/doi/fullHtml/10.1145/3411764.3445483
• Simulating Interaction Movements via Model Predictive Control: https://dl.acm.org/doi/10.1145/3577016

Tools and Developer Suites:

• Biomechnial Models in OpenSim: https://simtk-confluence.stanford.edu:8443/display/OpenSim/Upper+Extremity+Model
• Detailed Hand Models in MyoSuite: https://sites.google.com/view/myosuite
• MuJoCo: https://github.com/google-deepmind/mujoco
• sim-mpc: https://github.com/mkl4r/sim-mpc

Counterpart