Hey computer, use a GUI like a human would with mouse, keyboard and vision (kitware.com)

Reminds me of Scar, which has been around for a long time. Being able to run it in the background meant you could play all sorts of pranks on family members.

http://www.kaitnieks.com/scar/