Gui Automation With Python

- - Uncategorized

Hello Readers. It has been a bit longer delay in publishing my article. However today I will present to my awesome readers, an introduction to a GUI automation module in python (I.e pyautogui). Pyautogui is a GUI automation module for python2 and python3 which provides methods for controlling mouse and keystrokes. This decent module can be used to create bots to automate the repetitive tasks while you can enjoy your coffee.

Pyautogui can do anything a human user sitting at the computer can do, except spill coffee on the keyboard” says the geek responsible for this cool module.

Follow the link below to have pyautogui installed on your machine.

https://pyautogui.readthedocs.org/en/latest/install.html

With no further iteration about the introduction, I would like to present few basics about the module.

1. Locating coordinates of the mouse cursor.

>>> import pyautogui

>>> pyautogui.position()

(850, 504)

>>>

It returns the current x and y coordinate of the mouse cursor position. In a computer screen the left top point is the origin or (0,0)

2. Moving the mouse cursor

>>> pyautogui.moveTo(10,10)

>>> pyautogui.moveTo(10,10,duration=1)

The moveTo function takes x-coordinate and y-coordinate as parameters while duration can be passed as the third parameter which is optional used to specify the amount of time in seconds to reach to the specified coordinate. The second one is humanly approach while the first is an instant movement of cursor.

3. Clicking

>>> pyautogui.click(80,80)

>>> pyautogui.doubleClick(80,80)

>>> pyautogui.rightClick(80,80)

Clicking on a certain coordinate on the screen is possible via click method while it also provides doubleClick, rightClick methods taking parameter x-coordinate and y-coordinate in all cases.

4. Keystrokes

For typing, we will first need to locate an appropriate type area. Therefore, you might want to use this method after click on some coordinate which is writable. You can use two or more statements to run simultaneously one after another by separating each statement by semicolon. For instance, I’ve specified the coordinates of the url bar on my browser and then typed my name on it via following commands/statements

>>> pyautogui.click(50,80);pyautogui.typewrite(“Bhishan”)

>>> pyautogui.click(50,80);pyautogui.typewrite(“Bhishan”, interval=0.2)

We can pass an optional parameter interval in seconds to specify the time in seconds between each letter or keystroke.

5. Hot Key

The hotkey method can be used in cases we need to press two or more keys at the same time. A handy example is Ctrl + S to save a file or Ctrl + Shift + q to quit

>>> pyautogui.hotkey(‘Ctrl’,’Shift’,’q’)

You can see all the possible mapping keys of the keystrokes via this method

>>> pyautogui.KEYBOARD_KEYS

Well that’s enough to get you started and good at GUI automation via pyautogui. Below is a bot I have made using the module to automate a boring task for myself. Iterating the story behind the need for the bot. I am a fourth semester CS undergrad student(I mean lazy student). I never take notes in any of the classes I attend. At the time of exams, I rely on the photos of my friend’s notes which they send me. As always I got the photos but this time all the pictures were at landscape mode by some chance(near about 100-110 images). It would be kind of distracting to rotate each image to read it. So I wrote some 7-8 lines of code to make a bot that would open each image file rotate it and save it while I have my dinner. I’ve used time module along with pyautogui to keep some time gap between the statements.

import pyautogui
import time
pyautogui.click(450,450);pyautogui.typewrite('graphicsnotes');pyautogui.press('enter')
time.sleep(2)
for i in range(107):
  pyautogui.press('right');pyautogui.press('enter')
  pyautogui.hotkey('ctrl','r');pyautogui.hotkey('ctrl','s')
  time.sleep(2)
  pyautogui.press('esc')
  time.sleep(2)
  time.sleep(2)


The concept is to click anywhere on the desktop screen. I choose some random coordinate (450,450). Then typing folder name to locate the folder followed by enter to open it. Then iteratively clicking right to select the image file, opening the image file by pressing enter, followed by hotkey ‘Ctrl’ + ‘r’ to rotate the image clockwise then ‘Ctrl’ + ‘s’ to save it. Finally pressing esc to close the file and repeating the process to go to the next image file. I had total of 107 images so I’ve iterated 107 times in my program to reach up to all the image files. Tell me how you felt the article was in the comments section below so I can come up with a cool set of articles for the next week. Till then, happy automation with pyautogui :)

Here read the docs https://pyautogui.readthedocs.org

Post Tags:

bhishan

I am Bhishan Bhandari, a CS student and life hacker. I specialize in automation. I sell my services on fiverr. You can hire me for projects here Buy Services Follow me on github for code updates Github You can always communicate your thoughts/wishes/questions to me at bbhishan@gmail.com