Posts Tagged: "automate browser using selenium python"

Google Search Using Selenium And Python – Selenium Python Basics

- - Applications, Python, Tutorials

The intentions of this blog is to show through examples some of the most common methods of selenium. Selenium is a library used for automated browser testing. However, in this post we will discuss about using selenium module in python to make a google search. The post breaks down into various blocks explaining on how to open a url in the browser via selenium python, search presence of a url in a page, click links present in a page. These are the necessities to get started with selenium.

Prerequisites
  1. Python
  2. selenium module in Python
  3. Chrome driver (http://chromedriver.chromium.org/downloads)
Installation of selenium through pip in both Linux and Windows

pip install selenium

Google search using selenium python
from selenium import webdriver

search_query = input("Enter the search query")
search_query = search_query.replace(' ', '+') #structuring our search query for search url.
executable_path = "/path/to/chromedriver"
browser = webdriver.Chrome(executable_path=executable_path)


for i in range(20):
    browser.get("https://www.google.com/search?q=" + search_query + "&start=" + str(10 * i))
    matched_elements = browser.find_elements_by_xpath('//a[starts-with(@href, "https://www.thetaranights.com")]')
    if matched_elements:
        matched_elements[0].click()
        break
1. Import statements (Line 1)

It is the import statements that is required for initiating a browser later in our program and passing url parameters to the address bar in the browser. It can be thought of as a driver for the browser. We use various methods on an instance from webdriver.Chrome() instance to control interaction with the browser.

2. Get query for google search (Line 3 and 4)

Here, we are taking a query for the google search via input() in Python3(raw_input() for Python2). Here is an example url for a google search which requires the spaces between the words to be replaced by “+” , an additional parameter start=0 is seen which specifies the search result of page 1. Similarly start=10 gives the search result of page 2.
https://www.google.com/search?q=bhishan+bhandari&start=0“
Hence, after taking the input from the user, we replaces the spaces with +.

3. Instantiate a browser (Line 5)

The statement browser = webdriver.Chrome() opens up a new browser window. We can also customize the browser capabilities such as download location, etc.

4. Opening a url in the browser (Line 9)

For opening a url in the browser, all you need to do is pass the url as an argument to the browser.get method. Remember I’ve given browser.get because we instantiated the browser earlier with browser = webdriver.Chrome(executable_path=executable_path).

5. Searching for a presence of certain url/text in the search result (Line 10 to 15)

The following methods returns the browser elements which match the criteria that the href attribute of the anchor element starts with https://www.thetaranights.com

browser.find_elements_by_xpath('//a[starts-with(@href, "http://www.thetaranights.com"]')

There is also an alternative method find_element_by_xpath for getting the first element that matches the given xpath construct. Then we make a check whether or not any there was at least an element returned from the above statement, which if true we click using click() method on the first element that matched the criteria. This will open the link on the browser. Since the result we are looking for is found and clicked, we exit loop. Else continue searching for the link with the above criteria until 20 pages if not found. You can quit the browser using browser.quit() method.

We generally covered how to open a browser, search for link in the body of the page and click the link. You may also like to read my article on how to login to a website using selenium python.

Compiling My Year With An Interpreter – Python Programming

- - Applications, Python, Tutorials
To my dear readers, without whom this blog would have been a dead planet. Thanks for spending your valuable time here. I wish you a Happy New Year. 2015 is gone with almost an end to my sophomore year. I wouldn’t say year 2015 was any new to me than the others. Few achievements, life lessons and never ending learning curve. My very first article for 2016 will sum up my pythonic experience from 2015. I used some python throughout this semester/year to ease my life. Bear with novice’s automations.
 1. Enroll in 100% off courses at Udemy automatically.

from json import loads
from bs4 import BeautifulSoup
import mechanize
api_key = "8def4868-509c-4f34-8667-f28684483810%3AS7obmNY1SsOfHLhP%2Fft6Z%2Fwc46x8B2W3BaHpa5aK2vJwy8VSTHvaPVuUpSLimHkn%2BLqSjT6NERzxqdvQ%2BpQfYA%3D%3D"
growth_coupon_url = "https://api.import.io/store/data/a5ef05a9-784e-410c-9f84-51e1e8ff413c/_query?input/webpage/url=http%3A%2F%2Fgrowthcoupon.com%2Fcoupon-category%2F100-discount%2F&_user=8def4868-509c-4f34-8667-f28684483810&_apikey=" + api_key
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]
sign_in = br.open("https://www.udemy.com/join/login-popup/")
br.select_form(nr=3)
br["email"] = "email@domain.com"
br["password"] = "password"
logged_in = br.submit()

growth_coupon = br.open(growth_coupon_url)
json_obj = loads(growth_coupon.read())

for course_link in json_obj["results"]:
    try:
        course_page = br.open(str(course_link["couponcode_link"]))
        soup = BeautifulSoup(course_page)
        for link in soup.find_all("a"):
            req_link = link.get('href')
            if 'https://www.udemy.com/payment/checkout' in str(req_link):
                print req_link
                br.open(str(req_link))
                print "success"
                break
    except (mechanize.HTTPError,mechanize.URLError) as e:
        print e.code

This has been my favorite automation throughout the semester. The program checks growthcoupon.com for 100% off coupon codes for paid courses offered at Udemy and enrolls me to those courses. I have uploaded the program to pythonanywhere.com which allows me to have the script run daily(for free accounts) without me having to worry about it. At this time I have over 800 courses at my Udemy account, each courses on an average costs 75$.

2. Conversation between two cleverbots

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Firefox()
browser2 = webdriver.Firefox()
browser.get("http://cleverbot.com")
browser2.get("http://cleverbot.com")

time.sleep(15)

input_for_bot = browser.find_element_by_name("stimulus")
input_for_bot2 = browser2.find_element_by_name("stimulus")


output_from_bot = ""
output_from_bot2 = "Hey, friend, what's up"

for i in range(0, 200):    
    input_for_bot.send_keys(output_from_bot2)
    input_for_bot.send_keys(Keys.RETURN)
    output_from_bot = ""
    time.sleep(5)
    for elem in browser.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot = elem.text
    input_for_bot2.send_keys(output_from_bot)
    input_for_bot2.send_keys(Keys.RETURN)
    output_from_bot2 = ""
    time.sleep(5)
    for elem in browser2.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot2 = elem.text
    
    

This semester, I had to take Cognitive Science course. I enjoyed it. I had this assignment where I had to submit a page of conversation with cleverbot (www.cleverbot.com). I submitted the assignment and later decided to bridge a conversation between two cleverbots. I used selenium module in python to have this done. It was great and kind of felt like an achievement.

3. Using pyautogui before my exams

import pyautogui
import time
pyautogui.click(450,450);pyautogui.typewrite('graphicsnotes');pyautogui.press('enter')
time.sleep(2)
for i in range(107):
  pyautogui.press('right');pyautogui.press('enter')
  pyautogui.hotkey('ctrl','r');pyautogui.hotkey('ctrl','s')
  time.sleep(2)
  pyautogui.press('esc')
  time.sleep(2)

Being lazy to copy notes at class, I had to rely on photos of the notes sent by my friend. I discovered that the photos were all in landscape and were 107 pictures. I had come across pyautogui in A.I Sweigart’s course at Udemy and quickly wrote some 5-7 lines of code to open the picture and rotate the image and save it while I had my dinner. By the way I had no clue I had been enrolled in A.I Sweigart’s course until I opened my account to check which courses I had. All thanks to pythonanywhere.com which runs my program to enroll to Udemy courses on a daily basis.

4. Automate signups and making apps at Dropbox for unlimited space

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser= webdriver.Firefox()
browser.get("https://dropbox.com/register")

list_of_inputs = browser.find_elements_by_xpath("//div/input[starts-with(@id, 'pyxl')]")

list_of_inputs[7].send_keys("first name")

list_of_inputs[8].send_keys("last name")

list_of_inputs[9].send_keys("email")

list_of_inputs[10].send_keys("password")

list_of_inputs[11].click()

password.send_keys(Keys.RETURN)

I had been involved with some people from IIT for some app ideas. We needed cloud space and agreed to use Dropbox. I had been given a bunch of emails with common password. I wrote a program in python to do the signups and later wrote a program to make apps at dropbox and get the API keys and secret keys to access the files in there. Unfortunately the project never continued for some reasons.

Overall, I had a good year but most of the time was spent at college and internship (doing mobile apps using Apache Cordova). For the new year, I will talk to my manager about switching to some python projects. My new year resolution would be to “write more codes and continue blogging about it.” Well that’s all I can think for now.

I would want some suggestions for my new year resolution. Do comment below. Once again A Happy New Year .