Google Search Using Selenium And Python – Selenium Python Basics

- - Applications, Python, Tutorials

The intentions of this blog is to show through examples some of the most common methods of selenium. Selenium is a library used for automated browser testing. However, in this post we will discuss about using selenium module in python to make a google search. The post breaks down into various blocks explaining on how to open a url in the browser via selenium python, search presence of a url in a page, click links present in a page. These are the necessities to get started with selenium.

Prerequisites
  1. Python
  2. selenium module in Python
  3. Chrome driver (http://chromedriver.chromium.org/downloads)
Installation of selenium through pip in both Linux and Windows

pip install selenium

Google search using selenium python
from selenium import webdriver

search_query = input("Enter the search query")
search_query = search_query.replace(' ', '+') #structuring our search query for search url.
executable_path = "/path/to/chromedriver"
browser = webdriver.Chrome(executable_path=executable_path)


for i in range(20):
    browser.get("https://www.google.com/search?q=" + search_query + "&start=" + str(10 * i))
    matched_elements = browser.find_elements_by_xpath('//a[starts-with(@href, "https://www.thetaranights.com")]')
    if matched_elements:
        matched_elements[0].click()
        break
1. Import statements (Line 1)

It is the import statements that is required for initiating a browser later in our program and passing url parameters to the address bar in the browser. It can be thought of as a driver for the browser. We use various methods on an instance from webdriver.Chrome() instance to control interaction with the browser.

2. Get query for google search (Line 3 and 4)

Here, we are taking a query for the google search via input() in Python3(raw_input() for Python2). Here is an example url for a google search which requires the spaces between the words to be replaced by “+” , an additional parameter start=0 is seen which specifies the search result of page 1. Similarly start=10 gives the search result of page 2.
https://www.google.com/search?q=bhishan+bhandari&start=0“
Hence, after taking the input from the user, we replaces the spaces with +.

3. Instantiate a browser (Line 5)

The statement browser = webdriver.Chrome() opens up a new browser window. We can also customize the browser capabilities such as download location, etc.

4. Opening a url in the browser (Line 9)

For opening a url in the browser, all you need to do is pass the url as an argument to the browser.get method. Remember I’ve given browser.get because we instantiated the browser earlier with browser = webdriver.Chrome(executable_path=executable_path).

5. Searching for a presence of certain url/text in the search result (Line 10 to 15)

The following methods returns the browser elements which match the criteria that the href attribute of the anchor element starts with https://www.thetaranights.com

browser.find_elements_by_xpath('//a[starts-with(@href, "http://www.thetaranights.com"]')

There is also an alternative method find_element_by_xpath for getting the first element that matches the given xpath construct. Then we make a check whether or not any there was at least an element returned from the above statement, which if true we click using click() method on the first element that matched the criteria. This will open the link on the browser. Since the result we are looking for is found and clicked, we exit loop. Else continue searching for the link with the above criteria until 20 pages if not found. You can quit the browser using browser.quit() method.

We generally covered how to open a browser, search for link in the body of the page and click the link. You may also like to read my article on how to login to a website using selenium python.

Bhishan Bhandari [22] A one man army producing contents and maintaining this blog. I am a hobbyist programmer and enjoy writing scripts for automation. If you'd like a process to be automated through programming, I also sell my services at Fiverr . Lately, I like to refresh my Quora feeds. Shoot me messages at bbhishan@gmail.com  

Leave a Reply

Your email address will not be published. Required fields are marked *