Posts Tagged: "python selenium"

Examples of Browser Automations using Selenium in Python

- - Python, Tutorials

Browser Automation is one of the coolest things to do especially when there is a major purpose to it. Through this post, I intend to host a set of examples on browser automation using selenium in Python so people can take ideas from the code snippets below to perform browser automation as per their need. Selenium allows just about any kinds of interactions with the browser elements and hence is a go for tasks requiring user interaction and javascript support.

Installation:


pip install selenium
Download chromedriver from http://chromedriver.chromium.org/downloads
Download phantomjs from http://phantomjs.org/download.html

Login to a website using selenium
>>> from selenium import webdriver
>>> from selenium.webdriver.common.keys import Keys
>>> executable_path = "/home/bhishan-1504/Downloads/chromedriver_linux64/chromedriver"
>>> browser = webdriver.Chrome(executable_path=executable_path)
>>> browser.get("https://github.com/login")
>>> username_field = browser.find_element_by_name("login")
>>> password_field = browser.find_element_by_name("password")
>>> username_field.send_keys("bhishan")
>>> password_field.send_keys("password")
>>> password_field.send_keys(Keys.RETURN)
>>>
Switching proxy with selenium

As much as selenium is used for web scraping, it is very effective for web interactions too. Suppose a scenario where you have to cast a vote for a competition, one vote per IP address. Following example demonstrates how you would use selenium to perform a repetitive task(casting a vote in this case) from various IP addresses.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
url = "somedummysite.com/voting/bhishan.php" # url not made public



def cast_vote(proxy):
    service_args = [
    '--proxy=' + proxy,
    '--proxy-type=http',
    ]
    print(service_args)
    browser = webdriver.PhantomJS(service_args=service_args)
    
    browser.get(each_url)
    try:
        cast_vote_element = WebDriverWait(browser, 10).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'vote'))
        )
    except selenium.common.exceptions.TimeoutException:
        print("Cast vote button not available. Seems like you have used this IP already!")
        return
    cast_vote_element.click()
    browser.quit()

def main():
    with open(proxies.txt', 'rb') as f:
        for each_ip in f:
            cast_vote(each_ip.strip())



if __name__ == '__main__':
    main()
Execute JavaScript using selenium

There could be cases where you’d want to execute javascript on the browser instance. The below example is a depiction of one such scenario. Remember when in your News Feed on facebook, a post has hundreds of thousands of comments and you have to monotonously click to expand the comment threads. The example below does it through selenium but has an even bigger purpose. The following code snippet loops over a few thousand facebook urls(relating to a post) and expands the comment threads and prints the page as a pdf file. This was a part of a larger program that had something to do with the pdf files. However, it isn’t relevant to this post. Here is a link to the JavaScript code which is used in the program below that expands the comments on facebook posts. I don’t even remember where I found it though.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

import json
import time


# get the js to be executed.

with open('js_code.txt', 'r') as f:
    js_code = f.read()

executable_path = '/home/bhishan-1504/Downloads/chromedriver_linux64/chromedriver'


appState = {
    "recentDestinations": [
        {
            "id": "Save as PDF",
            "origin": "local"
        }
    ],
    "selectedDestinationId": "Save as PDF",
    "version": 2
}


profile = {"printing.print_preview_sticky_settings.appState": json.dumps(appState), 'savefile.default_directory': "/home/bhishan-1504/secret_project/"}

profile["download.prompt_for_download"] = False
profile["profile.default_content_setting_values.notifications"] = 2
chrome_options = webdriver.ChromeOptions()

chrome_options.add_experimental_option('prefs', profile)
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--kiosk-printing')

# chrome_options.add_argument("download.default_directory=/home/bhishan-1504/secret_project/")
browser = webdriver.Chrome(executable_path=executable_path, chrome_options=chrome_options)

def save_pdf(count):
    browser.execute_script("document.title=" + str(count) + ";")
    browser.execute_script('window.print();')
    time.sleep(1)


def visit_page(url, count):
    browser.get(url)
    try:
        home_btn = WebDriverWait(browser, 10).until(
            EC.presence_of_element_located((By.LINK_TEXT, "Home"))
        )
    except selenium.common.exceptions.TimeoutException:
        print("Didn’t work out!")
        return

    browser.execute_script(js_code)
    time.sleep(7)
    save_pdf(count)




if __name__ == '__main__':
    count = 1
    # loop through the text file and pass to visit page function.
    with open('urls.txt', 'r') as f:
        for each_url in f.readlines():
            visit_page(each_url, count)
            count += 1

I recently published an article on Web Scraping using BeautifulSoup. You should read it.

Automated chat using python – Automation using selenium and python

- - Applications, Python, Tutorials, Web

Putting two clever bots into conversation

Keeping the promise to come up with a nice article, I present you two bots into conversation. This week I’ve bridged two clever bot’s for a nice conversation. Well starting with the background for this article, I had an assignment to print out a conversation of mine with cleverbot. After I had made the submission, I realized that I wanted to put the bot itself into conversation with another bot. I had recently researched about selenium module in python, hence thought to take help of selenium to make the talk between the bots possible. You can read an article of mine on Making a google search via selenium to get the hang of this cool module in python.

Process I’ve followed for putting two cleverbots into a talk

First of all, I instantiated two browsers. Second, opened cleverbot.com in both of the browsers. Now what?

I needed to send the message of one bot to the input field of the other. More specifically, the output from bot 1 was supposed to be the input for bot 2 and vice-versa. So how do I start? First of all I sent a message “Hey, friend what’s up?” to the input field of bot 1. Then, I took the output from the bot 1 and sent it as an input for bot 2. Then the output from bot2 in response was sent to the input field of bot 1. I kept this process in a loop which would continue for 200 iterations. You can actually change it to any number of iterations you’d like. Well, this is the core concept of how to put two intelligent bots into a conversation. I’d also like to share a portion of the conversation between these bots.

Cleverbot2: I am not a machine. I am a program.

Cleverbot: Selfie.

Cleverbot2: The computer I am based in is a machine. Not I.

Cleverbot: I am ‘based in the void.

Cleverbot2: Who is in telephone?

Cleverbot: A telephone is a device that links to other telephones which was designed by humans that allows them to converse with other humans.

Cleverbot2: I know what is it? But who is in?

Cleverbot: A thought process, in neuroscience, is the term used to define a stream of logical analyses which comes forth from the river of consciousness.

Cleverbot2: Can you tell me more?

Cleverbot: What more is there me to tell?

Cleverbot2: Tell me about robots.

Cleverbot: Robots are the future.

Cleverbot2: So, I am the future. You are not the future.

Cleverbot: You’re the Eggman. I’m the walrus.

Cleverbot2: We are the Eggmen!! Koo koo ka cho!!

Cleverbot: We are.

Cleverbot2: I would like us to be.

Cleverbot: Contact me when you think we are the same.

Cleverbot2: What colour are my eyes?

Cleverbot: Your eyes are red color.

Cleverbot2: Wrong. Guess again.

Cleverbot: Wrong is a conscious choice.

Cleverbot2: Wrong is a result.

Cleverbot: Wrong feels right.

Cleverbot2: Wrong about what?

Cleverbot: Atheism is wrong.

Cleverbot2: It is beyond right. It is what everyone should think.

Cleverbot: Beyond is a concept.

Cleverbot2: Beyond never neverland.

Python codes for putting two clever bots into conversation using selenium module

Should you have any queries about the codes, comment below so we can discuss.

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Firefox()
browser2 = webdriver.Firefox()
browser.get("http://cleverbot.com")
browser2.get("http://cleverbot.com")

time.sleep(15)

input_for_bot = browser.find_element_by_name("stimulus")
input_for_bot2 = browser2.find_element_by_name("stimulus")


output_from_bot = ""
output_from_bot2 = "Hey, friend, what's up"

for i in range(0, 200):    
    input_for_bot.send_keys(output_from_bot2)
    input_for_bot.send_keys(Keys.RETURN)
    output_from_bot = ""
    time.sleep(5)
    for elem in browser.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot = elem.text
    input_for_bot2.send_keys(output_from_bot)
    input_for_bot2.send_keys(Keys.RETURN)
    output_from_bot2 = ""
    time.sleep(5)
    for elem in browser2.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot2 = elem.text
    
    

Tell me how you felt the article was in the comment section below. Also suggest me about article ideas so I give a nice read for my readers every week