Category "Applications"

Automated chat using python – Automation using selenium and python

- - Applications, Python, Tutorials, Web

Putting two clever bots into conversation

Keeping the promise to come up with a nice article, I present you two bots into conversation. This week I’ve bridged two clever bot’s for a nice conversation. Well starting with the background for this article, I had an assignment to print out a conversation of mine with cleverbot. After I had made the submission, I realized that I wanted to put the bot itself into conversation with another bot. I had recently researched about selenium module in python, hence thought to take help of selenium to make the talk between the bots possible. You can read an article of mine on Making a google search via selenium to get the hang of this cool module in python.

Process I’ve followed for putting two cleverbots into a talk

First of all, I instantiated two browsers. Second, opened cleverbot.com in both of the browsers. Now what?

I needed to send the message of one bot to the input field of the other. More specifically, the output from bot 1 was supposed to be the input for bot 2 and vice-versa. So how do I start? First of all I sent a message “Hey, friend what’s up?” to the input field of bot 1. Then, I took the output from the bot 1 and sent it as an input for bot 2. Then the output from bot2 in response was sent to the input field of bot 1. I kept this process in a loop which would continue for 200 iterations. You can actually change it to any number of iterations you’d like. Well, this is the core concept of how to put two intelligent bots into a conversation. I’d also like to share a portion of the conversation between these bots.

Cleverbot2: I am not a machine. I am a program.

Cleverbot: Selfie.

Cleverbot2: The computer I am based in is a machine. Not I.

Cleverbot: I am ‘based in the void.

Cleverbot2: Who is in telephone?

Cleverbot: A telephone is a device that links to other telephones which was designed by humans that allows them to converse with other humans.

Cleverbot2: I know what is it? But who is in?

Cleverbot: A thought process, in neuroscience, is the term used to define a stream of logical analyses which comes forth from the river of consciousness.

Cleverbot2: Can you tell me more?

Cleverbot: What more is there me to tell?

Cleverbot2: Tell me about robots.

Cleverbot: Robots are the future.

Cleverbot2: So, I am the future. You are not the future.

Cleverbot: You’re the Eggman. I’m the walrus.

Cleverbot2: We are the Eggmen!! Koo koo ka cho!!

Cleverbot: We are.

Cleverbot2: I would like us to be.

Cleverbot: Contact me when you think we are the same.

Cleverbot2: What colour are my eyes?

Cleverbot: Your eyes are red color.

Cleverbot2: Wrong. Guess again.

Cleverbot: Wrong is a conscious choice.

Cleverbot2: Wrong is a result.

Cleverbot: Wrong feels right.

Cleverbot2: Wrong about what?

Cleverbot: Atheism is wrong.

Cleverbot2: It is beyond right. It is what everyone should think.

Cleverbot: Beyond is a concept.

Cleverbot2: Beyond never neverland.

Python codes for putting two clever bots into conversation using selenium module

Should you have any queries about the codes, comment below so we can discuss.

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Firefox()
browser2 = webdriver.Firefox()
browser.get("http://cleverbot.com")
browser2.get("http://cleverbot.com")

time.sleep(15)

input_for_bot = browser.find_element_by_name("stimulus")
input_for_bot2 = browser2.find_element_by_name("stimulus")


output_from_bot = ""
output_from_bot2 = "Hey, friend, what's up"

for i in range(0, 200):    
    input_for_bot.send_keys(output_from_bot2)
    input_for_bot.send_keys(Keys.RETURN)
    output_from_bot = ""
    time.sleep(5)
    for elem in browser.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot = elem.text
    input_for_bot2.send_keys(output_from_bot)
    input_for_bot2.send_keys(Keys.RETURN)
    output_from_bot2 = ""
    time.sleep(5)
    for elem in browser2.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot2 = elem.text
    
    

Tell me how you felt the article was in the comment section below. Also suggest me about article ideas so I give a nice read for my readers every week 

Grab siteprice and write to google spreadsheet using python

- - Applications, Python, Tutorials

By the end of this read you will be able to grab site price from siteprice.org and write it to google spreadsheet using python. Every website has it’s competition. As our website evolves, we have more competitions and the competitors website also earns good value. It is vital to know the value of our website as well as our competition’s value. Siteprice.org is one of those websites which calculates a website’s value based on different factors.

Putting domain name of website in a text file with one domain per line will be our strategy for querying number of websites’s price. You may wish to put hundreds of websites in this txt file which are your competitions.

Python codes to extract site price and write in google spreadsheet

from bs4 import BeautifulSoup
from urllib2 import urlopen
import gdata.spreadsheet.service
import datetime
rowdict = {}
rowdict['date'] = str(datetime.date.today())
spread_sheet_id = '13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q'
worksheet_id = 'od6'
client = gdata.spreadsheet.service.SpreadsheetsService()
client.debug = True
client.email = 'email@domain.com'
client.password = 'password'
client.source = 'siteprice'
client.ProgrammaticLogin()
with open('websitesforprice.txt') as f:
    for line in f:
        soup = BeautifulSoup(urlopen("http://www.siteprice.org/website-worth/" + line).read())
        rowdict['website'] = str(line)
        rowdict['price'] = soup.find(id="lblSitePrice").string
        client.InsertRow(rowdict,spread_sheet_id, worksheet_id)

1. Line 1 to 4

These lines are import statements. Here in this program, we are using various python libraries. Gdata is used to access google spreadsheet. We are using BeautifulSoup because it allows us to get data via id which we will use to get the price of a website. Datetime is used to get the current date. Urlopen us used to open the webpage which contains the data we want.

2.Line 5 to 14

In order to write the extracted rank to google spreadsheet programmatically we are using the gdata module. In order to write to a spreadsheet we need the spreadsheet id, worksheet id and a dictionary containing values we want to write to the spreadsheet. The dictionary contains key as the column header and value as the string that is to be written to the spreadsheet(website, price, date for our program).

Go to docs.google.com when logged in and create a new spreadsheet. Fill the first three columns of the first row as website, price and date respectively. All the letter should be in lower case and no whitespaces. Now when you have created a new spreadsheet, take a look to the url. The url looks something like this one

https://docs.google.com/spreadsheets/d/13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q/edit#gid=0

The spreadsheet id(mentioned earlier) is present in the url.

13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q” in the above url is the spreadsheet id we need. By default the worksheet id is ‘od6‘.

Basically line 5 to 14 are codes to access google spreadsheet.

3. Line 15 to 20

Since we’re writing a program that can extract alexa ranks for hundreds of websites and append it to google spreadsheet, therefore taking url from console input is never a good solution. We have to write the url of websites we want to take care of in a text file. Each website in a single line in the format www.domain.com. Make sure there is a valid website, one in each line because we will read the url from python line by line.

Line 17 makes a soup element out of the url which has the information we are looking for. The soup element is of different websites in each iteration. Line 18 stores the value of the domain in the key “website” of json rowdict. Line 19 stores the price of the website in the key price of json rowdict. You can see we use BeutifulSoup to get data via id. Finally line 20 pushes the entire json element to google spreadsheet. This piece of codes runs for the number of times equal to the line in text file.

Thanks for reading :) Enjoy!! . If you have any questions regarding the post, feel free to comment below.

Google Search Using Selenium And Python – Selenium Python Basics

- - Applications, Python, Tutorials

The intentions of this blog is to show through examples some of the most common methods of selenium. Selenium is a library used for automated browser testing. However, in this post we will discuss about using selenium module in python to make a google search. The post breaks down into various blocks explaining on how to open a url in the browser via selenium python, search presence of a url in a page, click links present in a page. These are the necessities to get started with selenium.

Prerequisites
  1. Python
  2. selenium module in Python
  3. Chrome driver (http://chromedriver.chromium.org/downloads)
Installation of selenium through pip in both Linux and Windows

pip install selenium

Google search using selenium python
from selenium import webdriver

search_query = input("Enter the search query")
search_query = search_query.replace(' ', '+') #structuring our search query for search url.
executable_path = "/path/to/chromedriver"
browser = webdriver.Chrome(executable_path=executable_path)


for i in range(20):
    browser.get("https://www.google.com/search?q=" + search_query + "&start=" + str(10 * i))
    matched_elements = browser.find_elements_by_xpath('//a[starts-with(@href, "https://www.thetaranights.com")]')
    if matched_elements:
        matched_elements[0].click()
        break
1. Import statements (Line 1)

It is the import statements that is required for initiating a browser later in our program and passing url parameters to the address bar in the browser. It can be thought of as a driver for the browser. We use various methods on an instance from webdriver.Chrome() instance to control interaction with the browser.

2. Get query for google search (Line 3 and 4)

Here, we are taking a query for the google search via input() in Python3(raw_input() for Python2). Here is an example url for a google search which requires the spaces between the words to be replaced by “+” , an additional parameter start=0 is seen which specifies the search result of page 1. Similarly start=10 gives the search result of page 2.
https://www.google.com/search?q=bhishan+bhandari&start=0“
Hence, after taking the input from the user, we replaces the spaces with +.

3. Instantiate a browser (Line 5)

The statement browser = webdriver.Chrome() opens up a new browser window. We can also customize the browser capabilities such as download location, etc.

4. Opening a url in the browser (Line 9)

For opening a url in the browser, all you need to do is pass the url as an argument to the browser.get method. Remember I’ve given browser.get because we instantiated the browser earlier with browser = webdriver.Chrome(executable_path=executable_path).

5. Searching for a presence of certain url/text in the search result (Line 10 to 15)

The following methods returns the browser elements which match the criteria that the href attribute of the anchor element starts with https://www.thetaranights.com

browser.find_elements_by_xpath('//a[starts-with(@href, "http://www.thetaranights.com"]')

There is also an alternative method find_element_by_xpath for getting the first element that matches the given xpath construct. Then we make a check whether or not any there was at least an element returned from the above statement, which if true we click using click() method on the first element that matched the criteria. This will open the link on the browser. Since the result we are looking for is found and clicked, we exit loop. Else continue searching for the link with the above criteria until 20 pages if not found. You can quit the browser using browser.quit() method.

We generally covered how to open a browser, search for link in the body of the page and click the link. You may also like to read my article on how to login to a website using selenium python.

Grab Whois Information And Write To Google Spreadsheet

Hello Guys, Here I am with yet another program that can benefit you and many search engine optimizers. By the end of this read you will be able to write a program to extract the whois information of a number of domains stored in a text file and write the information about the domain in a google spreadsheet which has now been a medium to share data and findings online. As a Search Engine Optimizer, you need to keep track a number of websites including your competitions. Here I offer you a simple python program to keep track of. On the other hand if you are not a SEO expert like myself, you can still use this script to track various websites you are used to.

Prerequisites before beginning to code

We are going to have two files one of which is a .py file where we code our program. The other is a text file with .txt extention where we store the domain names we want to find whois information for. The text file must contain a domian name in a format www.domain.com one per each line.

Next, we need to create a google spreadsheet where we intend to write the whois information so we can share with others. Direct your browser to https://docs.google.com/spreadsheets/ and create a new spreadsheet named “Whois Info”. Once done, create three rows namely “website”, “whoisinformation” and “date”. The name of the domain name will be under the row website, the whois information will be under the row whoisinformation and the date we queried the whois information will remain under the row date.

Python code to extract whois information and write to google spreadsheet

from bs4 import BeautifulSoup
from urllib2 import urlopen
import gdata.spreadsheet.service
import datetime
rowdict = {}
rowdict['date'] = str(datetime.date.today())
spread_sheet_id = '1zE8Qe8wmC271hG2uW4XE68btUks79xX0OG-O4KDl_Mo'
worksheet_id = 'od6'
client = gdata.spreadsheet.service.SpreadsheetsService()
client.debug = True
client.email = "email@domain.com"
client.password = 'password'
client.source = 'whoisinfo'
client.ProgrammaticLogin()
with open('websitesforwhois.txt') as f:
    for line in f:
        soup = BeautifulSoup(urlopen("http://www.checkdomain.com/cgi-bin/checkdomain.pl?domain=" + str(line)).read())
        for pre in soup.find_all("pre"):
            whois_info = str(pre.string)
        #print whois_info
        rowdict['website'] = str(line)
        rowdict['whoisinformation'] = whois_info
        client.InsertRow(rowdict,spread_sheet_id, worksheet_id)

1. Line 1 to 4

These are the import statements. We use BeautifulSoup to make a soup object out of a url response. Urlopen to get the response of a url. Gdata to access the google spreadsheet. Datetime to get the current system time.

2. Line 5 and 6

In our program, we require to access the google spreadsheet and write to it hence we are using gdata module. Now in order to write to spreadsheet, we need to pass the data as a dictionary or generally known as json which has data as a key:value pair. Rowdict is a variable storing the data to pass to google spreadsheet. On line 6, we store the current date to the key “date” which if you remember is a row at our spreadsheet.

3. Line 7 to 14

Line 7 to 14 is a procedure to connect/access a specific google spreadsheet. We require spread_sheet_id and worksheet_id. Take a look to the url of your spreadsheet. The url looks something like this one

https://docs.google.com/spreadsheets/d/1VbNph0TfFetKLU8hphrEyuNXlJ-7m628p8Sbu82o8lU/edit#gid=0

The spreadsheet id(mentioned earlier) is present in the url. “1VbNph0TfFetKLU8hphrEyuNXlJ-7m628p8Sbu82o8lU” in the above url is the spreadsheet id we need. By default the worksheet id is ‘od6‘.

On line 13 is the client.source assigned to string ‘whoisinfo’. This is the file name or the spreadsheet name. Remember we named our spreadsheet “Whois Info”. The client.source is the spreadsheet name which is written in small alphabets excluding white spaces.

4. Line 15 to 16

Line 15 opens the text file where we’ve stored the names of the domain. Line 16 helps iterate through each lines in the file. At each iteration, the domain name at each line is stored to variable line.

5 Line 17

On line 17, we query the page giving the whois information for us and make a soup object out of it by invoking the BeautifulSoup method over the url response. The reason we are making a soup object is that we can access required data via tags and the data we need is inside a <pre></pre> tag.

6 Line 18 to 19

Now we know that there is only one “pre” tag in the soup element. We therefore iterate to find a pre tag and store the information inside of the pre tag to a variable whois_info.

7 Line 21 to 23

On line 21, we are assigning the domain name to the key “website” of the dictionary rowdict. On line 22, we are assigning the whois information stored in the variable whois_info to the key “whoisinformation” of the dictionary rowdict. Note that the key of the dictionary must match to the row name in our spreadsheet. Line 23 pushes the dictionary to the google spreadsheet and writes to it. The iteration goes until the domain names at a text file is finished.

If you have any questions/confusions regarding the article or code, please mention below in comments so we can discuss. Thanks for reading

Gui Automation With Python

- - Applications, Python, Tutorials, Web
Hello Readers. It has been a bit longer delay in publishing my article. However today I will present to my awesome readers, an introduction to a GUI automation module in python (I.e pyautogui). Pyautogui is a GUI automation module for python2 and python3 which provides methods for controlling mouse and keystrokes. This decent module can be used to create bots to automate the repetitive tasks while you can enjoy your coffee. Pyautogui can do anything a human user sitting at the computer can do, except spill coffee on the keyboard” says the geek responsible for this cool module.

Follow the link below to have pyautogui installed on your machine.

https://pyautogui.readthedocs.org/en/latest/install.html

With no further iteration about the introduction, I would like to present few basics about the module.

1. Locating coordinates of the mouse cursor.

>>> import pyautogui

>>> pyautogui.position()

(850, 504)

>>>

It returns the current x and y coordinate of the mouse cursor position. In a computer screen the left top point is the origin or (0,0)

2. Moving the mouse cursor

>>> pyautogui.moveTo(10,10)

>>> pyautogui.moveTo(10,10,duration=1)

The moveTo function takes x-coordinate and y-coordinate as parameters while duration can be passed as the third parameter which is optional used to specify the amount of time in seconds to reach to the specified coordinate. The second one is humanly approach while the first is an instant movement of cursor.

3. Clicking

>>> pyautogui.click(80,80)

>>> pyautogui.doubleClick(80,80)

>>> pyautogui.rightClick(80,80)

Clicking on a certain coordinate on the screen is possible via click method while it also provides doubleClick, rightClick methods taking parameter x-coordinate and y-coordinate in all cases.

4. Keystrokes

For typing, we will first need to locate an appropriate type area. Therefore, you might want to use this method after click on some coordinate which is writable. You can use two or more statements to run simultaneously one after another by separating each statement by semicolon. For instance, I’ve specified the coordinates of the url bar on my browser and then typed my name on it via following commands/statements

>>> pyautogui.click(50,80);pyautogui.typewrite(“Bhishan”)

>>> pyautogui.click(50,80);pyautogui.typewrite(“Bhishan”, interval=0.2)

We can pass an optional parameter interval in seconds to specify the time in seconds between each letter or keystroke.

5. Hot Key

The hotkey method can be used in cases we need to press two or more keys at the same time. A handy example is Ctrl + S to save a file or Ctrl + Shift + q to quit

>>> pyautogui.hotkey(‘Ctrl’,’Shift’,’q’)

You can see all the possible mapping keys of the keystrokes via this method

>>> pyautogui.KEYBOARD_KEYS

Well that’s enough to get you started and good at GUI automation via pyautogui. Below is a bot I have made using the module to automate a boring task for myself. Iterating the story behind the need for the bot. I am a fourth semester CS undergrad student(I mean lazy student). I never take notes in any of the classes I attend. At the time of exams, I rely on the photos of my friend’s notes which they send me. As always I got the photos but this time all the pictures were at landscape mode by some chance(near about 100-110 images). It would be kind of distracting to rotate each image to read it. So I wrote some 7-8 lines of code to make a bot that would open each image file rotate it and save it while I have my dinner. I’ve used time module along with pyautogui to keep some time gap between the statements.

import pyautogui
import time
pyautogui.click(450,450);pyautogui.typewrite('graphicsnotes');pyautogui.press('enter')
time.sleep(2)
for i in range(107):
  pyautogui.press('right');pyautogui.press('enter')
  pyautogui.hotkey('ctrl','r');pyautogui.hotkey('ctrl','s')
  time.sleep(2)
  pyautogui.press('esc')
  time.sleep(2)
  time.sleep(2)


The concept is to click anywhere on the desktop screen. I choose some random coordinate (450,450). Then typing folder name to locate the folder followed by enter to open it. Then iteratively clicking right to select the image file, opening the image file by pressing enter, followed by hotkey ‘Ctrl’ + ‘r’ to rotate the image clockwise then ‘Ctrl’ + ‘s’ to save it. Finally pressing esc to close the file and repeating the process to go to the next image file. I had total of 107 images so I’ve iterated 107 times in my program to reach up to all the image files. Tell me how you felt the article was in the comments section below so I can come up with a cool set of articles for the next week. Till then, happy automation with pyautogui 🙂

Here read the docs https://pyautogui.readthedocs.org

Python Selenium Time Tkinter Pyvirtualdisplay

- - Applications, Python, Technology, Web
Making use of libraries available and building something useful has always been my favorite. However, I haven’t yet been involved in making/contributing to one. I don’t even know the procedure to be involved in building libraries. My latest program also makes use of various libraries such as selenium, time, tkinter. In a nutshell, the program fetches jokes from www.laughfactory.com and displays a message box with a joke in a timely interval. I usually run the program while I am at work(internship). I’ve set 10 minutes interval between the jokes to be shown. By the way, I used time module and it’s time.sleep(time_in_seconds) method for dealing with intervals between each message box to appear. The program isn’t full fledged and you can make additions to it on your own for customization according to your need. I made use of selenium module along with pyvirtualdisplay to fetch the jokes and mute the display of the browser, since it would be distracting. The program only fetches 20 latest jokes from laughfactory.com and periodically displays one at a time in a messagebox. A little humor is always good at work. Anyway that’s my personal thought you may disagree to. I used tkinter module for displaying the message box. That’s all about the program. Have a look at the codes. Thank god, the import statements < program statements.

Python codes to get timely jokes

import tkinter
import tkinter.messagebox as mbox
from time import sleep
from selenium import webdriver
from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Firefox()
browser.get("http://www.laughfactory.com/jokes/latest-jokes")

sleep(40)
#print("finding jokes")
jokes = [str(joke.text) for joke in browser.find_elements_by_xpath("//div/p[starts-with(@id,'joke_')]")]

#print("found")
browser.quit()
display.stop()

window = tkinter.Tk()
window.wm_withdraw()

for joke in jokes:
    sleep(600)
    mbox.showinfo('Bored? Enjoy the JOk3!', joke)

Thank you for spending your time here. I have always tried to make my articles short and maybe now succeeded. You can utilize the saved time for another read in this blog, may be :p .Tell me how you felt the article was in the comments section below. This keeps me motivated to publish good content.

Compiling My Year With An Interpreter – Python Programming

- - Applications, Python, Tutorials
To my dear readers, without whom this blog would have been a dead planet. Thanks for spending your valuable time here. I wish you a Happy New Year. 2015 is gone with almost an end to my sophomore year. I wouldn’t say year 2015 was any new to me than the others. Few achievements, life lessons and never ending learning curve. My very first article for 2016 will sum up my pythonic experience from 2015. I used some python throughout this semester/year to ease my life. Bear with novice’s automations.
 1. Enroll in 100% off courses at Udemy automatically.

from json import loads
from bs4 import BeautifulSoup
import mechanize
api_key = "8def4868-509c-4f34-8667-f28684483810%3AS7obmNY1SsOfHLhP%2Fft6Z%2Fwc46x8B2W3BaHpa5aK2vJwy8VSTHvaPVuUpSLimHkn%2BLqSjT6NERzxqdvQ%2BpQfYA%3D%3D"
growth_coupon_url = "https://api.import.io/store/data/a5ef05a9-784e-410c-9f84-51e1e8ff413c/_query?input/webpage/url=http%3A%2F%2Fgrowthcoupon.com%2Fcoupon-category%2F100-discount%2F&_user=8def4868-509c-4f34-8667-f28684483810&_apikey=" + api_key
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]
sign_in = br.open("https://www.udemy.com/join/login-popup/")
br.select_form(nr=3)
br["email"] = "email@domain.com"
br["password"] = "password"
logged_in = br.submit()

growth_coupon = br.open(growth_coupon_url)
json_obj = loads(growth_coupon.read())

for course_link in json_obj["results"]:
    try:
        course_page = br.open(str(course_link["couponcode_link"]))
        soup = BeautifulSoup(course_page)
        for link in soup.find_all("a"):
            req_link = link.get('href')
            if 'https://www.udemy.com/payment/checkout' in str(req_link):
                print req_link
                br.open(str(req_link))
                print "success"
                break
    except (mechanize.HTTPError,mechanize.URLError) as e:
        print e.code

This has been my favorite automation throughout the semester. The program checks growthcoupon.com for 100% off coupon codes for paid courses offered at Udemy and enrolls me to those courses. I have uploaded the program to pythonanywhere.com which allows me to have the script run daily(for free accounts) without me having to worry about it. At this time I have over 800 courses at my Udemy account, each courses on an average costs 75$.

2. Conversation between two cleverbots

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Firefox()
browser2 = webdriver.Firefox()
browser.get("http://cleverbot.com")
browser2.get("http://cleverbot.com")

time.sleep(15)

input_for_bot = browser.find_element_by_name("stimulus")
input_for_bot2 = browser2.find_element_by_name("stimulus")


output_from_bot = ""
output_from_bot2 = "Hey, friend, what's up"

for i in range(0, 200):    
    input_for_bot.send_keys(output_from_bot2)
    input_for_bot.send_keys(Keys.RETURN)
    output_from_bot = ""
    time.sleep(5)
    for elem in browser.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot = elem.text
    input_for_bot2.send_keys(output_from_bot)
    input_for_bot2.send_keys(Keys.RETURN)
    output_from_bot2 = ""
    time.sleep(5)
    for elem in browser2.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot2 = elem.text
    
    

This semester, I had to take Cognitive Science course. I enjoyed it. I had this assignment where I had to submit a page of conversation with cleverbot (www.cleverbot.com). I submitted the assignment and later decided to bridge a conversation between two cleverbots. I used selenium module in python to have this done. It was great and kind of felt like an achievement.

3. Using pyautogui before my exams

import pyautogui
import time
pyautogui.click(450,450);pyautogui.typewrite('graphicsnotes');pyautogui.press('enter')
time.sleep(2)
for i in range(107):
  pyautogui.press('right');pyautogui.press('enter')
  pyautogui.hotkey('ctrl','r');pyautogui.hotkey('ctrl','s')
  time.sleep(2)
  pyautogui.press('esc')
  time.sleep(2)

Being lazy to copy notes at class, I had to rely on photos of the notes sent by my friend. I discovered that the photos were all in landscape and were 107 pictures. I had come across pyautogui in A.I Sweigart’s course at Udemy and quickly wrote some 5-7 lines of code to open the picture and rotate the image and save it while I had my dinner. By the way I had no clue I had been enrolled in A.I Sweigart’s course until I opened my account to check which courses I had. All thanks to pythonanywhere.com which runs my program to enroll to Udemy courses on a daily basis.

4. Automate signups and making apps at Dropbox for unlimited space

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser= webdriver.Firefox()
browser.get("https://dropbox.com/register")

list_of_inputs = browser.find_elements_by_xpath("//div/input[starts-with(@id, 'pyxl')]")

list_of_inputs[7].send_keys("first name")

list_of_inputs[8].send_keys("last name")

list_of_inputs[9].send_keys("email")

list_of_inputs[10].send_keys("password")

list_of_inputs[11].click()

password.send_keys(Keys.RETURN)

I had been involved with some people from IIT for some app ideas. We needed cloud space and agreed to use Dropbox. I had been given a bunch of emails with common password. I wrote a program in python to do the signups and later wrote a program to make apps at dropbox and get the API keys and secret keys to access the files in there. Unfortunately the project never continued for some reasons.

Overall, I had a good year but most of the time was spent at college and internship (doing mobile apps using Apache Cordova). For the new year, I will talk to my manager about switching to some python projects. My new year resolution would be to “write more codes and continue blogging about it.” Well that’s all I can think for now.

I would want some suggestions for my new year resolution. Do comment below. Once again A Happy New Year . 

Is It A WordPress Website Checker Script In Python

- - Applications, Python, Tutorials

By the end of this read, you will be able to code a program that will check for a number of domain names stored in a text file to verify whether or not a website is powered by wordpress. In this program we will write the result for each website in a google spreadsheet for later use but you can apply certain code if a website is wordpress powered at the mean time.

Python script to check if a bunch website is wordpress powered

Below is the program written in python programming language which reads a text file storing the names of the domain one per each line and checks if it a wordpress website and writes the status in a google spreadsheet.

 

from bs4 import BeautifulSoup
import mechanize
import gdata.spreadsheet.service
import datetime
rowdict = {}
rowdict['date'] = str(datetime.date.today())
spread_sheet_id = '1mvebf95F5c_QUCg4Oep_oRkDRe40QRDzVfAzt29Y_QA'
worksheet_id = 'od6'
client = gdata.spreadsheet.service.SpreadsheetsService()
client.debug = True
client.email = "email@domain.com"
client.password = 'password'
client.source = 'iswordpress'
client.ProgrammaticLogin()
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]
base_url = br.open("http://www.isitwp.com/")
with open('websitesforwpcheck.txt') as f:
    for line in f:
        rowdict['website'] = str(line)
        br.select_form(nr=0)
        br["q"] = str(line)
        isitwp_response = br.submit()
        isitwp_response = isitwp_response.read()
        if "Good news everyone" in a:
            rowdict['iswordpresswebsite'] = "yes"
        else:
            rowdict['iswordpresswebsite'] = "no"
        client.InsertRow(rowdict,spread_sheet_id, worksheet_id)

Isitwordpresswebsite code explanation

Before beginning , we need to create a spreadsheet and make three rows with names website, date, isitwordpresswebsite

1. Line 1 to 4

These are the import statements for the libraries we will use in our program. We use BeautifulSoup to convert a response from a request as a soup object. Mechanize is used in order to skip the length of code for sessions and cookies. Gdata is used to connect to our google account in order to access the spreadsheet we want to work with. Datetime is used to get the current date at the time of script run.

2. Line 5

In line 5, we create an empty dictionary where we later create a key:value pairs for date, website name and status of a website(is it wordpress or not). Also google spreadsheet accepts a dictionary item/json which is then written to the spreadsheet. In line 6 we store the current date at the time of script run to a key “date” which is later in our program pushed to the google spreadsheet.

3. Line 7 to 14

Now, before proceeding forward, we need to create a google spreadsheet. Now another step is to take a look at the url of the spreadsheet we created. We need the spreadsheet id to access the spreadsheet via our program. Below is a screenshot of how to get the spreadsheet id. The worksheet id is ‘od6‘ by default. Line 9 to 14 gets us logged in to our google account and accesses the spreadsheet we want to work with.

4. Line 15 to 18

In line 15, we use the mechanize module’s method to initiate a browser. Line 16 states to ignore the robots.txt file. Line 17 adds a user agent to the browser. Line 18 lets us open the website “isitwp.com” which we will be using to check if a website is wordpress powered or not.

5. Line 19

Line 19 allows us to open the text file where we have the names of the domain(one per each line).

6. Line 20 to 21

In line 20, we iterate through the lines of the file where we have stored domain names. At each iteration, the domain name at the current line is stored to the variable line. In line 21, we create a key:value pair where we store the domain name at each iteration to the key “website.

7. Line 22 to 30

Line 22 is a statement to select the form of index 0 i.e the first form present in the website isitwp.com. Now value of the name attribute of the seachbox is “q” which is where we store the name of the domain before submitting the form. Line 24 submits the form after passing the value of domain name in the br[‘q’] and stores the response in a variable isitwp_response. Line 25 is the complete webpage response stored in variable isitwp_response. In line 26 we check if “Good news everyone” substring is present in the response which means the website is powered by wordpress else it is not. Line 27 then makes a key:value pair where the key “iswordpresswebsite” is given value “yes” if the condition on line 26 passed and “no” if the condition failed. Now remember the key of the dictionary rowdict must be the name of the row in our spreadsheet.

This way we can test if a website is wordpress powered for a number of website names stored in a text file(one at each line). Thanks for reading :) . If you have any questions regarding the article or the code, comment below so we can discuss on that part.

Enroll In 100 Off Courses At Udemy Automatically Python Codes To Get Paid Courses

- - Applications, Python, Tutorials
Udemy is a teaching and learning platform with loads of courses in various categories. Now very often different coupon codes are available for purchasing courses in minimal amount or with a 100% discount. Various websites serve these coupon codes. One of those websites which I rely on is GrowthCoupon.com

Now, I am not writing a review of 100% off coupon providers. Through this post I will explain my code which I am using to extract the 100% off coupon codes from growthcoupon.com and then get those courses automatically. I have automated my code so that I do not need to worry about new coupon codes available and can save my time. The below code enrolls you in 10 latest 100% off courses available at growthcoupon.com when run a single time. You may wish to automate the script every hour or so.

Get 100%off Udemy courses automatically using python

from json import loads
from bs4 import BeautifulSoup
import mechanize
api_key = "8def4868-509c-4f34-8667-f28684483810%3AS7obmNY1SsOfHLhP%2Fft6Z%2Fwc46x8B2W3BaHpa5aK2vJwy8VSTHvaPVuUpSLimHkn%2BLqSjT6NERzxqdvQ%2BpQfYA%3D%3D"
growth_coupon_url = "https://api.import.io/store/data/a5ef05a9-784e-410c-9f84-51e1e8ff413c/_query?input/webpage/url=http%3A%2F%2Fgrowthcoupon.com%2Fcoupon-category%2F100-discount%2F&_user=8def4868-509c-4f34-8667-f28684483810&_apikey=" + api_key
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13")]
sign_in = br.open("https://www.udemy.com/join/login-popup/")
br.select_form(nr=3)
br["email"] = "email@domain.com"
br["password"] = "password"
logged_in = br.submit()

growth_coupon = br.open(growth_coupon_url)
json_obj = loads(growth_coupon.read())

for course_link in json_obj["results"]:
    try:
        course_page = br.open(str(course_link["couponcode_link"]))
        soup = BeautifulSoup(course_page)
        for link in soup.find_all("a"):
            req_link = link.get('href')
            if 'https://www.udemy.com/payment/checkout' in str(req_link):
                print req_link
                br.open(str(req_link))
                print "success"
                break
    except (mechanize.HTTPError,mechanize.URLError) as e:
        print e.code

The above program is a pure python code that extracts 10 latest 100% off coupon codes from GrowthCoupon.com and then enrolls you in those courses automatically.

1. Line 1 to 3

The first three lines are the import statements. In our program, we are using three python libraries. Amongst them, mechanize is used to login to the udemy account. BeautifulSoup is used to get the data on the basis of tags. Here in our program we use BeautifulSoup to get the links in certain page. Json’s loads is used to load the json response.

2. Line 4 and 5

We are using import.io API in order to extract data from growthcoupon. I got to know about this very cool resource in my Programming Synthesis class at my college. Here’s How to get and use import.io API. We store the API in a variable api_key. Then concatenate to the growth_coupon_url which is the standard post request url to get data in json format from growthcoupon.

3. Line 6 to 13

From line 6 to 13 is the procedure to login to a website (udemy in our case). Line 6 initializes a browser. Line 7 says ignore the robots.txt file. Line 8 adds a user agent to the browser. Line 9 opens the login url in the browser we initiated earlier.

The next thing you will need is the form you want to work with. By this I mean this is the login form. All you need to do is go to the username box ->> right click on it->> go to the inspect elements option. Now scroll up until you find the first form tag. In most cases you will find the form name attribute but some of the websites do not have this. If there exists then the value given to the name attribute under the form tag is the thing you need to access the form. Another way to access forms is by their index. The first form is indexed 0. Now in case the form name is not available, you will need to find how many forms are present in the login url(basically most of the websites have only one form because all you want the login page to do is login if authenticated). In this case the form index is 3.

Now you need to know the variable name that is assigned to take the value you enter to the email/username and password section. To get these values inspect element when you are inside the fields email/username and password. Below is a snapshot to give you insights of the variables you want to take care of.

udemyloginform

4. Line 15 and 16

Here on line 15, we are opening the url that gives us the data from growthcoupon in json format. Line 16 loads the url as a json object.

5. Line 18

Our json object is stored in json_obj variable. But the data we need is stored inside an array which is the value for the key “results” Hence we are iterating through this array.

6. Line 20

Now we open the couponcode link which is the value of the key “couponcode_link”. This url is present in each index of the array. On each loop the particular index’s url’s response is stored in variable course_page.

7. Line 21

We then convert the page response to a soup element by invoking the BeautifulSoup method over course_page.

8. Line 22 to 27

Now we want to iterate through each links found in the soup element. The url for enrolling in the udemy course starts with the string “https://www.udemy.com/payment/checkout”. Hence we check if the string is a substring of the link at each iteration. If the condition satisfies, we open that link to enroll ourselves in that course. Well that’s the end of the code that works.

Thanks for reading :) Enjoy ! If you have any questions regarding the codes/post, comment below.