Category "Tutorials"

Copying mutable objects in Python

- - Python, Tutorials

An assignment statement in python does not create copies of objects. It binds the name to the object. While working with mutable objects and/or collections of mutable objects, it creates inconsistencies and hence it would be of interest to us to have ways to make real copies of the objects. Essentially, we would require copies such that modifying it would not modify the original object. An example for what happens when we use assignment statements to make copies of mutable objects.

 

>>> fruits = ["apple", "mango", "orange"]
>>> fruits_copy = fruits
>>> id(fruits)
140684382177688
>>> id(fruits_copy)
140684382177688
>>> fruits_copy.append("grapes")
>>> fruits_copy
['apple', 'mango', 'orange', 'grapes']
>>> fruits
['apple', 'mango', 'orange', 'grapes']
>>>

 

The above example demonstrates how using an assignment statement for copying a mutable object has an impact on the original object when the later one is modified.

 

Python’s built-in mutable collections like lists, dicts, and sets can be copied by calling their factory functions on an existing collection.

 

new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)

However, this only makes a shallow copy of the objects and goes only a level deep in the recursion tree.

 

A shallow copy refers to construction of a new object followed by populating it with the references to the child objects found in the original object. This implies that a shallow copy is only one level deep. The recursion tree on copying does not proceed further on childs of the object.

 

>>> x = [[1, 2 , 3], [4, 5, 6]]
>>> x_copy = list(x)
>>> id(x) == id(x_copy)
False
>>> x_copy.append([7, 8, 9])
>>> x_copy
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> x
[[1, 2, 3], [4, 5, 6]]

 

In the above example, since x_copy is a new object and the contents upto one level deep are copied to the object, it does not affect the original object. But, remember a shallow copy is only one level deep while all other childs are only referenced and not copied as new ones. Following example should explain it.

 

>>> x = [[1, 2, 3], [4, 5, 6]]
>>> x_copy = list(x)
>>> x_copy[0][0] = "Changed"
>>> x_copy
[['Changed', 2, 3], [4, 5, 6]]
>>> x
[['Changed', 2, 3], [4, 5, 6]]

 

On the flip side, a deep copy is a recursive process. It initially constructs a new object followed by recursively populating it with copies of the child objects found in the original object. Deep copy walks the complete object tree to create a fully independent clone of the original object and all of its childs.

 

>>> import copy
>>> x = [[1, 2, 3], [4, 5, 6]]
>>> x_copy = copy.deepcopy(x)
>>> id(x) == x_copy
False
>>> x_copy
[[1, 2, 3], [4, 5, 6]]
>>> x
[[1, 2, 3], [4, 5, 6]]
>>> x_copy[0][0] = "New Object confirmation"
>>> x_copy
[['New Object confirmation', 2, 3], [4, 5, 6]]
>>> x
[[1, 2, 3], [4, 5, 6]]
>>>

The above example shows a deep copy and hence change in the copied object does not have any effect on the original object.

When the argument default value is mutable

- - Python, Tutorials

Python offers us to specify that a function argument is optional by providing a default value to it. While this is widely used and one of the major features of the language, it can lead to confusions when enough thought is not given to the implementation details of the feature. Especially when the default value is a mutable.

Example of a function implementation with optional argument with default value as a mutable.

>>> def example(numbers=[]):
...   numbers.append(1)
...   return numbers
...
>>>

Simply put, a mutable object can be changed after it is created, and an immutable object can’t. In python list, dictionaries are mutable objects while integers, strings, tuple, etc are immutable objects.

A common misconception is that the optional argument will be set to the default value provided each time the function when called without supplying the value for the optional argument. Let us see.

>>> example()
[1]
>>> example()
[1, 1]
>>> example()
[1, 1, 1]
>>> example()
[1, 1, 1, 1]
>>> example()
[1, 1, 1, 1, 1]
>>>

We would generally assume that each time example() is called without the optional argument, numbers would be assigned to and empty list. Inside the function, 1 would be appended to the list and returned. Well, our hypothesis is false in this case.

This is because, the default value for a function is only evaluated once which is during the function definition. This implies that the default value for the optional parameter numbers is initialized to the default value,  an empty list at the time of the function definition. Any further calls of the function example would then utilize the same list everytime it is called therefore the behavior we saw in the examples above.

How to mess up a Python codebase coming from Java Background.

- - Python, Tutorials

In python, you can make use of the augmented assignment operator to increase or decrease the value of a variable by 1. An augmented assignment is generally used to replace a statement where an operator takes a variable as one of its arguments and then assigns the result back to the same variable. It is very common to make use of augmented assignment operator in python.

Other languages such as JAVA has ++ and operators to perform similar operations. These operators can be used for either pre-increment or post-increment. When you use pre-increment, the value of the expression is taken after performing the increment. When you use post-increment, though, the value of the expression is taken before incrementing, and stored for later use, after the result of incrementing is written back into the variable.

 

Python doesn’t implement this directive for increment/decrement operations. However, what python does implement is the unary operator + and . For an example – when used ahead of a variable acts as a unary negation operator. This also entails that you can use any number of such operators ahead of an expression or variable.

 

Therefore ++variable and variable is perfectly legal in python. However the results aren’t the same for Python and JAVA. Following code blocks should help understand the intentions of the blog.

 

>>> i = 7
>>> - i
-7
>>> + i
7
>>> -- i
7
>>> ++ i
7
>>> --- i
-7
>>> +++ i
7
>>> i ++
  File "", line 1
    i ++
        ^
SyntaxError: invalid syntax
>>> i --
  File "", line 1
    i --
       ^
SyntaxError: invalid syntax
>>>

 

As a JAVA developer unknown of this scenario, you could clearly mess up a python codebase. However the unary operators + and are not valid post variable which totally makes sense.

Flask Essentials – Handling 400 and 404 requests in Flask REST API

- - Python, Tutorials

Flask is a microframework for Python based on Werkzeug, Jinja 2. Flask is simple and straightforward. However less has been discussed in terms of handling non-existent resource request on the Flask API. Since we are talking to and fro using AJAX and what not, it is highly important for the client to consume a proper acknowledgement of some sort in cases of failure. It is this very small implementation that does the work for us however we seem to never bring this up on online content.

Flask provides a mechanism through errorhandler decorator to implement a custom error handling based on the error status code. These errors are generally termed as route errors. Below is a sample implementation of the 400 and 404 errors on the Flask API.

@APP.errorhandler(400)
def not_found(error):
    """
    Gives error message when any bad requests are made.
    Args:
        error (string):
    Returns:
        Error message.
    """
    print error
    return make_response(jsonify({'error': 'Bad request'}), 400)
@APP.errorhandler(404)
def not_found(error):
    """
    Gives error message when any invalid url are requested.
    Args:
        error (string): 
    Returns:
        Error message.
    """
    print error
    return make_response(jsonify({'error': 'Not found'}), 404)

Similarly, you can handle application errors too. However, it is important to note that the application errors arise due to code blocks on the server and hence appropriate handling is essential. I might as well do a follow up on this post for application error handling with appropriate logging.

Appium Python Client

- - Python, Tutorials

Using Appium with Python

Appium is an open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. While the main purpose of appium is to perform automation testing, it can be utilized for variety of other things too. Appium has client libraries in various languages including Python.

Prerequisites:

  • Appium Node Server

  • Appium-Python-Client

The node server process usually listens at port 4723. You can customize the port you want the appium to run on using the -p or –port parameter. There are numerous options that you may utilize as per your need. Another useful parameter is –log that can be used to direct the log to a file.

appium --help

The idea behind the Appium-Python-Client is we create a driver that allows us to perform actions on the scope of the application under automation. We specify the capabilities that we want the driver to have. The driver basically maps the methods to the commands and passes it on to the appium process and waits for a http response. It is the adb commands that is being used under the hood for android. The Appium-Python-Client can be thought of as a wrapper.

Following is a simple example that uses Appium-Python-Client
from appium import webdriver
import time


def make_navigation_driver(apk_path, package_name, initial_activity, platform='Android', platform_version='8.1.0', device_id='emulator-5554', server_api='http://localhost:4723/wd/hub', **kwargs):
    """
    :param apk_path:
    :param package_name:
    :param initial_activity:
    :param platform:
    :param platform_version:
    :param device_id:
    :param server_api:
    :param kwargs: Probably will use for setting custom timeout and other capabilities. Implement later?
    :return:
    """
    desired_caps = {
        'platformName': platform,
        'platformVersion': platform_version,
        'deviceName': device_id,
        # 'app': apk_path, # Saving apk travel time to device since it is already installed on the device.
        'appPackage': package_name,
        'appActivity': initial_activity
    }

    return webdriver.Remote(server_api, desired_caps)


def perform_simple_automation(driver):
    """
    :param driver:
    :return:
    """
    clickables = driver.find_elements_by_android_uiautomator('new UiSelector().clickable(true)')

    # navigate to next activity or view via clicking on the first clickable.
    if len(clickables):
        clickables[0].click()

        
if __name__ == '__main__':
    platform = "Android"
    platform_version = "7.1.1"
    device_id = "367e7aa0"
    apk_path = "/Users/bhishan/Downloads/udemy.apk"
    package_name = "com.udemy.android"
    initial_activity = ".SplashActivity"
    driver = make_navigation_driver(apk_path, package_name, initial_activity, platform=platform,
                                    platform_version=platform_version, device_id=device_id)

perform_simple_automation(driver)

Appium Example Gist

Automated chat using python – Automation using selenium and python

- - Applications, Python, Tutorials, Web

Putting two clever bots into conversation

Keeping the promise to come up with a nice article, I present you two bots into conversation. This week I’ve bridged two clever bot’s for a nice conversation. Well starting with the background for this article, I had an assignment to print out a conversation of mine with cleverbot. After I had made the submission, I realized that I wanted to put the bot itself into conversation with another bot. I had recently researched about selenium module in python, hence thought to take help of selenium to make the talk between the bots possible. You can read an article of mine on Making a google search via selenium to get the hang of this cool module in python.

Process I’ve followed for putting two cleverbots into a talk

First of all, I instantiated two browsers. Second, opened cleverbot.com in both of the browsers. Now what?

I needed to send the message of one bot to the input field of the other. More specifically, the output from bot 1 was supposed to be the input for bot 2 and vice-versa. So how do I start? First of all I sent a message “Hey, friend what’s up?” to the input field of bot 1. Then, I took the output from the bot 1 and sent it as an input for bot 2. Then the output from bot2 in response was sent to the input field of bot 1. I kept this process in a loop which would continue for 200 iterations. You can actually change it to any number of iterations you’d like. Well, this is the core concept of how to put two intelligent bots into a conversation. I’d also like to share a portion of the conversation between these bots.

Cleverbot2: I am not a machine. I am a program.

Cleverbot: Selfie.

Cleverbot2: The computer I am based in is a machine. Not I.

Cleverbot: I am ‘based in the void.

Cleverbot2: Who is in telephone?

Cleverbot: A telephone is a device that links to other telephones which was designed by humans that allows them to converse with other humans.

Cleverbot2: I know what is it? But who is in?

Cleverbot: A thought process, in neuroscience, is the term used to define a stream of logical analyses which comes forth from the river of consciousness.

Cleverbot2: Can you tell me more?

Cleverbot: What more is there me to tell?

Cleverbot2: Tell me about robots.

Cleverbot: Robots are the future.

Cleverbot2: So, I am the future. You are not the future.

Cleverbot: You’re the Eggman. I’m the walrus.

Cleverbot2: We are the Eggmen!! Koo koo ka cho!!

Cleverbot: We are.

Cleverbot2: I would like us to be.

Cleverbot: Contact me when you think we are the same.

Cleverbot2: What colour are my eyes?

Cleverbot: Your eyes are red color.

Cleverbot2: Wrong. Guess again.

Cleverbot: Wrong is a conscious choice.

Cleverbot2: Wrong is a result.

Cleverbot: Wrong feels right.

Cleverbot2: Wrong about what?

Cleverbot: Atheism is wrong.

Cleverbot2: It is beyond right. It is what everyone should think.

Cleverbot: Beyond is a concept.

Cleverbot2: Beyond never neverland.

Python codes for putting two clever bots into conversation using selenium module

Should you have any queries about the codes, comment below so we can discuss.

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

browser = webdriver.Firefox()
browser2 = webdriver.Firefox()
browser.get("http://cleverbot.com")
browser2.get("http://cleverbot.com")

time.sleep(15)

input_for_bot = browser.find_element_by_name("stimulus")
input_for_bot2 = browser2.find_element_by_name("stimulus")


output_from_bot = ""
output_from_bot2 = "Hey, friend, what's up"

for i in range(0, 200):    
    input_for_bot.send_keys(output_from_bot2)
    input_for_bot.send_keys(Keys.RETURN)
    output_from_bot = ""
    time.sleep(5)
    for elem in browser.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot = elem.text
    input_for_bot2.send_keys(output_from_bot)
    input_for_bot2.send_keys(Keys.RETURN)
    output_from_bot2 = ""
    time.sleep(5)
    for elem in browser2.find_elements_by_xpath('.//span[@class="bot"]'):
        output_from_bot2 = elem.text
    
    

Tell me how you felt the article was in the comment section below. Also suggest me about article ideas so I give a nice read for my readers every week 

What are the must know features of python programming language

- - Python, Tutorials
List comprehensions

One of the major features of python is list comprehension. It is a natural way of creating a new list where each element is the result of some operations applied to each member of another sequence of an iterable. The construct of a list comprehension is such that it consists of square brackets containing an expression followed by a for clause then by zero or more for or if clause. List comprehensions always returns a list.

Simple example
squares = [x**2 for x in range(10)]

In a rather real usage scenarios, the expression after the bracket ‘[‘ is a call to a method/function.

some_list = [function_name(x) for x in some_iterable]
Generators

Before learning about generators, it is important and essential to understand what an iterable is. Putting it simple, an iterable is an object that can be looped over. Therefore, a list, string, dictionary, file, etc are iterable objects.
A generator is something that simplifies creating iterators. More specifically, a generator is a function that produces a sequence of results instead of a single value.

When a generator function is called, it returns a generator object without even beginning execution of the called function. Now when the next() method is called for the first time, the function starts executing until it reaches yield statement. Hence yielded value is returned by the next() call.

Sample Example
def gen_example():
    print("Beginning of the function")
    for i in range(5):
        print("before yield")
        yield i
        print("after yield")
    print("End of the function")
Generators Expressions

It is the generator version of list comprehension. Everything is same as the list comprehension except that it returns a generator.

Sample Example
squares = (x**2 for x in range(5))
Docstrings

A docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. Such a docstring becomes the __doc__ special attribute of that object. In fact, every module should have a docstring. Additionally all the functions and classes exported by a module should have a docstrings. Also, the public methods including constructor should also have docstrings.

Sample Example
def sum_elements(elements):
    """Returns the sum of elements of the passed list"""
    return sum(elements)
*args and **kwargs

*args and *kwargs allow you to pass a variable number of arguments to a function. It is not know beforehand about how many arguments can be passed to your function.

*args is used to send a non-keyworded variable length argument list to the function.
def func_with_args(*argv):
    for arg in argv:
        print(arg)

func_with_args('gopal', 'ramesh', 'paresh')

The above code produces the result as follows:

gopal
ramesh
paresh

On the other hand, **kwargs allows you to pass keyworded variable length of arguments to a function. Below is a basic example

def  func_with_kwargs(**kwargs):
    if kwargs:
        for key, value in kwargs.items():
            print("%s ==> %s" %(key, value))

func_with_kwargs(customer= "Gopal", salesman = "Ramesh")

Following is the output of the above code:

customer ==> Gopal
salesman ==> Ramesh

This is an open-ended article. Please comment below about the features you think should not be missed. Thanks for reading.

How to recover lenovo ideatab A1000-F after multiple incorrect pattern attempts

- - Tutorials

By the end of this read you will be able to recover your lenovo tablet that has been locked due to multiple incorrect pattern attempts.

Starting quickly, follow the steps below.

Recover lenovo tablet after several incorrect pattern attempts

Before beginning , make sure you have enough battery to start the process.

1. Press the volume up key that is present on the right side of your tablet while holding it with screen facing towards you. Immediately press the power on/off button that is present on the top edge of your tablet.

2. If you performed step one well, your tablet should power on. At this point you get to a list of menu with several options instead of the regular startup screen.

3. On the list you get an option of Wipe user data/factory reset. Use your tablet’s volume down button to walk through the menu/options. When on wipe user data/factory reset option, use your tablet’s volume up button to select the option.

4. Now you will be taken to another list of menu/option. Here you will see a big list of options where several options says NO. Only one of the option/menu is YES. Similar to step 3, use your tablet’s volume down button to walk through the list. When on YES option, use your tablet’s volume up button to select the option.

5. Finally you get a list of menu for the last time. This time amongst several options, you have an option reading reboot now. Use your tablet’s volume up button to select this option. On doing so the settings you choose earlier gets saved and all the user data including lock pattern get wiped out. You might have to wait for around a minute for the device to startup correctly. Initially you will see a black screen with lenovo at the middle for a while. Wait until it starts correctly. You need a little more patience.

When the device has started up correctly, you will be prompted for license agreement and wifi settings, etc. Walk through these steps which are easy and you will finally be able to open your tablet with a swipe. I hope this helped. Thanks for reading :)

Grab siteprice and write to google spreadsheet using python

- - Applications, Python, Tutorials

By the end of this read you will be able to grab site price from siteprice.org and write it to google spreadsheet using python. Every website has it’s competition. As our website evolves, we have more competitions and the competitors website also earns good value. It is vital to know the value of our website as well as our competition’s value. Siteprice.org is one of those websites which calculates a website’s value based on different factors.

Putting domain name of website in a text file with one domain per line will be our strategy for querying number of websites’s price. You may wish to put hundreds of websites in this txt file which are your competitions.

Python codes to extract site price and write in google spreadsheet

from bs4 import BeautifulSoup
from urllib2 import urlopen
import gdata.spreadsheet.service
import datetime
rowdict = {}
rowdict['date'] = str(datetime.date.today())
spread_sheet_id = '13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q'
worksheet_id = 'od6'
client = gdata.spreadsheet.service.SpreadsheetsService()
client.debug = True
client.email = 'email@domain.com'
client.password = 'password'
client.source = 'siteprice'
client.ProgrammaticLogin()
with open('websitesforprice.txt') as f:
    for line in f:
        soup = BeautifulSoup(urlopen("http://www.siteprice.org/website-worth/" + line).read())
        rowdict['website'] = str(line)
        rowdict['price'] = soup.find(id="lblSitePrice").string
        client.InsertRow(rowdict,spread_sheet_id, worksheet_id)

1. Line 1 to 4

These lines are import statements. Here in this program, we are using various python libraries. Gdata is used to access google spreadsheet. We are using BeautifulSoup because it allows us to get data via id which we will use to get the price of a website. Datetime is used to get the current date. Urlopen us used to open the webpage which contains the data we want.

2.Line 5 to 14

In order to write the extracted rank to google spreadsheet programmatically we are using the gdata module. In order to write to a spreadsheet we need the spreadsheet id, worksheet id and a dictionary containing values we want to write to the spreadsheet. The dictionary contains key as the column header and value as the string that is to be written to the spreadsheet(website, price, date for our program).

Go to docs.google.com when logged in and create a new spreadsheet. Fill the first three columns of the first row as website, price and date respectively. All the letter should be in lower case and no whitespaces. Now when you have created a new spreadsheet, take a look to the url. The url looks something like this one

https://docs.google.com/spreadsheets/d/13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q/edit#gid=0

The spreadsheet id(mentioned earlier) is present in the url.

13mX6ALRRtGlfCzyDNCqY-G_AqYV4TpE7rq1ZNNOcD_Q” in the above url is the spreadsheet id we need. By default the worksheet id is ‘od6‘.

Basically line 5 to 14 are codes to access google spreadsheet.

3. Line 15 to 20

Since we’re writing a program that can extract alexa ranks for hundreds of websites and append it to google spreadsheet, therefore taking url from console input is never a good solution. We have to write the url of websites we want to take care of in a text file. Each website in a single line in the format www.domain.com. Make sure there is a valid website, one in each line because we will read the url from python line by line.

Line 17 makes a soup element out of the url which has the information we are looking for. The soup element is of different websites in each iteration. Line 18 stores the value of the domain in the key “website” of json rowdict. Line 19 stores the price of the website in the key price of json rowdict. You can see we use BeutifulSoup to get data via id. Finally line 20 pushes the entire json element to google spreadsheet. This piece of codes runs for the number of times equal to the line in text file.

Thanks for reading :) Enjoy!! . If you have any questions regarding the post, feel free to comment below.

What are the most interesting web scraping modules for python

- - Python, Tutorials, Web
Python programming language is in the hype for over a decade. It is the most recommended language for the beginner programmers since it’s syntax are readable by almost every non-programmers too. At the same time recommended for web scraping, automation and data science. However python comes short in terms of speed when compared to languages such as C++ and JAVA. The plus for python programming language is the wide range of enthusiastic contributors and users around the globe. There are countless modules for doing various domain specific tasks which makes it even more popular today. From web scraping to gui automation, there are modules for almost everything. Here, in this post, I will list some of the most used and interesting python modules for web scraping that are lifesaver for a programmer.

Popular python modules for web scraping

1. Mechanize

mechanize is a popular python module for it allows creation of a browser instance. It also maintains sessions which aids as a toolkit to obtain tasks like login, signup automation, etc.

2. BeautifulSoup

BeautifulSoup is another beautiful python module which aids scraping the data required from html/xmls via tags. With beautiful you can scrape almost everything because it aids different methods like searching via tags, finding all links, etc.

3. Selenium

Although selenium is a well known module for automated browser testing, it can be used as a web scraping tool as well. I promise you, it pays pretty well. With methods to find element via ids, name, class, etc, selenium would allow you to have anything on the website.

4. lxml

lxml is another wonderful library for parsing xml/htmls, however I would say beautifulsoup beats it in terms of usability. You could option to use any of the modules among lxml and BeautifulSoup since they pretty much do the same stuff.

I have used all of the above modules extensively in my projects and they allowed me to move faster. I was able to do some cool things with these modules. For example: automating conversation between two cleverbots(AI featured bots), getting paid courses at udemy, finding the most popular facebook fan page among my friends,etc. Therefore I totally recommend them. Below is a link to some of the most interesting things I’ve done with these modules.

Cool stuffs with python

Tell me how you felt the article was in the comments section below and/or you can add some of your favorite modules too. There’s always thanks for reading