Idiomatic Python – Writing better Python

- - Python, Tutorials

This is a follow-up post of Idiomatic Python – Looping Approaches. The purpose of the article is to highlight on better code and encourage it.

Looping over dictionary keys

>>> books_price = {
...     'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17,
...     'The Self-Taught Programmer: The Definitive Guide to Programming Professionally': 15.09,
...     'The Art of Computer Programming, Volumes 1-4A Boxed Set': 174.96
... }
>>> for book in books_price:
...     print(book)
...

The above code snippet should not be used for mutating the dictionary. You do not want to change the size of the dictionary while you actively iterate over it.

>>> for book in books_price:
...     if book.startswith('The'):
...             del books_price[book]
... 
Traceback (most recent call last):
  File "", line 1, in 
RuntimeError: dictionary changed size during iteration
>>>
>>> books_price
{'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17, 'The Self-Taught Programmer: The Definitive Guide to Programming Professionally': 15.09}
>>>

If by any chance you wrapped the above code around a broad exception, you will then have an inconsistent data. See how in the above example one key value pair has been removed from the books_price dictionary.

Proper way of mutating dictionary while iterating over it’s keys:
>>> books_price = {'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17, 'The Art of Computer Programming, Volumes 1-4A Boxed Set': 174.96, 'The Self-Taught Programmer: The Definitive Guide to Programming Professionally': 15.09}
>>>
>>> for book in books_price.keys():
...     if book.startswith('The'):
...         del books_price[book]
...
>>>
>>> books_price
{'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17}
>>>

The books_price.keys() copies all the keys of the books_price dictionary and makes a list. What we are really doing is iterating over the list and modifying the dictionary. This way we avoid trying to mutate the dictionary while actively iterating on it.

Note that in python3 dict.keys() only creates an iterable that provides a dynamic view on the keys of the dictionary while not actually making a new list of keys. This is also known as dictionary view. Therefore you have to explicitly pass the dict.keys() to a list class i.e list(dict.keys())

Construct a dictionary from sequence pairs

>>> books = [
... 'Clean Code: A Handbook of Agile Software Craftsmanship',
... 'The Art of Computer Programming, Volumes 1-4A Boxed Set',
... 'The Self-Taught Programmer: The Definitive Guide to Programming Professionally']
>>> prices = [42.17, 174.96, 15.09]
>>>
>>> from itertools import izip
>>> books_price = dict(izip(books, prices))
>>> books_price
{'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17, 'The Art of Computer Programming, Volumes 1-4A Boxed Set': 174.96, 'The Self-Taught Programmer: The Definitive Guide to Programming Professionally': 15.09}
>>>

 

You could also use zip instead of izip. zip provides a list while izip provides an iterator instead of a list, hence the name izip (I for iterator). In python3 though, zip is equivalent to izip.

 

>>> books_price = dict(zip(books, prices))
>>> books_price
{'Clean Code: A Handbook of Agile Software Craftsmanship': 42.17, 'The Art of Computer Programming, Volumes 1-4A Boxed Set': 174.96, 'The Self-Taught Programmer: The Definitive Guide to Programming Professionally': 15.09}
>>>

Looping over dictionary keys and values

>>> for key in books_price:
...     print key, ' => ', books_price[key]
...
Clean Code: A Handbook of Agile Software Craftsmanship  =>  42.17
The Art of Computer Programming, Volumes 1-4A Boxed Set  =>  174.96
The Self-Taught Programmer: The Definitive Guide to Programming Professionally  =>  15.09
>>>

While the above code works as expected, it needs to re-hash every key and do a value lookup. OR you could do something like this:

>>> for key, value in books_price.items():
...     print key, " => ", value
...
Clean Code: A Handbook of Agile Software Craftsmanship  =>  42.17
The Art of Computer Programming, Volumes 1-4A Boxed Set  =>  174.96
The Self-Taught Programmer: The Definitive Guide to Programming Professionally  =>  15.09
>>>

The problem with the above code is that it creates a huge list of the keys and values. A better approach would be:

>>> for key, value in books_price.iteritems():
...     print key, " => ", value
...
Clean Code: A Handbook of Agile Software Craftsmanship  =>  42.17
The Art of Computer Programming, Volumes 1-4A Boxed Set  =>  174.96
The Self-Taught Programmer: The Definitive Guide to Programming Professionally  =>  15.09
>>>

iteritems() returns an iterator instead of list. Note that in python3, iteritems() is not present and python3’s items() is equivalent to iteritems() in python2.

You should read the predecessor article to this one which encourages on writing better python circling around list looping approaches at http://www.thetaranights.com/idiomatic-python-looping-approaches/

Zip files using Python

- - Python, Tutorials

Zipping files can be one part of a more complex operations that we perform using programming. This can usually happen when you are working on a data pipeline and/or products requiring data movement. Python has easy methods available for zipping files and directories. For the records, a ZIP is an archive file format that supports lossless data compression. A ZIP file may contain one or more files or directories that may have been compressed.

How to archive files/directories using shutil?

The shutil module offers a number of high-level operations on files and collections of files. Following code block will zip the files and directories present in the the source directory provided as the third argument to the make_archive function from shutil.

>>> from shutil import make_archive
>>> make_archive("July17-2018", "zip", "/home/bhishan-1504/shutil_test_archive")
Details about the parameters of make_archive function:

base_name : It is the name of the file to create. This filename is expected to be without the format specific extension.

format : It is the archive format which could be one of “zip”, “tar”, “gztar”, “bztar” or any other registered format.

root_dir : It is the directory that will be the root directory of the archive i.e we typically chdir into ‘root_dir’ before creating the archive.

base_dir : It is the directory where we start archiving from; ie. ‘base_dir’ will be the common prefix of all files and directories in the archive.

The make_archive function returns the filename of the archived file. Note that owner and group are used when creating a tar archive. By default, it uses the current owner and group.

How to archive selective files/directories using zipfile?

We also have control over what files and directories should be archived rather than the entire directory tree. This can be achieved by the following code block:

>>> from zipfile import ZipFile
>>> with ZipFile("testarchive.zip", "w") as zip_buff:
...     zip_buff.write("1.txt")
...     zip_buff.write("3.txt")
...
>>>

All we do is write to the ZipFile object the files to be archived.

Details about the parameters of the ZipFile class:

file: Either the path to the file, or a file-like object. If it is a path, the file will be opened and closed by ZipFile.

mode: The mode can be either read “r”, write “w” or append “a”.

compression: ZIP_STORED (no compression) or ZIP_DEFLATED (requires zlib).
allowZip64: if True ZipFile will create files with ZIP64 extensions when needed, otherwise it will raise an exception when this would be necessary.

Python Assignment Expression – PEP 572 – Python3.8

- - Python, Tutorials

A recent buzz in the Python Community is PEP 572’s acceptance for Python3.8 .
PEP stands for Python Enhancement Proposals and each such PEPs are assigned a number by the PEP editors and once assigned are never changed.

What exactly is PEP 572(Directly from PEP 572)?

Abstract

This is a proposal for creating a way to assign to variables within an expression using the notation NAME := expr. A new exception, TargetScopeError is added, and there is one change to evaluation order.

Rationale

Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts.
Additionally, naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few name := markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer.

What are assignment expressions?

As of now, in Python, assignment has to be a statement. This restricts for an example assignments from within if or while statements. Therefore following would be a Syntax Error in Python:

if x = foo():
    # do something
else:
    # do something else

If this were made valid in python, it would have led to errors for confusing an assignment (=) with comparison operator (==). The code would then still execute without errors but produce unintended results.

Interestingly PEP 572 introduces a new operator := that assigns and returns a value. This is no replacement for the assignment operator and has a different purpose. Let us see the use case:

In most contexts where arbitrary Python expressions can be used, a named expression can appear. This is of the form NAME := expr where expr is any valid Python expression other than an unparenthesized tuple, and NAME is an identifier.
The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:

Our scenario: We want to process the contents of a file in a chunk-wise fashion. What we would naturally do is :

chunk = file.read(64)
while chunk:
    process(chunk)
    chunk = file.read(64)

The above code has redundancy, the need to do file.read(64) twice.

You could also do the following to avoid redundancy:

while True:
    chunk = file.read(64)
    if not chunk:
        break
    process(chunk)

The above doesn’t communicate the intent very well.

With assignment expression, you could do:

while chunk := file.read(64):
   process(chunk)

Remember, we discussed := being assignment and return. The assignment to chunk happens at the while expression which also makes the data locally available in addition to deciding whether or not to exit the loop. It has no redundancy and communicates the intent very gracefully. That’s pretty awesome or is it not? There has been a lot of discussions regarding this on the internet.

One other example for use case of assignment expression:

# Share a subexpression between a comprehension filter clause and its output
filtered_data = [y for x in data if (y := f(x)) is not None]

PEP 572 has been discussed greatly on various forums such as reddit, hackernews. Here are a few such threads that are actually interesting to go through and bird a variety of viewpoints and opinions.
https://www.reddit.com/r/Python/comments/8ylelb/feedback_on_draft_post_to_pythonideas/
https://news.ycombinator.com/item?id=17448439

Idiomatic Python – Looping Approaches

- - Python, Tutorials, Web

Python has it’s own unique techniques and guidelines for looping. Through this article, I will present a few examples on bad and better approaches on looping. While the end goal can be achieved using both sets of the codes to follow, the purpose is to highlight on the better approaches and encourage it.

Looping over a range of numbers:

>>> for i in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]:
...     print i ** 2
...

Better approach when using Python3

>>> for i in range(10):
...     print i ** 2
...

Better approach when using Python2

>>> for i in xrange(10):
...     print i ** 2
...

Prior to Python3, range was implemented as a function that returns a list instance. This would mean that a list would be created before looping over it. This brought frictions in the community due to the memory and running complexities that came with it. In Python3, range is implemented with lazy evaluation in that it evaluates the item in the sequence as it furthers on the loop. This eliminates the memory and computation issues of the Python2 range function. It is not that this approach was never present. Python2’s xrange is equivalent to Python3 range.

Looping over a collection:

>>> fruits = ["apple", "mango", "grapes", "banana"]
>>> for i in range(len(fruits)):
...     print(fruits[i])
...

Better Approach:

>>> for fruit in fruits:
...     print(fruit)
...

In python list is an iterable and hence implements the __iter__ method that returns an iterator. An iterable is an object that has an __iter__ method which returns an iterator, or which defines a __getitem__ method that can take sequential indexes starting from zero (and raises an IndexError when the indexes are no longer valid). So an iterable is an object that you can get an iterator from.
An iterator is an object with a next (Python 2) or __next__ (Python 3) method.

Looping Backwards:

>>> fruits = ["apple", "mango", "grapes", "banana"]
>>> for i in range(len(fruits) - 1, -1, -1):
...     print(fruits[i])
...

Better Approach:

>>> for fruit in reversed(fruits):
...     print(fruit)
...

Looping over two collections:

>>> names = ["Hussein", "Mohammed", "Osama"]
>>> fruits = ["Apple", "Mango", "Banana"]
>>>
>>> n = min(len(names), len(fruits))
>>> for i in range(n):
...     print(names[i], " => ", fruits[i])
...

Good Approach:

>>> for name, fruit in zip(names, fruits):
...     print(name, " => ", fruit)
...

Better Approach:

>>> from itertools import izip
>>> for name, fruit in izip(names, fruits):
...     print(name, " => ", fruit)
...

zip provides a list while izip provides an iterator instead of a list, hence the name izip (I for iterator). You can argue scenarios where zip would be appropriate but not for the above example.

Idiomatic Python – Use of Falsy and Truthy Concepts

- - Python, Tutorials

Out of many, one reason for python’s popularity is the readability. Python has code style guidelines and idioms and these allow future readers of the code to comprehend to the intentions of it. It is highly important that the code is readable and concise. One such important tip is to use falsy and truthy concepts.

It should be at our best interest to avoid direct comparison to True, False, None. As such we should be well known about truthy and falsy concepts.

Truthy refers to the values that shall always be considered true. Similarly falsy refers to the values that shall always be considered false.

An empty sequence such as an empty list [], empty dictionaries, 0 for numeric, None are considered false values or falsy. Almost anything excluding the earlier mentioned are considered truthy.

An example of a code snippet that is considered bad:

i = 0
if i == 0:
    foo()
    # do something
else:
    bar()
    # do something else
x = True
if x == True:
    foo()
    # do something
else:
    bar()
    # do something else
numbers_list = [1, 2, 3, 4]
if len(numbers_list) > 0:
    foo()
    # do something

An example of code snippet that is considered relatively wise:

i = 0
if not i:
    foo()
    # do something
else:
    bar()
    # do something else
x = True
if x:
    foo()
    # do something
else:
    bar()
    # do something else
numbers_list = [1, 2, 3, 4]
if numbers_list:
    foo()
    # do something

Copying mutable objects in Python

- - Python, Tutorials

An assignment statement in python does not create copies of objects. It binds the name to the object. While working with mutable objects and/or collections of mutable objects, it creates inconsistencies and hence it would be of interest to us to have ways to make real copies of the objects. Essentially, we would require copies such that modifying it would not modify the original object. An example for what happens when we use assignment statements to make copies of mutable objects.

 

>>> fruits = ["apple", "mango", "orange"]
>>> fruits_copy = fruits
>>> id(fruits)
140684382177688
>>> id(fruits_copy)
140684382177688
>>> fruits_copy.append("grapes")
>>> fruits_copy
['apple', 'mango', 'orange', 'grapes']
>>> fruits
['apple', 'mango', 'orange', 'grapes']
>>>

 

The above example demonstrates how using an assignment statement for copying a mutable object has an impact on the original object when the later one is modified.

 

Python’s built-in mutable collections like lists, dicts, and sets can be copied by calling their factory functions on an existing collection.

 

new_list = list(original_list)
new_dict = dict(original_dict)
new_set = set(original_set)

However, this only makes a shallow copy of the objects and goes only a level deep in the recursion tree.

 

A shallow copy refers to construction of a new object followed by populating it with the references to the child objects found in the original object. This implies that a shallow copy is only one level deep. The recursion tree on copying does not proceed further on childs of the object.

 

>>> x = [[1, 2 , 3], [4, 5, 6]]
>>> x_copy = list(x)
>>> id(x) == id(x_copy)
False
>>> x_copy.append([7, 8, 9])
>>> x_copy
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> x
[[1, 2, 3], [4, 5, 6]]

 

In the above example, since x_copy is a new object and the contents upto one level deep are copied to the object, it does not affect the original object. But, remember a shallow copy is only one level deep while all other childs are only referenced and not copied as new ones. Following example should explain it.

 

>>> x = [[1, 2, 3], [4, 5, 6]]
>>> x_copy = list(x)
>>> x_copy[0][0] = "Changed"
>>> x_copy
[['Changed', 2, 3], [4, 5, 6]]
>>> x
[['Changed', 2, 3], [4, 5, 6]]

 

On the flip side, a deep copy is a recursive process. It initially constructs a new object followed by recursively populating it with copies of the child objects found in the original object. Deep copy walks the complete object tree to create a fully independent clone of the original object and all of its childs.

 

>>> import copy
>>> x = [[1, 2, 3], [4, 5, 6]]
>>> x_copy = copy.deepcopy(x)
>>> id(x) == x_copy
False
>>> x_copy
[[1, 2, 3], [4, 5, 6]]
>>> x
[[1, 2, 3], [4, 5, 6]]
>>> x_copy[0][0] = "New Object confirmation"
>>> x_copy
[['New Object confirmation', 2, 3], [4, 5, 6]]
>>> x
[[1, 2, 3], [4, 5, 6]]
>>>

The above example shows a deep copy and hence change in the copied object does not have any effect on the original object.

When the argument default value is mutable

- - Python, Tutorials

Python offers us to specify that a function argument is optional by providing a default value to it. While this is widely used and one of the major features of the language, it can lead to confusions when enough thought is not given to the implementation details of the feature. Especially when the default value is a mutable.

Example of a function implementation with optional argument with default value as a mutable.

>>> def example(numbers=[]):
...   numbers.append(1)
...   return numbers
...
>>>

Simply put, a mutable object can be changed after it is created, and an immutable object can’t. In python list, dictionaries are mutable objects while integers, strings, tuple, etc are immutable objects.

A common misconception is that the optional argument will be set to the default value provided each time the function when called without supplying the value for the optional argument. Let us see.

>>> example()
[1]
>>> example()
[1, 1]
>>> example()
[1, 1, 1]
>>> example()
[1, 1, 1, 1]
>>> example()
[1, 1, 1, 1, 1]
>>>

We would generally assume that each time example() is called without the optional argument, numbers would be assigned to and empty list. Inside the function, 1 would be appended to the list and returned. Well, our hypothesis is false in this case.

This is because, the default value for a function is only evaluated once which is during the function definition. This implies that the default value for the optional parameter numbers is initialized to the default value,  an empty list at the time of the function definition. Any further calls of the function example would then utilize the same list everytime it is called therefore the behavior we saw in the examples above.

How to mess up a Python codebase coming from Java Background.

- - Python, Tutorials

In python, you can make use of the augmented assignment operator to increase or decrease the value of a variable by 1. An augmented assignment is generally used to replace a statement where an operator takes a variable as one of its arguments and then assigns the result back to the same variable. It is very common to make use of augmented assignment operator in python.

Other languages such as JAVA has ++ and operators to perform similar operations. These operators can be used for either pre-increment or post-increment. When you use pre-increment, the value of the expression is taken after performing the increment. When you use post-increment, though, the value of the expression is taken before incrementing, and stored for later use, after the result of incrementing is written back into the variable.

 

Python doesn’t implement this directive for increment/decrement operations. However, what python does implement is the unary operator + and . For an example – when used ahead of a variable acts as a unary negation operator. This also entails that you can use any number of such operators ahead of an expression or variable.

 

Therefore ++variable and variable is perfectly legal in python. However the results aren’t the same for Python and JAVA. Following code blocks should help understand the intentions of the blog.

 

>>> i = 7
>>> - i
-7
>>> + i
7
>>> -- i
7
>>> ++ i
7
>>> --- i
-7
>>> +++ i
7
>>> i ++
  File "", line 1
    i ++
        ^
SyntaxError: invalid syntax
>>> i --
  File "", line 1
    i --
       ^
SyntaxError: invalid syntax
>>>

 

As a JAVA developer unknown of this scenario, you could clearly mess up a python codebase. However the unary operators + and are not valid post variable which totally makes sense.

Flask Essentials – Handling 400 and 404 requests in Flask REST API

- - Python, Tutorials

Flask is a microframework for Python based on Werkzeug, Jinja 2. Flask is simple and straightforward. However less has been discussed in terms of handling non-existent resource request on the Flask API. Since we are talking to and fro using AJAX and what not, it is highly important for the client to consume a proper acknowledgement of some sort in cases of failure. It is this very small implementation that does the work for us however we seem to never bring this up on online content.

Flask provides a mechanism through errorhandler decorator to implement a custom error handling based on the error status code. These errors are generally termed as route errors. Below is a sample implementation of the 400 and 404 errors on the Flask API.

@APP.errorhandler(400)
def not_found(error):
    """
    Gives error message when any bad requests are made.
    Args:
        error (string):
    Returns:
        Error message.
    """
    print error
    return make_response(jsonify({'error': 'Bad request'}), 400)
@APP.errorhandler(404)
def not_found(error):
    """
    Gives error message when any invalid url are requested.
    Args:
        error (string): 
    Returns:
        Error message.
    """
    print error
    return make_response(jsonify({'error': 'Not found'}), 404)

Similarly, you can handle application errors too. However, it is important to note that the application errors arise due to code blocks on the server and hence appropriate handling is essential. I might as well do a follow up on this post for application error handling with appropriate logging.

Appium Python Client

- - Python, Tutorials

Using Appium with Python

Appium is an open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. While the main purpose of appium is to perform automation testing, it can be utilized for variety of other things too. Appium has client libraries in various languages including Python.

Prerequisites:

  • Appium Node Server
  • Appium-Python-Client

The node server process usually listens at port 4723. You can customize the port you want the appium to run on using the -p or –port parameter. There are numerous options that you may utilize as per your need. Another useful parameter is –log that can be used to direct the log to a file.

appium –help

The idea behind the Appium-Python-Client is we create a driver that allows us to perform actions on the scope of the application under automation. We specify the capabilities that we want the driver to have. The driver basically maps the methods to the commands and passes it on to the appium process and waits for a http response. It is the adb commands that is being used under the hood for android. The Appium-Python-Client can be thought of as a wrapper.

Following is a simple example that uses Appium-Python-Client

from appium import webdriver
import time


def make_navigation_driver(apk_path, package_name, initial_activity, platform='Android', platform_version='8.1.0', device_id='emulator-5554', server_api='http://localhost:4723/wd/hub', **kwargs):
    """
    :param apk_path:
    :param package_name:
    :param initial_activity:
    :param platform:
    :param platform_version:
    :param device_id:
    :param server_api:
    :param kwargs: Probably will use for setting custom timeout and other capabilities. Implement later?
    :return:
    """
    desired_caps = {
        'platformName': platform,
        'platformVersion': platform_version,
        'deviceName': device_id,
        # 'app': apk_path, # Saving apk travel time to device since it is already installed on the device.
        'appPackage': package_name,
        'appActivity': initial_activity
    }

    return webdriver.Remote(server_api, desired_caps)


def perform_simple_automation(driver):
    """
    :param driver:
    :return:
    """
    clickables = driver.find_elements_by_android_uiautomator('new UiSelector().clickable(true)')

    # navigate to next activity or view via clicking on the first clickable.
    if len(clickables):
        clickables[0].click()

        
if __name__ == '__main__':
    platform = "Android"
    platform_version = "7.1.1"
    device_id = "367e7aa0"
    apk_path = "/Users/bhishan/Downloads/udemy.apk"
    package_name = "com.udemy.android"
    initial_activity = ".SplashActivity"
    driver = make_navigation_driver(apk_path, package_name, initial_activity, platform=platform,
                                    platform_version=platform_version, device_id=device_id)

perform_simple_automation(driver)

Appium Example Gist