Mastering Objectoriented Python

Table of Contents

Mastering Object-oriented Python

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code for this book

Errata

Piracy

Questions

Some Preliminaries

About casino Blackjack

Playing the game

Blackjack player strategies

Object design for simulating Blackjack

Performance – the timeit module

Testing – unittest and doctest

Unit testing and technology spikes

Docstrings – RST markup and documentation tools

The IDE question

About special method names

Summary

1. Pythonic Classes via Special Methods

Pythonic Classes via Special Methods

1. The __init__() Method

The implicit superclass – object

The base class object __init__() method

Implementing __init__() in a superclass

Using __init__() to create manifest constants

Leveraging __init__() via a factory function

Faulty factory design and the vague else clause

Simplicity and consistency using elif sequences

Simplicity using mapping and class objects

Two parallel mappings

Mapping to a tuple of values

The partial function solution

Fluent APIs for factories

Implementing __init__() in each subclass

Simple composite objects

Wrapping a collection class

Extending a collection class

More requirements and another design

Complex composite objects

Complete composite object initialization

Stateless objects without __init__()

Some additional class definitions

Multi-strategy __init__()

More complex initialization alternatives

Initializing static methods

Yet more __init__() techniques

Initialization with type validation

Initialization, encapsulation, and privacy

Summary

2. Integrating Seamlessly with Python Basic Special Methods

The __repr__() and __str__() methods

Non collection __str__() and __repr__()

Collection __str__() and __repr__()

The __format__() method

Nested formatting specifications

Collections and delegating format specifications

The __hash__() method

Deciding what to hash

Inheriting definitions for immutable objects

Overriding definitions for immutable objects

Overriding definitions for mutable objects

Making a frozen hand from a mutable hand

The __bool__() method

The __bytes__() method

The comparison operator methods

Designing comparisons

Implementation of comparison for objects of the same class

Implementation of comparison for objects of mixed classes

Hard totals, soft totals, and polymorphism

A mixed class comparison example

The __del__() method

The reference count and destruction

Circular references and garbage collection

Circular references and the weakref module

The __del__() and close() methods

The __new__() method and immutable objects

The __new__() method and metaclasses

Metaclass example 1 – ordered attributes

Metaclass example 2 – self-reference

Summary

3. Attribute Access, Properties, and Descriptors

Basic attribute processing

Attributes and the __init__() method

Creating properties

Eagerly computed properties

Setter and deleter properties

Using special methods for attribute access

Creating immutable objects with __slots__

Creating immutable objects as a tuple subclass

Eagerly computed attributes

The __getattribute__() method

Creating descriptors

Using a nondata descriptor

Using a data descriptor

Summary, design considerations, and trade-offs

Properties versus attributes

Designing with descriptors

Looking forward

4. The ABCs of Consistent Design

Abstract base classes

Base classes and polymorphism

Callables

Containers and collections

Numbers

Some additional abstractions

The iterator abstraction

Contexts and context managers

The abc module

Summary, design considerations, and trade-offs

Looking forward

5. Using Callables and Contexts

Designing with ABC callables

Improving performance

Using memoization or caching

Using functools for memoization

Aiming for simplicity using the callable API

Complexities and the callable API

Managing contexts and the with statement

Using the decimal context

Other contexts

Defining the __enter__() and __exit__() methods

Handling exceptions

Context manager as a factory

Cleaning up in a context manager

Summary

Callable design considerations and trade-offs

Context manager design considerations and trade-offs

Looking forward

6. Creating Containers and Collections

ABCs of collections

Examples of special methods

Using the standard library extensions

The namedtuple() function

The deque class

The ChainMap use case

The OrderedDict collection

The defaultdict subclass

The counter collection

Creating new kinds of collections

Defining a new kind of sequence

A statistical list

Choosing eager versus lazy calculation

Working with __getitem__(), __setitem__(), __delitem__(), and slices

Implementing __getitem__(), __setitem__(), and __delitem__()

Wrapping a list and delegating

Creating iterators with __iter__()

Creating a new kind of mapping

Creating a new kind of set

Some design rationale

Defining the Tree class

Defining the TreeNode class

Demonstrating the binary tree set

Summary

Design considerations and Trade-offs

Looking forward

7. Creating Numbers

ABCs of numbers

Deciding which types to use

The method resolution and the reflected operator concept

The arithmetic operator's special methods

Creating a numeric class

Defining FixedPoint initialization

Defining FixedPoint binary arithmetic operators

Defining FixedPoint unary arithmetic operators

Implementing FixedPoint reflected operators

Implementing FixedPoint comparison operators

Computing a numeric hash

Designing more useful rounding

Implementing other special methods

Optimization with the in-place operators

Summary

Design considerations and trade-offs

Looking forward

8. Decorators and Mixins – Cross-cutting Aspects

Class and meaning

Constructing the functions

Constructing the class

Some class design principles

Aspect-oriented programming

Using built-in decorators

Using standard library decorators

Using standard library mixin classes

Using the context manager mixin class

Turning off a class feature

Writing a simple function decorator

Creating separate loggers

Parameterizing a decorator

Creating a method function decorator

Creating a class decorator

Adding method functions to a class

Using decorators for security

Summary

Design considerations and trade-offs

Looking forward

2. Persistence and Serialization

Persistence and Serialization

9. Serializing and Saving – JSON, YAML, Pickle, CSV, and XML

Understanding persistence, class, state, and representation

Common Python terminologies

Filesystem and network considerations

Defining classes to support persistence

Rendering a blog and posts

Dumping and loading with JSON

Supporting JSON in our classes

Customizing JSON encoding

Customizing JSON decoding

The security and the eval() issue

Refactoring the encode function

Standardizing the date string

Writing JSON to a file

Dumping and loading with YAML

Formatting YAML data on a file

Extending the YAML representation

Security and safe loading

Dumping and loading with pickle

Designing a class for reliable pickle processing

Security and the global issue

Dumping and loading with CSV

Dumping simple sequences to CSV

Loading simple sequences from CSV

Handling containers and complex classes

Dumping and loading multiple row types in a CSV file

Filtering CSV rows with an iterator

Dumping and loading joined rows in a CSV file

Dumping and loading with XML

Dumping objects using string templates

Dumping objects with xml.etree.ElementTree

Loading XML documents

Summary

Design considerations and trade-offs

Schema evolution

Looking forward

10. Storing and Retrieving Objects via Shelve

Analyzing persistent object use cases

The ACID properties

Creating a shelf

Designing shelvable objects

Designing keys for our objects

Generating surrogate keys for objects

Designing a class with a simple key

Designing classes for containers or collections

Referring to objects via foreign keys

Designing CRUD operations for complex objects

Searching, scanning, and querying

Designing an access layer for shelve

Writing a demonstration script

Creating indexes to improve efficiency

Creating top-level indices

Adding yet more index maintenance

The writeback alternative to index updates

Schema evolution

Summary

Design considerations and trade-offs

Application software layers

Looking forward

11. Storing and Retrieving Objects via SQLite

SQL databases, persistence, and objects

The SQL data model – rows and tables

CRUD processing via SQL DML statements

Querying rows with the SQL SELECT statement

SQL transactions and the ACID properties

Designing primary and foreign database keys

Processing application data with SQL

Implementing class-like processing in pure SQL

Mapping Python objects to SQLite BLOB columns

Mapping Python objects to database rows manually

Designing an access layer for SQLite

Implementing container relationships

Improving performance with indices

Adding an ORM layer

Designing ORM-friendly classes

Building the schema with the ORM layer

Manipulating objects with the ORM layer

Querying post objects given a tag string

Improving performance with indices

Schema evolution

Summary

Design considerations and trade-offs

Mapping alternatives

Keys and key designs

Application software layers

Looking forward

12. Transmitting and Sharing Objects

Class, state, and representation

Using HTTP and REST to transmit objects

Implementing CRUD operations via REST

Implementing non-CRUD operations

The REST protocol and ACID

Choosing a representation – JSON, XML, or YAML

Implementing a REST server – WSGI and mod_wsgi

Creating a simple REST application and server

Implementing a REST client

Demonstrating and unit testing the RESTful services

Using Callable classes for WSGI applications

Designing RESTful object identifiers

Multiple layers of REST services

Creating the roulette server

Creating the roulette client

Creating a secure REST service

The WSGI Authentication application

Implementing REST with a web application framework

Using a message queue to transmit objects

Defining processes

Building queues and supplying data

Summary

Design considerations and trade-offs

Schema evolution

Application software layers

Looking forward

13. Configuration Files and Persistence

Configuration file use cases

Representation, persistence, state, and usability

Application configuration design patterns

Configuring via object construction

Implementing a configuration hierarchy

Storing the configuration in the INI files

Handling more literals via the eval() variants

Storing the configuration in PY files

Configuration via class definitions

Configuration via SimpleNamespace

Using Python with exec() for the configuration

Why is exec() a nonproblem?

Using ChainMap for defaults and overrides

Storing the configuration in JSON or YAML files

Using flattened JSON configurations

Loading a YAML configuration

Storing the configuration in property files

Parsing a properties file

Using a properties file

Storing the configuration in XML files – PLIST and others

Customized XML configuration files

Summary

Design considerations and trade-offs

Creating a shared configuration

Schema evolution

Looking Forward

3. Testing, Debugging, Deploying, and Maintaining

Testing, Debugging, Deploying, and Maintaining

14. The Logging and Warning Modules

Creating a basic log

Creating a shared class-level logger

Configuring the loggers

Starting up and shutting down the logging system

Naming the loggers

Extending the logger levels

Defining handlers for multiple destinations

Managing the propagation rules

Configuration gotcha

Specializing logging for control, debug, audit, and security

Creating a debugging log

Creating audit and security logs

Using the warnings module

Showing API changes with a warning

Showing configuration problems with a warning

Showing possible software problems with a warning

Advanced logging – the last few messages and network destinations

Building an automatic tail buffer

Sending logging messages to a remote process

Preventing queue overrun

Summary

Design considerations and trade-offs

Looking forward

15. Designing for Testability

Defining and isolating units for testing

Minimizing the dependencies

Creating simple unit tests

Creating a test suite

Including edge and corner cases

Mocking dependencies for testing

Using more mocks to test more behaviors

Using doctest to define test cases

Combining doctest and unittest

Creating a more complete test package

Using setup and teardown

Using setup and teardown with OS resources

Using setup and teardown with databases

The TestCase class hierarchy

Using externally defined expected results

Automated integration or performance testing

Summary

Design considerations and trade-offs

Looking forward

16. Coping With the Command Line

The OS interface and the command line

Arguments and options

Parsing the command line with argparse

A simple on/off option

An option with an argument

Positional arguments

All other arguments

--version display and exit

--help display and exit

Integrating command-line options and environment variables

Providing more configurable defaults

Overriding configuration file settings with environment variables

Overriding environment variables with the configuration files

Making the configuration aware of the None values

Customizing the help output

Creating a top-level main() function

Ensuring DRY for the configuration

Managing nested configuration contexts

Programming In The Large

Designing command classes

Adding the analysis command subclass

Adding and packaging more features into an application

Designing a higher-level composite command

Additional composite command design patterns

Integrating with other applications

Summary

Design considerations and trade-offs

Looking forward

17. The Module and Package Design

Designing a module

Some module design patterns

Module versus class

The expected content of a module

Whole module versus module items

Designing a package

Designing a module-package hybrid

Designing a package with alternate implementations

Designing a main script and the __main__ module

Creating an executable script file

Creating a __main__ module

Programming in the large

Designing long-running applications

Organizing code into src, bin, and test

Installing Python modules

Summary

Design considerations and trade-offs

Looking forward

18. Quality and Documentation

Writing docstrings for the help() function

Using pydoc for documentation

Better output via the RST markup

Blocks of text

The RST inline markup

RST directives

Learning RST

Writing effective docstrings

Writing file-level docstrings, including modules and packages

Writing API details in RST markup

Writing class and method function docstrings

Writing function docstrings

More sophisticated markup techniques

Using Sphinx to produce the documentation

Using the Sphinx quickstart

Writing the Sphinx documentation

Filling in the 4+1 views for documentation

Writing the implementation document

Creating the Sphinx cross-references

Refactoring Sphinx files into directories

Writing the documentation

Literate programming

Use cases for literate programming

Working with a literate programming tool

Summary

Design considerations and trade-offs

Index

Mastering Object-oriented Python

Mastering Object-oriented Python

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: April 2014

Production Reference: 1150414

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK

ISBN 978-1-78328-097-1

www.packtpub.com

Cover Image by Duraid Fatouhi (<duraidfatouhi@yahoo.com>)

Credits

Author

Steven F. Lott

Reviewers

Mike Driscoll

Róman Joost

Sakis Kasampalis

Albert Lukaszewski, Ph.D

Hugo Solis

Commissioning Editor

Usha Iyer

Acquisition Editor

Gregory Wild

Content Development Editor

Shaon Basu

Technical Editors

Kapil Hemnani

Monica John

Akashdeep Kundu

Copy Editors

Insiya Morbiwala

Kirti Pai

Stuti Srivastava

Project Coordinator

Akash Poojary

Proofreaders

Stephen Copestake

Clyde Jenkins

Linda Morris

Jonathan Todd

Indexer

Mariammal Chettiyar

Graphics

Abhinash Sahu

Production Coordinator

Alwin Roy

Cover Work

Alwin Roy

About the Author

Steven F. Lott has been programming since the 70s, when computers were large, expensive, and rare. As a contract software developer and architect, he has worked on hundreds of projects from very small to very large. He's been using Python to solve business problems for over 10 years.

Steven is currently a technomad who lives in various places on the east coast of the US. His technology blog is: http://slott-softwarearchitect.blogspot.com

I owe deep gratitude to Floating Leaf for all her support and guidance.

About the Reviewers

Mike Driscoll has been programming in Python since 2006. He enjoys writing about Python on his blog at http://www.blog.pythonlibrary.org/. He has co-authored Core Python refcard for DZone. Mike has also been a technical reviewer for various books of Packt Publishing, such as Python 3 Object Oriented Programming, Python 2.6 Graphics Cookbook, and Tkinter GUI Application Development Hotshot. Mike recently wrote the book Python 101.

I would like to thank my beautiful wife, Evangeline, for always supporting me. I would also like to thank my friends and family for all that they do to help me. I would also like to thank Jesus Christ for saving me.

Róman Joost first learned about open source software in 1997. He is the project manager of GIMP's user documentation. He has contributed to GIMP and Python/Zope open source projects for eight years. Róman works for Red Hat in Brisbane, Australia.

Sakis Kasampalis is based in the Netherlands, where he currently works as a Software Engineer for a location-based B2B provider. He is not dogmatic about particular programming languages and tools; his principle is that the right tool should be used for the right job. One of his favorite tools is Python because he finds it very productive.

Among the FOSS activities of Kasampalis is maintaining a GitHub repository that is related to implementing design patterns in Python, which are available at https://github.com/faif/python-patterns. He was also a technical reviewer of the book Learning Python Design Patterns, Packt Publishing.

Albert Lukaszewski, Ph.D, is principal consultant for Lukaszewski Consulting Services in southeast Scotland. Having programmed computers for over 30 years, he consults on the system design and implementation. Previously, he served as Chief Engineer for ACCESS Europe GmbH. Much of his experience is related to text processing, database systems, and Natural Language Processing (NLP). In addition to MySQL for Python, Packt Publishing, he previously wrote a column on Python for the New York Times subsidiary, About.com.

Hugo Solis is an assistant professor in the Physics department at the University of Costa Rica. His current research interests are computational cosmology, complexity, and the influence of hydrogen on material properties. He has wide experience with languages including C/C++ and Python for scientific programming and visualization. He is a member of the Free Software Foundation and has contributed code to some free software projects. Currently, he is in charge of the IFT, a Costa Rican scientific, non-profit organization for the multidisciplinary practice of physics (http://iftucr.org).

I'd like to thank Katty Sanchez, my beloved mother, for her support and vanguard thoughts.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.