Categories
Python

Python 3.14.0 is Here: Template Strings, True Parallelism, and 30% Speed Boost

Python 3.14.0 dropped on October 7, 2025, and honestly, it’s one of the more interesting releases we’ve had in a while. Not because of any single killer feature, but because of how many long-standing Python pain points it addresses. Template strings finally give us a proper way to handle untrusted input, free-threading is no longer experimental, and there’s a new interpreter implementation that can make your code run significantly faster.

Template Strings: Because F-Strings Weren’t Quite Enough

F-strings were great when they arrived in Python 3.6, but they’ve always had one problem: they evaluate everything immediately, which makes them dangerous for user input. Template strings (t-strings) fix this by creating a template object that you can process however you want.

from string.templatelib import Template

# Looks like an f-string, acts very differently
name = "World"
greeting: Template = t"Hello {name}"

# Here's why this matters
def html_safe(template: Template) -> str:
    """Auto-escape HTML to prevent XSS attacks"""
    result = []
    for item in template:
        if isinstance(item, Interpolation):
            # Automatically escape dangerous characters
            safe_value = str(item.value).replace("<", "&lt;").replace(">", "&gt;")
            result.append(safe_value)
        else:
            result.append(item)
    return "".join(result)

# User input is now safe by default
user_input = "<script>alert('gotcha!')</script>"
safe_output = html_safe(t"<p>User said: {user_input}</p>")
# Output: <p>User said: &lt;script&gt;alert('gotcha!')&lt;/script&gt;</p>

The real power here isn’t just HTML escaping. You can build SQL query builders, shell command constructors, or any domain-specific language that needs to distinguish between literal strings and interpolated values. It’s the kind of feature that seems minor until you need it, then you wonder how you lived without it.

Free-Threading is Finally Real

The Global Interpreter Lock has been the subject of countless Python debates over the years. With 3.14, free-threaded builds are officially supported, not experimental. This means actual parallel execution across multiple cores.

What’s impressive is they’ve gotten the single-threaded performance penalty down to just 5-10%. Earlier experimental builds had much worse overhead, which made them impractical for general use. Now you can reasonably run free-threaded Python in production if you need the parallelism.

A word of caution though: not all C extensions are ready for free-threading yet. The popular ones are being updated, but check your dependencies before jumping in.

The Speed Improvements Are Real

Python 3.14 introduces a new interpreter that uses tail calls between small C functions instead of a giant switch statement. With Clang 19+ on x86-64 or AArch64, you might see up to 30% performance improvements.

Now, before you get too excited, there was some controversy here. Initial benchmarks showed even better numbers, but a compiler bug was discovered that was artificially slowing down the baseline. The real improvement is more like 3-5% in typical cases, with some workloads seeing up to 30%. Still solid, just not the miracle it first appeared to be.

To use this, you need to build Python with specific flags:

./configure --with-tail-call-interp --enable-optimizations

Annotations That Don’t Hurt Performance

One of those “finally!” moments: annotations are now evaluated lazily. This means two things. First, forward references work without quotes:

# Before Python 3.14
class TreeNode:
    def add_child(self, child: "TreeNode") -> None:  # Quotes required
        self.children.append(child)

# Python 3.14
class TreeNode:
    def add_child(self, child: TreeNode) -> None:  # Just works
        self.children.append(child)

Second, and perhaps more importantly, defining annotations no longer has a runtime cost unless you actually introspect them. For large codebases with extensive type hints, this can noticeably improve import times.

The from __future__ import annotations import that many of us have been using is now deprecated, though it won’t be removed until after Python 3.13 reaches end-of-life in 2029. Plenty of time to migrate.

Exception Handling Gets a Minor Cleanup

Small syntax improvement that makes code slightly cleaner:

# Parentheses are now optional for multiple exceptions
try:
    connect_to_server()
except TimeoutError, ConnectionRefusedError:
    print("Network issues detected")

# Also works with except*
try:
    process_batch()
except* ValueError, TypeError, KeyError:
    print("Data validation failed")

Not revolutionary, but it’s one less thing to think about when writing exception handlers.

Actually Useful Developer Tools

The ability to attach a debugger to a running Python process is genuinely useful:

python -m pdb -p 1234  # Attach to process 1234

This uses a new safe debugging interface (PEP 768) that allows external tools to inject code at safe execution points. No more having to restart your application just to debug that one weird issue that only happens in production after running for three days.

For async code, there are new introspection tools:

python -m asyncio pstree 12345  # Show task hierarchy
python -m asyncio ps 12345      # List all tasks

These give you visibility into what your async tasks are actually doing, which anyone who’s debugged a hung asyncio application will appreciate.

Zstandard Compression Built In

Facebook’s Zstandard compression algorithm is now included:

from compression import zstd

data = b"Your data here" * 1000
compressed = zstd.compress(data, level=3)
decompressed = zstd.decompress(compressed)

# Integrated with tarfile and zipfile
import tarfile
with tarfile.open("archive.tar.zst", "w:zst") as tar:
    tar.add("large_file.dat")

Zstandard offers better compression ratios than gzip with similar or better speed. Having it built-in means one less external dependency for applications that need modern compression.

Multiple Interpreters in the Same Process

This is one of those features that most developers won’t use directly, but it enables interesting architectural patterns:

from concurrent.interpreters import Interpreter

with Interpreter() as interp:
    interp.run("""
    # Completely isolated execution environment
    import sys
    print(sys.version)
    """)

Each interpreter is isolated with its own GIL (in non-free-threaded builds), modules, and global state. Think of it as multiprocessing’s isolation with threading’s efficiency. Useful for plugin systems, sandboxing, or any scenario where you need isolated Python execution without separate processes.

Breaking Changes Worth Noting

Multiprocessing on Linux

The default start method changed from fork to forkserver on Linux. This is safer (avoids various threading-related issues) but can break code that relies on fork’s copy-on-write behavior or shared global state.

If your multiprocessing code breaks after upgrading:

import multiprocessing as mp

# Force the old behavior
ctx = mp.get_context('fork')
with ProcessPoolExecutor(mp_context=ctx) as executor:
    # Your code here
    pass

Better yet, fix your code to not rely on fork-specific behavior. The forkserver method is more robust.

Union Type Changes

The old typing.Union and new union syntax (|) now create the same runtime type:

from typing import Union

# These create identical objects now
Union[int, str]
int | str

# This means repr() output changed
print(repr(Union[int, str]))  # Shows: "int | str"

Most code won’t care, but if you’re doing runtime type introspection, be aware that Union[int, str] is Union[int, str] is no longer guaranteed to be True (unions aren’t cached anymore).

Other Improvements Worth Mentioning

The incremental garbage collector is a big deal for applications with large heaps. Instead of stop-the-world collections that could pause your application for hundreds of milliseconds, the new GC spreads the work across multiple smaller increments. If you’ve ever seen GC spikes in your application metrics, this should help considerably.

The REPL now has syntax highlighting by default. After years of staring at monochrome Python prompts, this is surprisingly pleasant. You can disable it if you want, but I’m not sure why you would.

Error messages continue to improve. The interpreter now catches common typos in keywords and offers suggestions. It also provides better messages for unpacking errors and various syntax mistakes. These might seem minor, but they add up to a much better development experience, especially for beginners.

Should You Upgrade?

If you’re starting a new project, absolutely use 3.14. The performance improvements alone make it worthwhile, and the new features are genuinely useful.

For existing projects, the upgrade path is relatively smooth. The multiprocessing change on Linux is the biggest potential issue. Run your test suite, check your multiprocessing code if you’re on Linux, and you should be fine.

The free-threading build is interesting but probably not ready for most production use cases yet. Give it another release or two for the ecosystem to catch up. The JIT compiler is in a similar position—promising but still experimental.

Bottom Line

Python 3.14 feels like a release focused on fixing long-standing issues rather than adding flashy new features. Template strings address the string interpolation security problem. Lazy annotations fix the forward reference annoyance. The incremental GC reduces pause times. Free-threading provides an official path forward for true parallelism.

None of these changes are revolutionary on their own, but together they make Python noticeably better. It’s the kind of release that makes you realize how many small pain points you’d just gotten used to working around.

Check the official Python downloads page for binaries and the full release notes for complete details.


WordPress Tags

Python, Python 3.14, Programming, Software Development, Python Tutorial, Web Development, Performance Optimization, Parallel Programming, Template Strings, Free Threading, Python GIL, Asyncio, Python JIT, Developer Tools, Programming Languages, Software Engineering, Python Updates, Type Hints, Python Performance, Zstandard Compression

SEO Keywords

Primary Keywords:

  • Python 3.14
  • Python 3.14.0 features
  • Python template strings
  • Python free threading
  • Python GIL removal

Secondary Keywords:

  • Python performance improvements
  • Python JIT compiler
  • Python deferred annotations
  • Python asyncio improvements
  • Python multiprocessing forkserver
  • Zstandard compression Python
  • Python remote debugging
  • Python incremental GC
  • Python UUID v6 v7 v8
  • Python 3.14 breaking changes

Long-tail Keywords:

  • What’s new in Python 3.14
  • Python 3.14 template strings tutorial
  • How to use Python free threading
  • Python 3.14 performance benchmarks
  • Migrating to Python 3.14 guide
  • Python 3.14 vs Python 3.13 comparison
  • Python without GIL performance
  • Python tail call interpreter setup
Categories
Django Python

Using the Django Per-Site Cache with the Nginx HTTP Memcached Module

For a long time I thought that the most interesting problems in my field were in scalability. Some people may be more interested in scaling, and others might be more into slick interfaces and fast animations. But for me, scalability has continued to be my passion. For awhile though, it was a unicorn. That unattainable thing that I wanted to work on but couldn’t find anywhere to do it at. That is, until I started work at Future US.

Future is a media company. Originally they started in old media focusing heavily on gaming and tech magazines. Eventually the internet became prominent in everyday life, so more of their old media properties made the transition to the web. The one that really matters to me though is PC Gamer. I’ve been a huge fan of PC Gamer since I was about 7 years old. I still have fond memories getting demo disks in the mail with my subscription.

When I was hired at Future it was to help facilitate the move of PC Gamer from its existing platform (WordPress) to Django. Future had experienced success moving other properties to Django, so it made sense to do it with PC Gamer. When it eventually came time to implement our caching layer, we thought about a lot of different ways that it could be done. Varnish came up as an option, but we decided against it since nobody on the team had experience configuring it (and people elsewhere in the organization had experienced issues with it). Eventually we settled on having Nginx serve pages directly from Memcache. For us, this method works great because PC Gamer doesn’t have a lot of interaction (its almost completely consumption from the user end). Anything that does require back-and-forth between the server is handled via javascript, which makes full page caching super easy to do.

The high level architecture for pc gamer.
The high level architecture for pc gamer.

So how does it all work? The image above describes PC Gamer’s server architecture from a high level. Its pretty basic and works quite well for us. We end up having two types of requests: cache hits & cache misses. The flow for a cache hit is: request -> load balancer -> nginx -> memcache -> your browser. The flow for a cache miss is: request -> load balancer -> nginx -> application server (django) -> (store page in cache) -> your browser.

Since we’re basically running a static site, deciding what content to cache is easy: EVERYTHING!

Cache all the things!
Cache all the things!

Luckily for us Django already has a nice way of doing this: The per-site cache. But it is not without its issues. First of all, the cache keys it creates are insane. We needed something a little simpler for our setup so Nginx could build the cache key of the current request on the fly.

How It Works

The meat and potatoes of overriding Django’s per-site cache key comes in the `_generate_cache_key` function.

def _generate_cache_key(request, method, headerlist, key_prefix):
    if key_prefix is None:
        key_prefix = settings.CACHE_MIDDLEWARE_KEY_PREFIX
    cache_key = key_prefix + get_absolute_uri(request)
    return hashlib.md5(cache_key).hexdigest()

To make things easier for Nginx to understand we just take the url and md5 it. Simple!

On the Nginx side of things, the setup is equally simple.

        set            $combined_string "$host$request_uri";
        set_by_lua     $memcached_key "return ngx.md5(ngx.arg[1])" $combined_string;
 
        # 404 for cache miss
        # 502 for memcached down
        error_page     404 502 504 = @fallback;
 
        memcached_pass {{ cache.private_ip }}:11211;

All this setup does is take the MD5 of the host + request URI and then check to see if that cache key exists in memcache. If it does then we serve the content at that cache key, if it doesn’t we fall back to our Django application servers and they generate the page.

Thats it. Seriously. It’s simple, extremely fast, and works for us. Your mileage may vary, but if you have relatively simple caching requirements I highly suggest looking into this method before looking at something like Varnish. It could help you remove quite a bit of complexity from your setup.

Categories
Django Python Uncategorized

Getting around memory limitations with Django and multi-processing

I’ve spent the last few weeks writing a data migration for a large high traffic website and have had a lot of fun trying to squeeze every bit of processing power out of my machine. While playing around locally I can cluster the migration so it executes on fractions of the queryset. For instance.

./manage.py run_my_migration --cluster=1/10
./manage.py run_my_migration --cluster=2/10
./manage.py run_my_migration --cluster=3/10
./manage.py run_my_migration --cluster=4/10

All this does is take the queryset that is generated in the migration and chop it up into tenths. No big deal. The part that is a big deal is that the queryset contains 30,000 rows. In itself that isn’t a bad thing, but there are a lot of memory and cpu heavy operations that happen on each row. I was finding that when I tried to run the migration on our Rackspace Cloud servers the machine would exhaust its memory and terminate my processes. This was a bit frustrating because presumably the operating system should be able to make use of the swap and just deal with it. I tried to make the clusters smaller, but was still running into issues. Even more frustrating was that this happened at irregular intervals. Sometimes it took 20 minutes and sometimes it took 4 hours.

Threading & Multi-processing

My solution to the problem utilized the clustering ability I already had built into the program. If I could break the migration down into 10,000 small migrations, then I should be able to get around any memory limitations. My plan was as follows:

  1. Break down the migration into 10,000 clusters of roughly 3 rows a piece.
  2. Execute 3 clustered migrations concurrently.
  3. Start the next migration after one has finished.
  4. Log the state of the migration so we know where to start if things go poorly.

One of the issues with doing concurrency work with Python is the global interpreter lock (GIL). It makes writing code a lot easier, but doesn’t allow Python to spawn proper threads. However, its easy to skirt around if you just spawn new processes like I did.

Borrowing some thread pooling code here, I was able to get pretty sweet script running in no time at all.

import sys
import os.path
 
from util import ThreadPool
 
def launch_import(cluster_start, cluster_size, python_path, command_path):
    import subprocess
 
    command = python_path
    command += " " + command_path
    command += "{0}/{1}".format(cluster_start, cluster_size)
 
    # Open completed list.
    completed = []
    with open("clusterlog.txt") as f:
        completed = f.readlines()
 
    # Check to see if we should be running this command.
    if command+"\n" in completed:
        print "lowmem.py ==> Skipping {0}".format(command)
    else:
        print "lowmem.py ==> Executing {0}".format(command)
        proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        output = proc.stdout.read() # Capture the output, don't print it.
 
        # Log completed cluster
        logfile = open('clusterlog.txt', 'a+')
        logfile.write("{0}\n".format(command))
        logfile.close()
 
 
if __name__ == '__main__':
 
    # Simple command line args checking
    try:
        lowmem, clusters, pool_size, python_path, command_path = sys.argv
    except:
        print "Usage: python lowmem.py <clusters> <pool_size> <path/to/python> <path/to/manage.py>"
        sys.exit(1)
 
    # Initiate log file.
    if not os.path.isfile("clusterlog.txt"):
        logfile = open('clusterlog.txt', 'w+')
        logfile.close()
 
    # Build in some extra space.
    print "\n\n"
 
    # Initiate the thread pool
    pool = ThreadPool(int(pool_size))
 
    # Start adding tasks
    for i in range(1, int(clusters)):
        pool.add_task(launch_import, i, clusters, python_path, command_path)
 
    pool.wait_completion()

Utilizing the code above, I can now run a command like:

python lowmem.py 10000 3 /srv/www/project/bin/python "/srv/www/project/src/manage.py import --cluster=" &

Which breaks the queryset up into 10,000 parts and runs the import 3 sets at a time. This has done a great job of keeping the memory footprint of the import low, while still getting some concurrency so it doesn’t take forever.

Categories
Django Javascript Python

Two Frameworks

The past couple of months have found me working diligently on work stuff, but also consistently dropping an hour a day on my current side project.  It just so happens that the side project and my actual work share the same language (Python) and framework (Django).  This has been nice because it’s given my brain a moment to relax with regards to learning new material,  but at the same time I feel stagnate.

Django is my framework of choice.   I know it inside and out, can bend it to my will, and work extremely fast in it.  However I’m not blind to the fact that the popularity of the old monolithic frameworks(Rails, Django, Cake, etc) for new projects is waning.  People these days are starting new projects with a service oriented architecture in mind.  They’re using Node.js with Express on the backend for an API, and then Angular on the front end to create a nice single page app.  I’ve done this sort of development before extensively, but I’m out of practice.  So I’ve come to a fork in the road.

Over the years I’ve come to realize that I can only hold two frameworks in my mind at one time. It doesn’t matter if they are written in different languages or not (those seem to stick with me easier for some reason), but two frameworks is the max I can handle.  So my choices are as follows:  1) Learn Android, 2) Get good at Node.

I’ve made one Android app before when I worked at a marketing firm.  It was fun.  I enjoyed not doing web stuff for once.  I found Java overly verbose,  but as long as you stayed within the “modern Java” lines it was fine.  As for Node, I already know it but I’m just out of practice.  I feel like it would be valuable to become an expert in but sometimes I feel burnt out on the web.

After a lot of deliberation, I think I’m going to move forward with Android development by making an Android app for RedemFit.  It’ll give me a chance to break out of Web development for awhile and hopefully will become something I enjoy doing as much as web.

Categories
AngularJS Python

Using Restangular with Django TastyPie

TastyPie’s JSON responses split the response into two sections: meta data and actual data. The meta is really nice, because it helps you data pagination, result counts, and the like, but it kind of gets in the way of Restangular. Integrating with Restangular is easy though!

Add the following to your app.js file.

yourApp.config(function(RestangularProvider) {
    RestangularProvider.setBaseUrl("/api");
    RestangularProvider.setResponseExtractor(function(response, operation, what, url) {
        var newResponse;
        if (operation === "getList") {
            newResponse = response.objects;
            newResponse.metadata = response.meta;
        } else {
            newResponse = response;
        }
        return newResponse;
    });
    RestangularProvider.setRequestSuffix('/?');
});

Done.

Categories
Python

Virtualenv Error: Could not find the …site.py element of the Setuptools distribution

For a few hours I was running into a problem whenever I would try to install my PIP requirements file. The install would go alright until it got to Distribute, at which point I would get an that ended up in a stack trace with:

Could not find the /lib/python2.7/site-packages/site.py element of the Setuptools distribution

After much searching and digging, the issue was that my virtual environment needed to be instantiated with the –distribute flag.

virtualenv venv --distribute

Problem solved.

Categories
Behavior Driven Development (BDD) Django Lettuce Python Testing

Integration Testing With Django and Lettuce: Getting Started

The Ruby on Rails community has long been a proponent of Behavior Driven Development(BDD) and has a great ecosystem around it supporting that testing methodology. From Cucumber to Capybara, RoR developers have it made when it comes to BDD. But what about Django? What about Python? Django and Python don’t have access to Cucumber or Capybara, but what we do have is a fantastic port of Cucumber called Lettuce.

What is Behavior Driven Development

Before we can get started talking about Lettuce and all the cool things you can do with it, we first need to talk about BDD.

Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software developers and business analysts with shared tools and a shared process to collaborate on software development. (Wikipedia)

BDD arose out of the need for the business side of software and the engineering side of software to communicate more effectively. Prior to BDD, it was a lot more difficult to communicate the business requirements of a project to developers. Sure there were spec documents, but those still needed to be translated into a language the computer can understand. With BDD, tests and acceptance criteria are more accessible to everybody involved. Dan North suggested a few guidelines for BDD, and then the development community took it from there.

  • Tests should be grouped into user stories. Essentially narratives about the expected functionality.
  • Stories should have a title. The title should be clear and explicit.
  • There should be a short narrative at the beginning of the story, that explains who the primary stakeholder of the story is, what effect the story should have, and what business value the stakeholder derives from this from this effect.
  • Scenarios(tests) should follow the format of first describing the initial conditions for the scenario, then which event(s) triggers the start of the scenario, and finally what the expected outcome of the scenario should be.
  • All of these steps should be written out in natural language, preferably using the Gherkin syntax.

An example feature using Gherkin.

Feature: Authentication
    In order to protect private information
    As a registered user
    I want to log in to the admin portal
 
    Scenario: I enter my password correctly
        Given the user "Jack" exists with password "password"
        And I am at "/login/"
        When I fill in "Login" with "Jack"
        And I fill in "Password" with "password"
        And I press "Login"
        Then I should be at "/portal/"
        And I should see "Welcome to the admin portal"

So now that we know the gist of BDD, why would you want to use it? There are probably more reasons than the 3 I’m going to list, but I found these to justify my use of BDD in most cases.

  1. It’s easy for business minded people to understand what you’re trying to test.
  2. It’s easier to translate complicated business requirements into tests.
  3. Some things are easier to explain in natural language.

Alright, now we’re done with the background information. Let’s get rolling on some testing.

Getting Started

To follow the rest of this article, you’re going to need the following:

  • A little Python experience
  • A little Django experience
  • Extremely basic knowledge of regular expressions
  • Knowledge of how to set up a virtual environment using virtualenv (I also use virtualenvwrapper to make my life a bit easier)
  • Firefox – Yes, I know you don’t need Firefox to do this, but its probably the easiest to use with Selenium.

On the bright side, no previous testing experience required!

The best place to start with all this getting the virtual environment set up.

jacks$ mkvirtualenv learning_lettuce

After that, lets get Django installed.

(learning_lettuce)jack:repos jacks$ pip install django

And now we’ll need to create a new Django project.

(learning_lettuce)jack:repos jacks$ django-admin.py startproject learning_lettuce
(learning_lettuce)jack:repos jacks$ cd learning_lettuce/
(learning_lettuce)jack:learning_lettuce jacks$ ls -la
total 8
drwxr-xr-x   4 jacks  staff  136 Jun 19 07:50 .
drwxrwxrwx  28 jacks  staff  952 Jun 19 07:50 ..
drwxr-xr-x   6 jacks  staff  204 Jun 19 07:50 learning_lettuce
-rw-r--r--   1 jacks  staff  259 Jun 19 07:50 manage.py

At this point, I also like to CHMOD manage.py so I can execute it without calling Python directly.

(learning_lettuce)jack:learning_lettuce jacks$ chmod +x manage.py
(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py runserver
Validating models...
 
0 errors found
June 19, 2013 - 06:53:29
Django version 1.5.1, using settings 'learning_lettuce.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

If you can run the server and see the image below, then we can proceed.
Django Powered!

Now that we have Django set up, lets go ahead and create the app we’ll be testing in.

(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py startapp blog

And then go ahead and add the blog app to INSTALLED_APPS in your settings.py file.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
 
    # Authored
    'blog',
)

Also, lets configure our project to use SQLite3.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': 'learning_lettuce.db',
    }
}

Now that we have a Django project and one app set up, its time to take a break and talk about Lettuce.

Lettuce

No, not the vegetable you add to salads. I’m talking about the the BDD testing framework for Python (http://www.lettuce.it). Lettuce is basically a port of a BDD testing framework from the RoR community called Cucumber. The Lettuce website contains extensive documentation and is a great source for learning best practices with it. It’s worth noting however, that at the time of this writing the Lettuce website is undergoing some design changes. They’re incomplete and have made it pretty hard to extract information from the site. Hopefully by the time you need it for reference it’s back to being usable again.

Alright, back to work. Lets install Lettuce, Selenium, and Nose and then freeze a requirements file so we can replicate this environment if we ever need to.

(learning_lettuce)jack:learning_lettuce jacks$ pip install lettuce selenium django-nose
(learning_lettuce)jack:learning_lettuce jacks$ pip freeze > requirements.txt
(learning_lettuce)jack:learning_lettuce jacks$ cat requirements.txt 
Django==1.5.1
django-nose==1.1
fuzzywuzzy==0.2
ipdb==0.7
ipython==0.13.2
lettuce==0.2.18
nose==1.3.0
selenium==2.33.0
sure==1.2.2
wsgiref==0.1.2

You’ll also need to add Lettuce to INSTALLED_APPS in your settings.py file.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
 
    # 3rd party
    'lettuce.django',
 
    # Authored
    'blog',
)

So now that you have Lettuce installed, lets see that it actually works.

(learning_lettuce)jack:learning_lettuce jacks$ ./manage.py harvest
Django's builtin server is running at 0.0.0.0:8000
Oops!
could not find features at ./blog/features

Great, Lettuce worked! It didn’t find any tests to run, but thats ok. At least we’ve verified that we installed everything correctly.

Your First Test

Before you can test anything, you should probably have some content to test on. So let’s quickly wire up a simple view in the blog app.

# blog/views.py
from django.http import HttpResponse
 
def quick_test(request):
	return HttpResponse("Hello testing world!");
# learning_lettuce/urls.py
from django.conf.urls import patterns, include, url
 
from blog.views import quick_test
 
urlpatterns = patterns('',
    url(r'^quick-test/$', quick_test),
)

Great! Now when you go to http://127.0.0.1:8000/quick-test/ you should see “Hello testing world!”.

The next step is to create a folder inside of the blog app called “features”. And inside of that create a file called “test.feature”. It’s worth noting that Lettuce doesn’t actually care what your file is named, so long as the extension is “.feature”. In “test.feature”, add the following:

Feature: Test
	As someone new to testing
	So I can learn behavior driven development
	I want to write some scenarios
 
	Scenario: I can view the test page
		Given I am at "/quick-test/"
		Then I should see "Hello testing world!"

Look at all that plain english! Even without me telling you anything, you can probably figure out what we’re trying to test. But let me break it down for you.

  • Line 1: This loosely describes what all of the scenarios below are testing. Think of it as a way to logically group tests together.
  • Lines 2-4: This is the narrative. It explains why you’re testing in the first place.
  • Line 6: The title of your scenario. This describes what you are specifically testing in this instance.
  • Lines 7-8: These are called “steps”. Steps are how you test your scenario. Each step maps to a method in your code.

Alright, so now that you have your first test written, run it using “./manage.py harvest”. You should see the following:
Learning Lettuce - Unimplemented Steps
Look at all that beautiful output! But what does it all mean?! It’s telling you that Lettuce attempted to run one scenario, and that the two steps within that scenario aren’t implemented yet (remember, each step maps to a method in your code). And because Lettuce is great, it gives you some code to help you implement those two steps.

The Terrain File

Lettuce keeps all of it’s settings and configuration is a file called terrain.py in the root of your Django project. It’s here that we’re going to configure the test database, Firefox, and Selenium. Go ahead and create a terrain.py file in the root of your Django project, and drop the following in it.

from django.core.management import call_command
from django.test.simple import DjangoTestSuiteRunner
 
from lettuce import before, after, world
from logging import getLogger
from selenium import webdriver
 
try:
	from south.management.commands import patch_for_test_db_setup
except:
	pass
 
logger = getLogger(__name__)
logger.info("Loading the terrain file...")
 
@before.runserver
def setup_database(actual_server):
	'''
	This will setup your database, sync it, and run migrations if you are using South.
	It does this before the Test Django server is set up.
	'''
	logger.info("Setting up a test database...")
 
	# Uncomment if you are using South
	# patch_for_test_db_setup()
 
	world.test_runner = DjangoTestSuiteRunner(interactive=False)
	DjangoTestSuiteRunner.setup_test_environment(world.test_runner)
	world.created_db = DjangoTestSuiteRunner.setup_databases(world.test_runner)
 
	call_command('syncdb', interactive=False, verbosity=0)
 
	# Uncomment if you are using South
	# call_command('migrate', interactive=False, verbosity=0)
 
@after.runserver
def teardown_database(actual_server):
	'''
	This will destroy your test database after all of your tests have executed.
	'''
	logger.info("Destroying the test database ...")
 
	DjangoTestSuiteRunner.teardown_databases(world.test_runner, world.created_db)
 
@before.all
def setup_browser():
	world.browser = webdriver.Firefox()
 
@after.all
def teardown_browser(total):
	world.browser.quit()

In your settings.py file, you’re going to need some additions too.

# Nose
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
 
# Lettuce
LETTUCE_SERVER_PORT = 9000

We use the Nose test runner because it’s faster than Django’s default test runner, and we change the server port for running tests so it doesn’t collide with our development server. At this point if you run `./manage.py harvest` again, you’ll still get notices for unimplemented steps, but you’ll also see Firefox open and close real quick. That means we’ve done our job correctly.

Your First Step Definition

Alright, lets make something happen. If you look at the output from the harvest command, you’ll see that it gave you some code to help you implement the new steps that you wrote. Go ahead and copy that code into the bottom of the terrain.py file (and make sure to import ‘step’ at the top). Now, re-run ./manage.py harvest. You should get the following output.
Lettuce - Failing ouput
So why did our steps fail? If you look that the code that was generated for you, there is a line that essentially says “False is equal to some string”. This is obviously not true, so our step fails. So why don’t we make the test pass? We’re going to change a few things:

  • Change the decorator – We want this step to match even if we use other Gherkin keywords like “when”, “and”, and “then”.
  • Change the function name and args – “group1” isn’t very descriptive
  • Write the code – We need this to do something, and right now it doesn’t!
@step(u'I am at "([^"]*)"')
def i_am_at_url(step, url):
    world.browser.get(url)

Now if you run ./manage.py harvest command again your tests will still fail, but this time for a different reason. The reason is the url that we’re passing into the step definition isn’t well formed. We were hoping to be able to pass relative urls in, but we can’t. So go ahead and modify the step in your scenario to look like this.

Given I am at "http://127.0.0.1:9000/quick-test/"

Run ./manage.py harvest again. You’ll see one passing test and one failing test!
Lettuce - Passing and failing

To make the next step pass, we need to make our web page a bit more formal. Go ahead and create a folder called “templates” inside of the “blog” app. Inside that folder, add a file called “base.html” and populate it with:

<html>
	<head>
		<title>Learning Lettuce!</title>
	</head>
	<body id='content'>
		{% block content %}{% endblock %}
	</body>
</html>

Now create a file called “blog.html” inside the same folder. Give it the following content:

{% extends "base.html" %}
 
{% block content %}
Hello testing world!
{% endblock %}

You’ll also need to update the view:

from django.shortcuts import render_to_response
 
def quick_test(request):
	return render_to_response("blog.html", {})

And you’ll need to update your settings file.

## Add this at the top of settings.py
import os.path
root = os.path.dirname(__file__).replace('\\','/')
 
## Make your TEMPLATE_DIRS variable look like this
TEMPLATE_DIRS = (
    root + "/../blog/templates/",
)

Now that our template is more formalized, lets update the step definition in “terrain.py”.

@step(u'I should see "([^"]*)"')
def i_should_see_content(step, content):
	if content not in world.browser.find_element_by_id("content").text:
		raise Exception("Content not found.")

This code explains itself pretty easily. We check to see if the content that is passed in via the step exists inside the body of the page. This has a few drawbacks:

  • What if we don’t want to check the body? What if we want to check a different element?
  • What if the content isn’t visible? (CSS hidden)

Since this is a simple example, we’re going to ignore these issues for now and just run our tests.
Lettuce - Passing tests!
Passing tests!

Next Steps

Now that you have passing steps, you’re well on your way to writing serious integration tests for your code. But there is still a lot more to learn. The next article in this series will cover using Lettuce Webdriver to handle common step definitions, tables, scenario outlines, and much much more.

All code related to this post can be found at https://github.com/vital101/Learning-Lettuce

Categories
Django Python

Lettuce Tags

Lettuce is a BDD (Behavior Driven Development) testing tool for Python based on the excellent Cucumber project. It has most of the same features that Cucumber has, and has proven invaluable in my projects. I discovered an undocumented feature the other day called “Tags”. Cucumber has them, so I also assumed that Lettuce had them. Tags allow you to selectively skip or run scenarios. For instance:

Feature: Some Feature
 
	Scenario: This is scenario 1
		Given I do stuff
		And I see stuff
		Then I am stuff
 
	@mytag
	Scenario: This is scenario 2
		Given I do more stuff
		And I see more stuff
		Then I am more stuff

You can use tags in many ways.

lettuce --tag=mytag # Run only scenarios with this tag
lettuce --tag=-mytag # Don't run scenarios with this tag
./manage.py harvest --tag=mytag # Django/Lettuce way of using tags.