tech.agilitynerd.com

scratching that itch... 
Filed under

python

 

Confidently Refactoring Django URLs, Views, and Templates

Googility.com is my first Django website and under the covers the oldest code looked like it. I had originally written it with the sole intent of allowing people to enter dog agility businesses and websites into a database that I could use to create a Dog Agility Google Custom Search Engine. The primary mistake I made was making the "project" (in Django speak) effectively equivalent to the primary application. In other words I didn't divide the major features of the site into standalone applications (which would allow them to be more easily reused, extended and tested).

As I continued to work on it I learned more about organizing Django projects. When I added the periodical search to the website I created it as a standalone application. I recently split out my django-shrinktheweb application from the main code base.

The Custom Search Engine (CSE) functionality is a worthwhile application that I'm planning on releasing as its own reusable application. I had already created an application directory called "cse" into which I had placed my models, views, urls, and tests specific to the CSE functionality. But I wanted to make the following changes:

  • Move CSE templates into a cse template subdirectory
  • Name the templates to match the views that use them
  • Name the urls in the urls.py prefixed with the application name ("cse_")
  • Covert all reverse() calls in the views and url template tags to use the named urls

Those are enough changes that I was concerned that I might miss something that would fail either in the view code or in rendering of the templates.

The Django test client makes it easy to test the forward and reverse url matching, calling the view and rendering the template. It is kind of a coarse grained test but the changes I was making were perfect for this tool. Given a urls.py:

urlpatterns = patterns('cse.views',
                    url(r'^site/view/(?P<id>\d+)/$', 'view', name='cse_view'),
)

and a view:

def view(request, id, template='cse/view.html'):
    """Display an end user read only view of the site information"""
    site = get_object_or_404(Annotation, pk=id)
    return render_to_response(template,
                          {'site': site,
                           'labels': get_labels_for(site, cap=None),
                           },
                          context_instance=RequestContext(request))

I then wrote a test class to create the required test instances and tests for each url to verify that the url can be found by name (via reverse()), the url maps to a view, the view invokes the desired template(s), and the {% url %} calls within the template can all be resolved:

from django.test import TestCase
from django.test.client import Client
from django.conf import settings
from django.core.urlresolvers import reverse
from cse.models import Label, Annotation

class ViewsTestCase(TestCase):

    def setUp(self):
        self.client = Client()
        self.ROOT_URLCONF = settings.ROOT_URLCONF
        # can provide a custom urls.py for testing so the tests can be run when
        # the application is incorporated into another project
        # settings.ROOT_URLCONF = 'cse.tests.cse_test_urls'
        # override the template context processors if there are special ones in place
        # that either you want to test or want to avoid
        self.TEMPLATE_CONTEXT_PROCESSORS = settings.TEMPLATE_CONTEXT_PROCESSORS
        settings.TEMPLATE_CONTEXT_PROCESSORS = ()
        # Create some instances on which we can invoke views
        self.label = Label(name='name', description='description')
        self.label.save()
        self.annotation = Annotation(comment='Site Name', original_url='http://example.com/')
        self.annotation.save()
        self.annotation.labels.add(self.label)
        self.annotation.save()

    def tearDown(self):
        # put settings back so the next tests aren't effected
        settings.ROOT_URLCONF = self.ROOT_URLCONF
        settings.TEMPLATE_CONTEXT_PROCESSORS = self.TEMPLATE_CONTEXT_PROCESSORS


    def test_view(self):
        response = self.client.get(reverse('cse_view', kwargs={"id":self.annotation.id}))
        self.assertEquals(200, response.status_code)
        self.assertTemplateUsed(response, 'cse/view.html')

The normal unittest asserts are available in the tests. I'm using one of the special asserts provided by the Django test Client to verify that the template I expected was used. All the templates used (due to template inheritance) are collected by the client and can also be verified.

I used these tests in a TDD-ish manner, I wrote the test for a view, ran the tests and kept resolving errors in the templates as I made the changes in my bullet list. It made a tedious job simple and gave me good confidence that I'd found all the renamed urls, views, and templates.

Filed under  //   django   googility   python   tdd   testing  

Comments [0]

Initial Release of django-stw

I have been using the free website thumbnail service from Shrink The Web on my dog agility search website Googility since I launched it. It is quick and easy to use and it adds a lot to the look of the pages.

I had created a simple Django template tag for inserting the little snippet of HTML needed by their service.

Recently they asked me to add support for their advanced features to my template tag. I used this opportunity to convert my templatetag to a Django application. This mostly makes it a lot easier to install but it also let me to bundle tests and an example template with the template tag.

I kept the existing shrinkthewebimage template tag and added a new tag called stwimage to enable the new features.

I'm hosting the example page included in the package here so you can see how the template tags work.

I've hosted the project source on github and uploaded the initial release to the CheeseShop for easy installation.

 

Filed under  //   django   github   googility   pypi   python   shrink the web   web development  

Comments [0]

Embedding JSON Within Generated HTML

Ran into an interesting problem at work this past week that had a simple and pleasing resolution. We have an in house developed JavaScript grid on some of our pages and when users entered some text strings we'd generate invalid JSON payloads that would give the user an error page. If they entered strings that looked like an HTML Entity i.e. &#13 which (with the addition of a trailing ; ) is a non-visible HTML character (carriage return) the text wasn't displayed in the widget. To further complicate things some of the content displayed in the grid is HTML which is inserted into the grid as is and can contain escaped HTML characters.

The grid gets its content as a JSON payload from within a hidden div in the HTML which is generated via a template mechanism. Heres a portion of the template where <%= and %> stringifying of the value of the Python variable(s)/code they surround:

<div style="display:none;" id="grid-init-args-<%= count %>">
    <textarea>
  <!-- this is the JSON payload loaded via the grid JavaScript -->
      <%= 
           [ columnsIndex,
            indexColumns,
            columns,
            rowBuffer,
            contractComponentCount,
            contractId,
            projectId,
            row.contractComponentID,
            row.changeOrderID,
            component.changeOrderType,
            footerRows,
            formulas,
            "false",
            rf.test] %>
    </textarea>
</div>

This approach has a number of problems:

  1. By using the template mechanism to create the JSON payload this template was relying on the similarity of the string representation of Python objects to JSON. After some testing I found the following scenarios: If a string contained a single quote character the string representation was a double quoted string around the text and the single quote; a valid JSON string. If the string contained a double quote character the string representation was a single quoted string around the text and the double quote; an invalid JSON string. If the string contained both a single and a double quote the string representation would be a single quoted string containing a slash escaped single quote and the double quote; an invalid JSON string. Depending on the browser (of course) the JSON string would fail to parse correctly when the double quote was encountered within the single quoted string.
  2. The JSON payload had to be HTML encoded (converting <, >, ", and &) since it was parsed by the browser as HTML.
  3. The HTML encoding would encode or double encode HTML to be inserted directly into the grid's DOM.

The variation in single/double quoting was an easy fix, I changed to simplejson.dumps() which correctly double quotes key/values in dicts and escapes embedded double quotes (single quotes don't need to be escaped). I didn't time it but with the C extension it may be faster than the template engine for our larger datasets.

I played around with (not) encoding various portions of the payload and then it hit me that I should change the grid to get its payload from a non HTML element so that only HTML destined for insertion into the DOM would be HTML encoded (which is as you'd expect for normal HTML handling). I started changing the payload to be stored in JavaScript generated in the template but didn't like the impact the change would have on all the existing templates. So I started Googling and found Ben Nadel's blog post on using script tags as data containers.

So here's my solution:

<div style="display:none;" id="grid-init-args-<%= count %>">
       <script type="application/json">
       <%= simplejson.dumps(
            [ columnsIndex,
             indexColumns,
             columns,
             rowBuffer,
             contractComponentCount,
             contractId,
             projectId,
             row.contractComponentID,
             row.changeOrderID,
             component.changeOrderType,
             footerRows,
             formulas,
             "false",
             rf.test]) %>
        </script>
     </div>
 

There were two changes:

  1. Used simplejson.dumps to correctly double quote and escape double quotes within the variables in the payload.
  2. Change the textarea to a script element.

By converting to a script tag within the hidden div the HTML parser no longer parsed the content of the JSON payload. so the JSON payload only needed to HTML encode HTML elements that were being inserted into the DOM created by the grid.

This change also meant I was able to delete the unnecessary HTML encoding of non-HTML JSON payload data. Got to love solutions that involve deleting code.

Ultimately, we'll convert to loading the JSON payload as a separate AJAX request from the page to the server, but for now this simplifies the markup and handles all types of user input and HTML encoded characters correctly.

Filed under  //   html   javascript   json   python   web development  

Comments [1]

Django Shrink The Web Template Tag Updated

I recently updated my Django template tag for simplifying the use of Shrink The Web images. They recently announced a CDN based distribution of images and they took the opportunity to modify their API.

The updated template tag is on django snippets.

The STW folks have asked be to extend my template tag with support for their PRO features. With luck I'll make that available sometime this weekend.

Filed under  //   django   python   shrink the web   web development  

Comments [0]

Using django-sitemap with django-tagging

I was adding django-sitemap to googility.com yesterday and found that Tags don't implement get_absolute_url(). Which makes sense since the site developer would want to decide how to expose them in the URL space.

It is also arguable that links to pages displaying the tag view already exist in the page for models that are already in the sitemap so they don't need to be put in the sitemap explicitly. For example, a page for an Article might be at /article/django-11-release and that page would contain the links to pages linked with the tags for that article e.g. /tag/django/ and /tag/python/

But I figured having the tag pages indexed by Google would be useful. It also allows a different priority to be specified for the pages. So I made a little class that derives from GenericSitemap that allows the url and suffix for the Tag name to be specified:



class SlugSitemap(GenericSitemap):
    """Use for objects that don't implement
     get_absolute_url but have a slug field used in 
     creating their url"""
    def __init__(self, info_dict, priority=None, changefreq=None):
        GenericSitemap.__init__(self, info_dict, 
                                  priority=priority, 
                                  changefreq=changefreq)
        self.url = info_dict.get('url', '/')
        self.slugfield = info_dict['slugfield']
        self.suffix = info_dict.get('suffix', '')

    def location(self, obj):
        return "%s%s%s" % (self.url, 
                             getattr(obj, self.slugfield), 
                             self.suffix)


Here's how I use it:



sitemaps = {
    'tag_detail': SlugSitemap({'queryset':Tag.objects,
                               'url':'/tag/',
                               'slugfield':'name',
                               'suffix':'/'},
                              changefreq='monthly',
                              priority='0.5'),
}


The urls for tags are at /tag/slugname/ where /tag/ is prepended to tag.name and / is appended to the end

This class can be used to create sitemap entries for any url parameterized on a single field of an instance returned by the QuerySet.

Filed under  //   django   django-sitemap   django-tagging   python   web development  

Comments [0]