tech.agilitynerd.com

Ran into an interesting problem at work this past week that had a simple and pleasing resolution. We have an in house developed JavaScript grid on some of our pages and when users entered some text strings we'd generate invalid JSON payloads that would give the user an error page. If they entered strings that looked like an HTML Entity i.e. &#13 which (with the addition of a trailing ; ) is a non-visible HTML character (carriage return) the text wasn't displayed in the widget. To further complicate things some of the content displayed in the grid is HTML which is inserted into the grid as is and can contain escaped HTML characters.

The grid gets its content as a JSON payload from within a hidden div in the HTML which is generated via a template mechanism. Heres a portion of the template where <%= and %> stringifying of the value of the Python variable(s)/code they surround:

<div style="display:none;" id="grid-init-args-<%= count %>">
    <textarea>
  <!-- this is the JSON payload loaded via the grid JavaScript -->
      <%= 
           [ columnsIndex,
            indexColumns,
            columns,
            rowBuffer,
            contractComponentCount,
            contractId,
            projectId,
            row.contractComponentID,
            row.changeOrderID,
            component.changeOrderType,
            footerRows,
            formulas,
            "false",
            rf.test] %>
    </textarea>
</div>

This approach has a number of problems:

1. By using the template mechanism to create the JSON payload this template was relying on the similarity of the string representation of Python objects to JSON. After some testing I found the following scenarios: If a string contained a single quote character the string representation was a double quoted string around the text and the single quote; a valid JSON string. If the string contained a double quote character the string representation was a single quoted string around the text and the double quote; an invalid JSON string. If the string contained both a single and a double quote the string representation would be a single quoted string containing a slash escaped single quote and the double quote; an invalid JSON string. Depending on the browser (of course) the JSON string would fail to parse correctly when the double quote was encountered within the single quoted string.
2. The JSON payload had to be HTML encoded (converting <, >, ", and &) since it was parsed by the browser as HTML.
3. The HTML encoding would encode or double encode HTML to be inserted directly into the grid's DOM.

The variation in single/double quoting was an easy fix, I changed to simplejson.dumps() which correctly double quotes key/values in dicts and escapes embedded double quotes (single quotes don't need to be escaped). I didn't time it but with the C extension it may be faster than the template engine for our larger datasets.

I played around with (not) encoding various portions of the payload and then it hit me that I should change the grid to get its payload from a non HTML element so that only HTML destined for insertion into the DOM would be HTML encoded (which is as you'd expect for normal HTML handling). I started changing the payload to be stored in JavaScript generated in the template but didn't like the impact the change would have on all the existing templates. So I started Googling and found Ben Nadel's blog post on using script tags as data containers.

So here's my solution:

<div style="display:none;" id="grid-init-args-<%= count %>">
       <script type="application/json">
       <%= simplejson.dumps(
            [ columnsIndex,
             indexColumns,
             columns,
             rowBuffer,
             contractComponentCount,
             contractId,
             projectId,
             row.contractComponentID,
             row.changeOrderID,
             component.changeOrderType,
             footerRows,
             formulas,
             "false",
             rf.test]) %>
        </script>
     </div>
 

There were two changes:

1. Used simplejson.dumps to correctly double quote and escape double quotes within the variables in the payload.
2. Change the textarea to a script element.

By converting to a script tag within the hidden div the HTML parser no longer parsed the content of the JSON payload. so the JSON payload only needed to HTML encode HTML elements that were being inserted into the DOM created by the grid.

This change also meant I was able to delete the unnecessary HTML encoding of non-HTML JSON payload data. Got to love solutions that involve deleting code.

Ultimately, we'll convert to loading the JSON payload as a separate AJAX request from the page to the server, but for now this simplifies the markup and handles all types of user input and HTML encoded characters correctly.

Loading mentions Retweet

I decided to stop displaying the default YouTube video players within posts on my AgilityNerd blog and I started looking for a light boxed player. Their were two main reasons. The smallest video playback window provided by YouTube for HD videos is too wide for my two column layout and now that I'm posting more videos the load time of the page is delayed by the communication with all the off site webservers; serving the YouTube static image of the video will be much faster/lighter weight.

I looked around and really liked the lightbox containing the default YouTube player provided by VideoLightBox and started playing around with their demo. VideoLightBox (VLB) has an interesting approach. You download an application (PC or Mac), configure how you want your video(s) to look and it generates a directory of files on your local disk (or uploads files to your website via FTP) along with an index.html file from which you copy the code to put in the <head> and <body> of your web page. For YouTube it also downloads a static image for each selected video which is used as the image link within the HTML page. Straight forward and works well.

For my purposes there was a problem with their approach, its locates the image used to launch the light box using an element id. This assumes a single video or gallery of videos per web page. On my blog's main page or the category pages there will be multiple videos (possibly multiple videos within a single post). I figured a little bit of CSS and JQuery hacking would solve the problem and it did.

I decided to modify their HTML/CSS/JS to use a CSS class instead of an element id to allow for multiple videos per page. At first I just modified the generated files. Then I saw that VLB has template files in their installation. So I started modifying the templates to output the new code. Two hours later I bailed. Using procmon it looks like the client app reads the template files but then doesn't actually use the files to generate the output files(?). I was only able to modify one of the three template files that needed to change and have it effect the generated files.

I'm going to provide my edits to the VLB developers in case they are interested.

So the solution is to edit one of the template files and then edit two of the generated files; not ideal but once you put the generated files on your webserver you'll probably not touch them unless you are changing CSS styles. The modifications aren't hard but you need to be careful and typos will definitely break things. You should backup the VLB directory before your start or be prepared to uninstall and reinstall from their installation program.

1. Navigate to the VideoLightBox directory (i.e. C:\Program Files (x86)\VideoLightBox)
2. Change the permissions on the templates subdirectory to give your user full access to overwrite the files
3. For each directory in the templates subdirectory open the videolightbox.js file in an programming editor (a keyboard macro makes this trivial):
   
   1. Globally replace $("#videogallery a[rel]") with
      $(".videogallery a[rel]").each(function(idx){$(this)
   2. Go to the end of the line and add });
   3. Save the file

Then generate the output files using the VLB executable for one or more videos, saving the results to your local file system

1. Navigate to the output directory
2. Open the index.html file in an programming editor
   1. Globally replace #videogallery with .videogallery
   2. Globally replace id="videogallery" with class="videogallery"
   3. Save the file
3. In the engine/css subdirectory open the videolightbox.css file in an programming editor
   
   1. Globally replace #videogallery with .videogallery
   2. Save the file

Then you can copy the files just as specified by the VLB installation instructions.

The other change I'll be making for my deployment is to rename the video images. They are named 0.png, 1.png, etc. I'm going to put them all in a directory on my resource webserver so I'll rename the files and their references in the code copied from the index.html to use the YouTube video id.

I'll be changing my existing web posts over to this new scheme over time...

Loading mentions Retweet