Running with Code Like with scissors, only more dangerous

31Mar/091

My Job is Tools

Posted by Rob

For the last few months I’ve been working on a fairly large-sized web application for work.  The application is mostly Flex-based, so I’m working on back-end components, which are accessed via FluorineFx (an ActionScript-.NET connector).  The site also needs to integrate with Ektron, a somewhat perilous enterprise-level CMS, because a lot of the company’s web-facing data is already stored their and they want to reuse as much as possible.  The site is a completely new version of a much less-ambitious site designed for the same purpose, driven by a 400mb Access database.

So, as part of our deployment setup, our projects consist of:

  • 3 projects that encompass most of the site’s actual functionality.  One is for Ektron integration, one is for control development, and one is for the service implementations.
  • 1 tool that imports data from the old site.
  • 1 tool that imports localization data from Ektron.
  • 1 tool that creates Ektron Smart Forms in order to create that localization data.
  • 1 tool that encrypts member passwords (because until that’s done we can’t see the old members database).
  • 1 tool that both backs up Ektron staging data and also restores that data to a clean environment.
  • A team test project.
  • A test harness for other parts of the system.
  • Oh yeah, the web site.

I generally find myself on this approach whenever I’m working on a large system.  Importing data from SMF?  Write a tool.  Need to transform something into an XML format your site understands?  Write a tool. 

For me, the hardest part about getting tooling correct is making sure that I’m not wasting time when I’m designing one.  For instance, smart forms in Ektron are a cool idea, at least in concept.  I’ve found them somewhat difficult to use, but it’s still a cool idea for a CMS.  Our client wants to use Ektron to store localization data, but right now all I have are the dictionaries of English text in a Flex ActionScript file.  Here’s our process:

  • Grep the text out of the ActionScript dictionaries into a .CSV file.
  • Using Sebastien Lorien’s CSV reader, read each dictionary in one at a time.
  • Create a .xml file corresponding to the Ektron smart form exported file format.
  • Manually import or update each corresponding Ektron smart form.

When this eventually goes to the client (or to a new staging database for ourselves), we’ll run a tool on our database in “backup mode,” which will back up all of the smart forms, as well as the corresponding content rows, into a binary file.  Then we’ll run the tool on the new database in “restore mode,” which will update all of the content to the latest versions.

What does this allow us to do?

  • QA content on the test site and be confident that it’s going to be represented on the production site.  Because entering content into Ektron in this manner isn’t particularly easy, we don’t feel comfortable that we could do it with no errors.  By automating the process, we can be reasonably confident that we’ll eliminate typos.
  • Avoid repeated input time.  It might have cost me the same amount of time to write the backup/restore tool as it would have taken to just do it by hand.  However, I was fairly confident (and correct) that I’d have to do it more than once.  We get the added bonus of being able to deliver the tool to the client, who will also have to run it in their production environment.

I love writing tools. It makes everyone’s lives easier, and that’s never, ever bad.

Tagged as: , 1 Comment
17Mar/090

Invoking a Partial Update from Flash

Posted by Rob

This morning I received an email that posed a question so interesting that I thought I would blog about the answer.  The question was, essentially, how can we invoke a partial update (using ASP.NET AJAX triggers), from an on(up) button handler in Flash?

There are a few different ways to approach this problem.  I believe the method I’m going to write out here is what I like to call the “path of least resistance” – it’ll get you there quickly.  However, it will create some interdependencies among your controls.  However, using this technique as a baseline, you should be able to adapt it to fit other better techniques, which I’ll describe later.

Flash lives in a sandbox relative to your web page; it doesn’t interact with the rest of your page’s code (by default), and so you need to provide it with a way to do so.  In ActionScript 1 and 2 it’s relatively easy to invoke JavaScript using getURL():

Invoking a JavaScript call

I won’t get into the details of how to access script with ActionScript; I’ll simply leave it to the professionals (and use this as an example).  With this capability we should have everything we need to build an AJAX-enabled Flash button.  I’ve created a sample ASPX page that will host what we need:

<%@ Page Language="C#" AutoEventWireup="true"  CodeFile="Default.aspx.cs" Inherits="_Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" width="150" height="60" id="blog" align="middle">
            <param name="allowScriptAccess" value="always" />
            <param name="allowFullScreen" value="false" />
            <param name="movie" value="blog.swf" /><param name="quality" value="high" /><param name="bgcolor" value="#ffffff" />    
            <embed src="blog.swf" quality="high" bgcolor="#ffffff" width="150" height="60" name="blog" align="middle" allowScriptAccess="always" allowFullScreen="false" type="application/x-shockwave-flash" pluginspage="http://www.macromedia.com/go/getflashplayer" />
        </object>
    <asp:ScriptManager ID="sm" runat="server" EnablePartialRendering="true">
        
    </asp:ScriptManager>
    
    <asp:UpdatePanel ID="update" runat="server">
        <Triggers>
            
        </Triggers>
        <ContentTemplate>
            <asp:Label ID="lblText" runat="server" Text="Click the Flash button to turn me blue." />
        </ContentTemplate>
    </asp:UpdatePanel>
    </div>
    </form>
</body>
</html>

We still need a way to invoke the partial update.  Unfortunately, what I termed the “path of least resistance” is going to involve a little voodoo: we’re going to create a Button, set its display to none (so that it’s on the page but hidden), and then treat it as a trigger for the UpdatePanel:

        <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" width="150" height="60" id="blog" align="middle">
            <param name="allowScriptAccess" value="always" />
            <param name="allowFullScreen" value="false" />
            <param name="movie" value="blog.swf" /><param name="quality" value="high" /><param name="bgcolor" value="#ffffff" />    
            <embed src="blog.swf" quality="high" bgcolor="#ffffff" width="150" height="60" name="blog" align="middle" allowScriptAccess="sameDomain" allowFullScreen="false" type="application/x-shockwave-flash" pluginspage="http://www.macromedia.com/go/getflashplayer" />
        </object>
        <asp:Button runat="server" style="display: none;" ID="btnFlashGo" />
    <asp:ScriptManager ID="sm" runat="server" EnablePartialRendering="true">
        
    </asp:ScriptManager>
    
    <asp:UpdatePanel ID="update" runat="server">
        <Triggers>
            <asp:AsyncPostBackTrigger ControlID="btnFlashGo" EventName="Click" />
        </Triggers>
        <ContentTemplate>
            <asp:Label ID="lblText" runat="server" Text="Click the Flash button to turn me blue." />
        </ContentTemplate>
    </asp:UpdatePanel>

We’re still missing one piece: we need to generate the function call to actually make a postback.  There’s a relatively convenient way to do that, using the ClientScriptManager; drop this Literal onto the page:

        <asp:Button runat="server" style="display: none;" ID="btnFlashGo" OnClick="btnFlashGo_Click" />
        <asp:Literal ID="flashScript" runat="server" />

and wire it up in the backend:

public partial class _Default : System.Web.UI.Page 
{
    protected void Page_Load(object sender, EventArgs e)
    {
        this.flashScript.Text = string.Format(@"<script type=""text/javascript"">
function flashClicked()
{{
    {0}
}}
</script>", ClientScript.GetPostBackEventReference(this.btnFlashGo, ""));
    }

    protected void btnFlashGo_Click(object sender, EventArgs e)
    {
        lblText.ForeColor = Color.Blue;
    }
}

This setup should give us everything we need to make the AJAX call, and sure enough:

image

Better Approaches

I would be happier – much happier – with this solution if it didn’t depend on so much black magic.  There are a few ways to get it to work better, and while I don’t want to take the time to demonstrate them here, I can describe them a bit.

A FlashButton Control

In my mind, a FlashButton control is the best solution; it can derive from Control or WebControl (although to be honest simply Control is preferable), and can automatically do the heavy lifting.  You could incorporate SWFObject as a script resource and provide it automatically.  FlashButton could expose its own events, which means that you could eliminate that Button (the hidden one) and instead create the script callback reference pointing to the FlashButton’s event itself (and the UpdatePanel’s trigger could point there as well). 

A FlashButtonManager Control

A FlashButtonManager could extend the support for FlashButton much like ScriptManager does for UpdatePanel.  While the approach with a single FlashButton works well when only a single FlashButton is on the page, incorporating multiple FlashButton objects becomes tricky when you factor in things like handling multiple callbacks (for instance, naming the flashClicked() function).  A FlashButtonManager could be designed such that it handles each flashButton on the page, perhaps setting up FlashButtons with a FlashVar to specify a parameter when calling flashClicked(), and then using that parameter to determine which one was clicked and firing the appropriate postback event.

Final Thoughts

You need to enable your Flash app to talk to JavaScript in order to make AJAX partial updates work correctly.  Fortunately it’s not super-challenging to do so!  You should be careful though – there is some voodoo going on in the back-end of this kind of solution – but with creative architecture, you can avoid a lot of headache.

13Mar/090

Facebook UI for ASP.NET Launches

Posted by Rob

I’ve officially launched the Facebook UI for ASP.NET project at CodePlex at http://facebookui.codeplex.com/.  It’s kind of a preview edition of what’s to come – and there’s quite a bit.  Here’s a quick overview:

Now:

  • Design-time and run-time server support for about 30 FBML controls.
  • Visual Studio IntelliSense support for FBML controls
  • Web Site project templates in Visual Studio 2008

Version 1:

  • All FBML tags within: Users/Groups, Profile-Specific, Embedded Media, Visibility on Profile, Tools, Forms, Message Attachments, Additional Permissions, Notifications and Requests, Status Messages, Page Navigation, Dialog, and Wall sections.
  • Complete API support.
  • Extensibility points within the FBML controls framework and API framework to enable updates from Facebook to be rolled in without requiring an immediate release of a new library.

Version 1.5:

  • LINQ-to-FQL (or as some folks at the user group mentioned, perhaps it’ll be called LINQ-to-Facebook).
  • Higher-level controls will be included, such as validators and simple logic controls, like a birthday display.

Version 2:

  • Facebook internationalization should be supported.
  • Mobile platform support should be extended.

Beyond that, I’m not sure where we’re going to take the project.  We’ll have to see!

13Mar/090

Speedy C#, Part 4: Using – and Understanding – CLR Profiler

Posted by Rob

CLR Profiler is a free and incredibly useful tool offered by Microsoft.  I'm fairly certain its primary use (at least from Microsoft's perspective) is to illustrate use of the CLR Profiling COM APIs, which aren't exceptionally clear-cut (in my opinion), particularly from a .NET programmer's point of view.  The really difficult part of using CLR Profiler is becoming accustomed to its interface and the data it presents; however, once you do so, I'm certain you'll find it incredibly helpful in addressing difficulties with memory usage.  This article aims to introduce you to the "important parts" of CLR Profiler - specifically, which graphs you should view, how you should interpret them, and how to address the problems you find.  This article will not review some of the more complicated parts of injecting CLR Profiler into something such as your ASP.NET application; there are other resources for that purpose.

For the purposes of this article, I've re-introduced a wasteful error into BN# that I found by using CLR Profiler.  We'll work through finding it in this article.

Getting Started

Once you have CLR Profiler "installed" - and I use the term loosely - you can start the application from the install path (don't look for a Start Menu item).  There are two versions of binaries, x86 and x64 versions; you should know which edition of the application you'd like to run.  If you're running a platform-neutral application (most .NET apps would fall under this category), and you're on an x64 system, you should use that one.  If you're running 32-bit Windows, or are running a program specifically targeted to x86, then you should run the x86 version of CLR Profiler.

As an important note, for Windows Vista users, if you're running with UAC enabled, make sure to run CLR Profiler as an administrator.  CLR Profiler works by injecting a COM DLL into the target, but it can't do that if you're not running the process as an administrator.

CLR Profiler while it's not running anything

When profiling memory, I turn off Calls tracking: it's located in the bottom-right of the UI window.

If your application requires access to the local application directory - for instance, by using the Application class in Windows Forms - you should go through the explicit Profile Application menu item within the File menu, and set the working directory option of that UI.  Otherwise, go ahead and click Start Application, browse to your application, and go.

During Operation

Other than the fact that your application will be measurably slower, you should be able to run the application as you otherwise would.  Your mileage will vary, but you'll get better results with more memory in your system.  But all developers have at least 4gb powering their boxes now, right?

During the application, you can click on the Show Heap now button on the main CLR Profiler GUI, which will display a heap graph of the current application, displaying the path to all currently allocated memory:

Heap Graph of current profile

To be honest, I find the heap graph to be relatively confusing, but the good news is that you don't need to keep using it.  But once you've dumped that temporary log, you can view the current heap and interesting information by closing that window and, in the main CLR Profiler window, going to the View menu, and choosing Summary, which displays a cool window:

A result summary of a profile

This window helps you understand what's happening:

  • Allocated bytes is really interesting – it relates the total amount of memory that you’ve allocated within managed code.
  • Final Heap Bytes is the amount of managed memory that currently is in use on the heap.  This doesn't necessarily reflect unmanaged items.
  • Relocated Bytes is the amount of memory that has been moved by the garbage collector during compaction operations.
  • Gen X collections shows the number of garbage collections that have occurred for each generation.
  • Garbage Collector Generation Sizes shows the number of bytes being used by each heap.

What's Happening with BN#?

I had a suspicion based on memory usage (reported by Task Manager) that BN# wasn’t quite as efficient as I would have hoped.  I wanted to do some investigation, so I plugged in CLR Profiler.  After a 30-second (or so) connection to Battle.net, joining Clan Recruitment, this is what I saw:

Profile of BN# with intentional memory bug

That’s pretty heavy – 31mb or so total allocated memory but only ending up with about 3mb on the heap and only 3.5mb were relocated throughout the lifetime of the app – that told me that I was doing a lot of allocating and freeing very rapidly.  What’s the next step?

I clicked on the Allocation Graph button and took a look:

Allocation graph indicating 10mb of byte[] on the heap.

In this we can see that byte arrays are on the heap frequently and account for about 35% of all memory allocations.  That’s a big problem – especially since I pooled their creation already!  CLR profiler helps me track it down though, as I follow the highlighted call chain back to its source:

The culprit

This image indicates that I have a problem with a method called DataReader::get_m_data().  Now, as I mentioned, I had to recreate this problem, and the path of least resistance for me was to change the identifier m_data (used frequently in DataReader) to be a property instead of a field, so originally this said get_Data.  I thought that was odd until I saw its implementation:

        protected virtual byte[] Data
        {
            get
            {
                byte[] dataCopy = new byte[_m_data.Length];
                Buffer.BlockCopy(_m_data, 0, dataCopy, 0, dataCopy.Length);
                return dataCopy;
            }
        }

So here, for every operation that accesses the Data property (in the original implementation, it was every operation, because the Data property was virtual), I was duplicating the entire arrayEVERY TIME.

I then changed the implementation so that operations defined within the base class wouldn’t needlessly go through a property, and derived classes had direct access to the buffer by reference (via the UnderlyingBuffer property).  What were my results?

Final Results

I think that fairly well speaks to the effectiveness of using tools like this. :)  A decrease of 27% in allocations, 33% in gen-0 collections, and 53% decrease of the amount of byte[] allocations:

Updated allocation graph

Further Reading

The "Speedy C#" Series:

7Mar/090

Teaching Synchronization at University

Posted by Rob

I enrolled in the software engineering program at Arizona State, primarily because that program was willing to take me without having completed an undergraduate degree in computer science (the M.S. Computer Science and M.C.S. programs were not).  But as I went through the process I decided that I wanted to complete a thesis instead of a project, and I found a topic that I wanted to pursue that was clearly out of the realm of what I would consider software “engineering.”  So I began working on undergraduate coursework to make up the deficiencies; the first course I took was Operating Systems.  I thought that we would be actually creating a primitive operating system in the class and was consequently very excited (I’ve read a lot about the topic and have had friends who have worked on them, but I never have myself) – it turned out to be somewhat less exciting, but not too bad to talk about the way OSes allocate memory, manage resources, and provide synchronization services.

Now, I had just built up an app that made extensive use of mutexes to provide non-busy waiting while waiting for external hardware to do some work.  By the time class started, I had a solid understanding of the idea of synchronization primitives – you wait(), and you signal(); waiting allows a thread to pass through if the primitive has free slots and otherwise blocks the thread in a non-busy wait; signalling permits the primitive to allow one or more threads through.

One of our “lab” assignment was to implement a reader-writer lock using Java.  Except that it had mostly been completed already, and if that wasn’t adequate, there was “pseudo-code” to help us out.  Let’s take a look.

public final class Semaphore
{
   public Semaphore() {
      value = 0;
   }
   
   public Semaphore(int v) {
      value = v;
   }
   
   public synchronized void P() {
      while (value <= 0) {
         try {
            wait();
         }
         catch (InterruptedException e) { }
      }
      
      value --;
   }
   
   public synchronized void V() {
      ++value;
      
      notify();
   }
   
   private int value;
}
     

This class irritates me for one particular reason: its methods are named P and V.  Now, this class was almost a full year ago, and I wasn’t going to blog about it, except that I stumbled upon the Wikipedia article about semaphores today:

The canonical names P and V come from the initials of Dutch words. V stands for verhogen, or "increase". Several explanations have been given for P (including proberen for "to test", passeer for "pass", probeer "try", and pakken "grab"), but in fact Dijkstra wrote that he intended P to stand for the made-up portmanteau word prolaag, short for probeer te verlagen, or "try-and-decrease" (A less ambiguous, and more accurate, English translation would be "try-to-decrease".) This confusion stems from the unfortunate characteristic of the Dutch language that the words for increase and decrease both begin with the letter V, and the words spelled out in full would be impossibly confusing for non–Dutch-speakers.

I’m fully aware of the great things Edsgar Dijkstra has given to computing.  But let’s also consider that this concept made it into the ALGOL 68 programming language with better names than P and V.  In fact, Java has an intrinsic Semaphore class already built in that has methods named acquire and release.  When I submitted the assignment I chose to rename them to wait() and signal(), but that’s really beside the point.

The point of this whole rant is this: how can we expect students to come out of the university and be effective software developers if we stick to, frankly, archaic naming conventions like this, particularly in assignments which are presented like this by the professor?

See, while I understand the idea of “increase” and “decrease” being legitimate names for a data structure like this when you’re inventing it, now that we’re in the world of object-oriented programming, how can a university course really help its students?  I would submit that a semaphore has basic behaviors: wait and signal (or acquire and release), and that semaphores predefine a fixed number of threads that can acquire them.  But once that number is defined, what do we care how the semaphore is implemented?

We should be helping students get into the habit of following the rules of encapsulation.  Naming the methods of a semaphore “Increase” and “Try to decrease” not only leaks information about the internal mechanics of a semaphore (which may be inaccurate – semaphores could be implemented in hardware, for example, and the internal mechanics of a software semaphore representation of that hardware component may be entirely different), but it also misleads the developer because the names don’t really describe the behavior of what’s happening.  Arguably, wait() doesn’t because the calling thread only waits sometimes (and so I prefer acquire() in this situation), but at least that’s relatively close.

When the wet-behind-the-ears college graduate gets out of school, we don’t expect years upon years of experience from which they may draw.  But I think that it wouldn’t hurt to get these habits in front of students as early and frequently as possible.

Tagged as: No Comments