image

I thought I would do a best of post, as many others seems to be doing it.

Most viewed posts (in order):

  1. IoC Container Benchmark - Unity, Windsor, StructureMap and Spring.NET
  2. NHibernate 2.0 Events and Listeners
  3. JQuery and Seperation of Concerns
  4. Cleanup your html with JQuery
  5. Breadcrumb menu using JQuery and ASP.NET MVC
  6. Url Routing Fluent Interface
  7. Creating a WatiN DSL using MGrammar
  8. View Model Inheritance
  9. NHibernate 2.0 Statistics and a MonoRail filter

image This was actually my first year of blogging. I have had ambitions to start blogging for years but just never got around to actually do it, mainly because of two reasons. First I thought that if I was going to start blogging I would need to setup a custom domain, fix hosting, install and configure some blogging software, etc. All that initial setup time was a road block to start blogging, I could just not find the time to do it because of a heavy work load and a lot of overtime. The second reason was that if I started blogging I wanted it to be a serious blog with good valuable content, and for that I felt that I needed to mature and gain some more experience.

But in retrospect I regret that I did not start sooner, during 2005-2007 I was a lead developer on a multi-tenant B2B system where I did some interesting work with url rewriting with WebForms (before there was much information about it), started using NHibernate and Castle Windsor, SAP integration logging using AOP, etc, in short I had a lot to blog about that could have been valuable and helpful to the .NET community.

I am very glad that I eventually started blogging because it has been a very fun, rewarding and learning experience. One of the reasons that I eventually started was the extremely easy setup that Google's blogger service provided where you could register a domain and start blogging in a mater of minutes. Google's blogger service has been pretty great as a way to get started, the only problem is the bad comment system and some lack of flexibility. I will probably be moving to a hosted solution where I can run and configure the blogging software myself during 2009 but right now it is not a top priority.

I want thank everyone who subscribes or reads this blog, I have been very pleasantly surprised by the amount of people who subscribe, it is very motivating to see that people find value in things I write and keeps me wanting to write more and better.

Merry Christmas & Happy New Year

If you have worked with DTS or SSIS packages you probably know that they can quickly become painful to work with, especially concerning versioning, logging and above all deployment.

I recently tried Ayende´s open source ETL framework RhinoETL, it tackles the ETL problem from a completely different angle compared to DTS / SSIS. At it's heart RhinoETL is a very simple .NET framework to handle an ETL process, the key components are processes, pipelines and operations. 

The process that I needed was a very simple one, namely to update a text table stored in multiple databases. The update could be of different types, for example swap every occurrence of a text translation to another and delete or update a specific row. In previously releases this was handled by writing manual update scripts. The release that I am working on currently however requires extensive changes to the texts in these tables spread over many databases and writing repetitive SQL scripts was not something that I felt doing. It felt like a good opportunity to try RhinoETL.

I began writing this input operation:

public class ReadWordList : InputCommandOperation
{
    public ReadWordList(string connectionStringName) 
      : base(connectionStringName) {  }

    protected override Row CreateRowFromReader(IDataReader reader)
    {
        return Row.FromReader(reader);                
    }

    protected override void PrepareCommand(IDbCommand cmd)
    {
        cmd.CommandText = "SELECT * FROM Wordlists";
    }
}

This is the first operation, its responsibility is to fill the pipeline with rows from the Worldlists table. The next operation is the one updating the rows. It takes as input a list of ITextChange instances that is the object that performs the change.

public class TextChangeOperation : AbstractOperation
{
    private IList<ITextChange> changes;
  
    public TextChangeOperation(IList<ITextChange> changes)
    {
        this.changes = changes;
    }

    public override IEnumerable<Row> Execute(IEnumerable<Row> rows)
    {
        foreach (var row in rows)
        {
            foreach (var change in changes)
            {
                if (change.IsValidFor(row))
                    change.Perform(row);
            }

            yield return row;
        }
    }
}

There is a class hierarchy representing the different types of text changes:

 

image

The code for the GeneralTextSwap class is very simple:

public class GeneralTextSwap : ITextChange
{
    public string TextOld { get; set; }
    public string TextNew { get; set; }

    public bool IsValidFor(Row row)
    {
        var text = ((string) row["Text"]).Trim();
        return text == TextOld;
    }

    public void Perform(Row row)
    {
        row["TextOld"] = row["Text"];
        row["Text"] = TextNew;
    }
}

The point of storing the old text value is that I have an operation after the TextChangeOperation that logs all changes to a csv file. The changes for a specific release is then just defined as static list on a static class. For example:

public class ReleaseChanges_For_Jan09
{
  public static IList<ITextChange> List;

  static ReleaseChanges_For_Jan09()
  {
    List = new List<ITextChange>()
    {            
      new GeneralTextSwap() 
      {
         TextOld = "some old link value",
         TextNew = "some new link value"
      },
      new UpdateTextRow()
      {
          DbName = "DN_NAME",
          WorldList = "USER_PAGE",
          Name = "CROSS_APP_LINK",
          TextNew = "New link value"
      },
      new DeleteTextRow()
      {
          DbName = "DN_NAME_2",
          WorldList = "SOME_PAGE",
          Name = "SOME_LINK"
      }
    }
  }
}

The above is just an example in reality I have hundreds of general and specific changes, and yes this is legacy system which handles text and links very strangely. The above code could easily be handled by a simple sql script with about the same number of lines of TSQL code, the benefit of placing this inside an ETL written in C# is that I can easily reuse the change logic over multiple databases and I also get great logging and tracing of all changes. The point of this post is to show how easy it can be to use OOP to model and abstract an ETL process using RhinoETL. Even though this ETL process i very simple it serves as good example for showing how a TSQL script, DTS or SSIS packages can be rewritten and simplified using the power of an object oriented language.

Fredrik Normén recently posted about how crosscutting concerns are implemented with boring and often duplicated code, which can be handled by using aspect oriented programming (AOP). I have used AOP with great success in previous projects to handle scenarios like logging, transactions and change tracking.

The problem with AOP is that it is not well supported or integrated into the CLR or the .NET framework and toolset. Sure there are great AOP frameworks that solve the problem via dynamic proxies, but such solutions have some big restrictions, for example they only work with virtual functions and you need to instantiate the object via a proxy creator. PostSharp handled AOP differently, it is a framework and a .NET post compiler that injects your crosscutting concerns at compile time. The problem I have found with PostSharp is that the post compile step is kind of slow, which is an issue if you do TDD (because you are constantly recompiling). Beside the compile time issue PostSharp is great tool and framework with great flexibility and power.

I was kind of disappointed that Microsoft didn't present any plan to make AOP scenarios easier and more natural on the .NET platform at this years PDC. I am not sure how, but I just feel that the AOP experience on .NET could be vastly improved :)

A couple of weeks ago I was invited to a presentation by Scott Ambler on "Scaling Agile Software Development: Strategies for Applying Agile in Complex Situations", it was an interesting talk. He showed a lot of diagrams and stats from a 2008 Agile Adoption rate survey, you can find the info here. The survey results are for example agile adoption rates, success rates, increase / decrease in code quality and costs. The problem with the survey is that it does not include any definition of what qualifies as agile. There are many different definitions on what makes software development agile, and many definitions like the manifesto (if you can even call that a definition) are very vague. I am not saying there is anything wrong with the manifesto or that agile needs to be better defined, what I am saying is that because agile can mean a lot of things to different people, and that many think they are doing agile while others might heartedly disagree, makes agile surveys like this very hard to interpret.

For example, the survey that covers modelling and documentation practices concludes that:

  • Agile teams are more likely to model than traditional teams.
  • Traditional teams are equally as likely to create deliverable documentation.
  • For all the talk in the agile community about acceptance test driven development, few teams are actually doing it in practice.
  • etc..

Without some minimum criteria for what qualifies as "doing agile" these surveys don't tell me that much. Even if I would disagree with the definition I think they would make surveys like this a lot more interesting.

I am not saying that the surveys are worthless, they do tell you something and they are a great resource for convincing management of agile practices so it is great that someone is taking the time to make them. 

image I am very interested in how natural languages work and evolve (see my review of the book The Language Instinct) and ever since I began playing with MGrammar I have wanted to see if it was possible to define English sentence structure using it.

Lets begin with a simple sentence:

The boy likes the girl.

This sentence is composed of the noun phrase (NP) "the boy" and a verb phrase(VP) "likes the girl". So lets begin with this MGrammar syntax:

syntax Main = S*;        
syntax S = NP VP ".";

If we look at the noun phrase "the boy", it is composed of determiner followed by noun, like wise if we look at the verb phrase "likes the girl" we see that it is composed of verb followed by a noun phrase. The MGrammar should then be:

syntax NP = Det? N;
syntax VP = V NP;

Then we just need to add some determiners, verbs and nouns:

syntax Det = "a" | "the" | "one";
syntax N = "boy" | "girl" | "dog" | "school" | "hair";
syntax V = "likes" | "bites" | "eats" | "discuss";

If you add an interleave rule to skip whitespace the sentence should be correctly parsed. That was a really simple sentence, lets add an adjective.

The nerdy boy likes the girl.

We need to modify the noun phrase rule. Before the noun an optional amount of adjectives (A*) can be placed. This is a simple change, just add A* to the noun phrase rule and add some adjectives.

syntax NP = Det? A* N;
syntax A = "happy" | "lucky" | "tall" | "red" | "nerdy";

That was simple, lets add something more to the sentence, for example:

The nerdy boy likes the girl from school with red hair .

I added a nested prepositional phrase (PP). A prepositional phrase is, according to wikipedia, composed of a preposition(P) and a noun phrase.

syntax NP = Det? A* N PP?;
syntax PP = P NP;
syntax P = "on" | "in" | "from" | "with";

The recursive nature of the PP phrase makes it possible to nest infinite number of prepositional phrases inside each other. Here is an illustration of the syntax tree for "girl from school with red hair":

image

I think I will stop here because this post is turning in to an English grammar lesson and I don't won't to loose all my subscribers :) Defining the English sentence structure in MGrammar is pretty pointless, unless you are building a grammar checker, in which case you are still out of luck as it will probably be impossible to define grammar for how words are built and you will run into trouble with ambiguity (which most natural languages have). But it was a fun try, and it is a good example for showing how recursive rules are parsed.

If you missed Martin Fowlers post on Oslo, it is a good read, I like how he defines it as a Language Workbench.

PS. I have started twittering, I know am late to the game, I just didn't get the point of twitter. I have been using it for a two days now and I am beginning to see the light. Oh, and please skip pointing out the irony with the inevitable grammatical errors in this post :)

imageMy main area of interest in programming during my school and university years was in computer graphics. I did a lot of hobby programming with games, like the Quake 3 mod Rocket Arena 3, and specifically with fractal graphics. This generated a quite serious outcome during a course on research methodologies where I together with a friend wrote a scientific paper on 4D Julia rendering.

The 2D Julia fractal to the left is a shape that everyone recognizes. The simple formula that defines the Julia and Mandelbrot set is also something that many know:

image

The variable z and constant c are in a normal 2D Julia set complex numbers. A complex number is a tuple of two values, a real and imaginary part, which can be visualized in the complex pane, which is a 2D geometric representation of complex numbers. 

image

Complex numbers can be extended to a four-tuple value with one real and three imaginary units. These are called Quaternions and can be visualized as a four dimensional space. When using the Julia equation with Quaternions we can define a 4D dimensional object. It becomes a lot trickier to visualize however. What is needed is a 4D camera from which you can cast rays (vectors). The algorithm traverse these vectors step by step (from the camera) and for each point the Julia equation is recursively run to determine if the point is part of the Julia set or not.

Here is 2D illustration of how a ray is thrown and sampled at regular intervals:

image

The process of sampling each point can be very time consuming so the distance between sample points can not be to small, but it can not be the to high either as it directly relates to the point resolution you get as exemplified by these two images:

image

The left image has a higher distance between sample points compared to the image to the right. You can clearly see the artifact of how the algorithm works in the rings in the image to the left (each ring is a sample point along the ray). To optimise the sampling process we can move back after a sample hit and then move forward in smaller steps.

image

Here is the Julia set viewed from the side, the constant (Julia seed) is (0.4 0.5i 0j 0k).

image

The raytracing gives coordinates in a 4D space that belongs to the Julia set, each point can be given a color depending on how far the point is from the camera (as above). Further each point can be lighted and shaded. To accurately light a point you need it's surface normal, these normals can be calculated by looking at the relative heights of neighbouring points (pixels).

image

The most fascinating aspect of 4D Julia sets can be viewed when you animated them. To animate them you render pictures with small change to the Julia seed constant or to the camera position. What is particularly interesting is when you move the camera in the fourth dimension. What you get is an animation of how the Julia set looks like from different four dimensional angles. It is hard to describe what happens, but basically the shape morphs into different shapes when viewed from different four dimensional angles :)

During the process of coding our raytracer we accidentally made an error in the quaternion multiplication, which resulted in a very strange shape:

image

We named these über quaternions. Later when we were writing the paper we discovered that we had used commutative multiplication rules which resulted in hypercomplex numbers.

Here is another picture showing how the shape gets more detail depending on the number of times you recurse over the Julia algorithm:

image

Isn't it fascinating and beautiful how such a simple algorithm can define such an intricate structure? The raytracer was witten in C++ and my plan is to (someday, maybe next year) port / rewrite it in F# :)

The paper is very technical so I doubt that it will interest many, but here is the link and abstract:

By using rules defined by quaternion algebra the Julia set can be extended to four dimensions. Techniques for visualizing the four dimensional Julia set as a three dimensional object has previously been explored, however those techniques usually ignores the fourth dimension. In this paper we describe our attempt to extend the already established technique of ray tracing Julia sets to fully incorporate its four dimensional properties. We also discuss an optimisation algorithm that drastically increases the amount of details in images and shortens rendering time.

Link to the paper: Raytracing4D.pdf

Movies:

  1. 4D Camera move: 4d_cameramove.avi
  2. Another 4D Camera move: r4d.avi
  3. Moving the seed in circle (on the complex pane): cycle.avi
  4. Moving the seed from left to right (on the complex pane): cwalk.avi
  5. Moving the camera in a circle around the hypercomplex julia set:  uber_highres.avi

It is worth pointing our that in the two first movies with 4D camera movies it is only the camera that is moving the julia seed never changes, only the angle that the object is viewed at. The paper was written with LaTeX, which was a lot of fun to learn, although quite hard to use. But the output looks so much more professional.

Were you also a graphics geek? Did you write a flame or plasma effect in assembler, did you use allegro to write DOS games? I sure did :)

My first try with writing a DSL for WatiN was such fun experience that I decided to have another go. I wanted to try to create something with a little more natural sentence like syntax. Here are some tests showing of the new syntax:

--- Can filter by author ---
goto address "http://demo.codesaga.com/". 
click the link with the text "MvcContrib".
select "torkel" from the list with the id #author-fitler.
page should contain the text "Filtering view by author torkel".

--- Can filter by date --- 
goto address "http://demo.codesaga.com". 
click the link with the text "xUnit".
set focus to the textbox with the id #date-filter.
click the link with the text "2".
page should contain the text "Filtering view by date".

--- Can expand diff in changsest view (via ajax) ---
goto address "http://demo.codesaga.com/history/xUnit?cs=25434".
click the element with the class name @cs-item-diff.
page should contain the element with the class name @code-cell, wait for
it 3 seconds.

The added verbosity might be to much for programmers but the point of making something like this more readable is to make acceptance tests understandable by non-programmers. I am not saying that acceptance tests is something that shouldn't involve developers. But having them accessible to for non-programmers can be very valuable. I am not sure why, I kind of like the verbosity in this case (I usually don't). It would be very easy to make some words optional so one can write "click #edit" as a shortening of "click the link with the id #edit".

The MGrammar for this language:
module CodingInstinct {
    import Language;
    import Microsoft.Languages;
    export BrowserLang;
 
    language BrowserLang {
                  
        syntax Main = t:Test* => t;
        
        syntax Test = name:TestName
            a:ActionList => Test { Name { name }, ActionList { a } };
                                      
        syntax ActionList
          = item:Action => [item]
          | list:ActionList item:Action => [valuesof(list), item];
                             
        syntax Action = a:ActionDef "." => a;
        syntax ActionDef
            = a:GotoAction => a
            | a:ClickAction => a
            | a:SelectAction => a
            | a:TextAssert => a
            | a:ElementAssert => a
            | a:TypeAction => a
            | a:SetFocusAction => a;
                            
        syntax GotoAction = "goto" "address"? theUrl:StringLiteral
            => GotoAction { Url { theUrl } };
                
        syntax ClickAction = "click" "the"? ("link" | "element")? ec:ElementConstraint 
            => ClickAction { Constraint { ec } };
        
        syntax TypeAction = "type" value:StringLiteral "into" "the" "textbox" ec:ElementConstraint
            => TypeAction { Value { value> }, Constraint { ec } };
            
        syntax SelectAction = "select" value:StringLiteral "from" "the" "list" ec:ElementConstraint
            => SelectAction { Value { value }, Constraint { ec } };
        
        syntax TextAssert = "page should contain" "the" "text" text:StringLiteral 
            => TextAssert { Value { text } };
            
        syntax ElementAssert = "page should contain" "the" "element" ec:ElementConstraint 
            wait:ElementWait?
            => ElementAssert { Constraint { ec }, Wait { wait } };
            
        syntax ElementWait = "wait" "for" "it" sec:Base.Digits ("second" | "seconds") 
            => sec;
            
        syntax SetFocusAction = "set" "focus" "to" "the" "textbox" ec:ElementConstraint 
            => SetFocusAction { Constraint { ec } };
                 
        syntax ElementConstraint
            = "with" "the" "text" name:StringLiteral => TextConstraint { Value { name } } 
            | "with" "the" "id" name:ElementId => IdConstraint { Value { name } } 
            | "with" "the" "class" "name" name:ElementClass => ClassConstraint { Value { name } };
                  
        token TestName = "--- " (Base.Letter|Base.Whitespace)+ " ---";    
        token ElementId = '#' (Base.Letter|'-'|'_')+;
        token ElementClass = '@' (Base.Letter|'-'|'_')+;
                                                    
        interleave Skippable
          = Base.Whitespace+ 
          | Language.Grammar.Comment
          | Base.NewLine
          | ",";
                
        syntax StringLiteral
          = val:Language.Grammar.TextLiteral => val;        
    }

}

Another improvement in this new DSL syntax is the format for specifying an element or class name. In the above grammar these are defined as tokens, where ids begin with # and class names with @ followed by any word. The nice thing with a token is that you can add a Classification attribute to it where you specify what token category it belongs to. Classification names are linked to font and color styles (i.e. syntax highlighting).

image 

To get the DSL to actually execute you need the MGraph node tree that the parser spits. The MGraph is not something that you want work with directly as it is pretty low level. When I did the first version of this WatiN DSL I spent the majority of the time figuring out how to parse and deserialize the MGraph into a custom set of AST classes. In the process I wrote a very basic generic MGraph -> .NET classes deserializer.

Luckily, as Don Box pointed out in the comments to my previous post, SpankyJ has written a much better deserializer that converts the MGraph into Xaml via an MGraphXamlReader. It was very easy to switch to his implementation as he had some useful method extensions on the DynamicParser.

Example:

DynamicParser parser = LoadExampleGrammar();

var xamlMap = new Dictionary<Identifier, Type>
    { { "Person", typeof(Person) } };

var people = parser.Parse<List<object>>(testInput, xamlMap);

But having to define the mapping between MGraph node names and .NET classes manually like this was something I did not like. I wanted something with a more convention based approach. Roger Alsing is also doing some work with MGrammar and he gave me this great piece of code which I modified slightly:

public Dictionary<Identifier, Type> GetTypeMap()
{
  return Assembly
    .GetExecutingAssembly()
    .GetTypes()
    .Where(t => t.Namespace.StartsWith("WatinDsl.Ast"))
    .Where(t => !t.IsAbstract)
    .ToDictionary
    (
      t => (Identifier)t.Name,
      t => t
    );
}

Pretty simple code really,  it just creates a dictionary of all the non abstract types in the namespace WatinDsl.Ast.

I basically rewrote the AST for this new version, now most actions have a Constraint property that determines what element the action is targeting. Here is an sample:

public class ClickAction : IAction
{
    public IElementConstraint Constraint { get; set; }
    
    public void Execute(IBrowser browser)
    {
        browser.Element(Constraint.Get()).Click();
    }
}

public class IdConstraint : IElementConstraint
{
  public string Value { get; set; }

  public AttributeConstraint Get()
  {
    return Find.ById(Value.Substring(1));
  }
}

For the full code: WatinDsl_2.zip

This is still just an experimental spike for learning MGrammar, but it is also an interesting scenario for exploring the potential in a browser automation language. Is a browser automation language, like the one I have created, something that you would find useful? What would your syntax look like?