Kappy Space by Joao Matos Silva

It’s time to move on from Agile Software Development (It's not working) - My Opinionated Review

2024-06-25T00:00:00+00:00

Recently I’ve bumped into this video from Dee called “It’s time to move on from Agile Software Development (It’s not working)”. It’s awesome! If you’re not following her, do so. I did. The video touches on a lot of the bad adaptations of so-called Agile methodologies to Software Development. While I really agree with most of the content, there are some ideas that I feel were only focused on the software developer side and not the company or product delivered. This is my opinionated review.

I’ll start with the bigger point for me, in this case a user comment saying that “Waterfall done well is better than agile done poorly”. Yes, that’s absolutely true. And while there are quite a lot of companies doing agile poorly, there are (or were, before agile became the trend) companies doing waterfall poorly as well.

Waterfall was replaced by agile almost as an avalanche. It did so for a reason, the ability to change and the iterative nature of agile. Yes, the developer is loving waterfall, he’s probably at the middle or close to the end of the project. Is only at the end where the waterfall problems start. It’s the final delivery, and the client sees the product and it doesn’t match the expectations. Worse, the time of development has already been entirely consumed and a lot of changes arise. The end result? Extra time for the developer that needs to implement the changes that resulted from bad interpretation of the initial project requirements. Is that or the company needs to delay the start of new projects because it can’t close the current one, but the developer is always pressured to deliver, since for the client, it is the company that is behind schedule. Both waterfall and agile will lead to the same outcome, over-stress.

This is a great bridge to the next point. The over saturation and burn out of the developers on agile. This is a pretty serious one. Not only developers, but almost every role in IT has this issue. This industry is stressful by nature. It’s not going to be the software development methodology that is going to change it. We live by estimates, and while they are just that, rough guess estimations, they are often treated as contracts. Not just it, there’s always tech debt rising, client changes, bugs that make you lose more time than expected… It’s not easy! Working without estimates or compromises, is like bartending on a Jamaica beach. A dream job! The nature of development it’s money/cost/profit at the end of the day. That’s why this job is stressful. Estimates and the release schedules are the contracts the company establishes with the client, and yes, with shorter cycles, they can represent more iterations , more estimations and hence more stress, but at the same time the feedback and steering gets shorter so there is less wasted time overall.

Imagine a high performance athlete. He/she has established aggressive objectives and trains every day to reach them. When the objectives aren’t met, it causes frustration. It’s the same with development when the estimates are not met, but they are needed because that is the only way the company has to macro plan its operations.

My next and final point is the “Meetings, Meetings, Meetings…”. Oh, so true… It gets even worse as you gain more responsibilities, your time coding gets replaced by more and more meetings. The key is actually to make them relevant. Does everyone need to attend all team meetings? I’m a strong believer that no. Stands ups should be short and focused on the daily steering or prioritization of the team. They are not for reporting who is working on what. For that end there are issue tracking tools. Since they’re short, it’s always good to have all the team, especially now with so much remote work. Sometimes it is the only time of the day where you actually interact with some of your team members. Grooming, refinements, planning meetings should not stop the entire team. Most of them are focused on the negotiation of 3 vectors: Product, Management and Engineering. At most it should be ok with a representative of each. Product or Project manager will always push to get the things done, define their priority and represent the client. The Manager or Team Lead act as the company and negotiate the deadlines and represent all the team. As for engineering, this is where I have my personal opinion that booking all the development team for hours is not efficient. This role can be fulfilled with a more senior engineer, or better yet, the so called Staff Engineer or Principal Engineer. It’s a role that needs to be aware of the project and needs to translate the product requirements into implementation details. It’s the one responsible for the technical executions of the project, so it needs to be aware of its implementation. As such, it’s the perfect role to make the estimations for the team instead of all of them spending hours playing poker.

A short word on retrospectives, as they tend to be long and full of discussions. It helps quite a lot of you bring your topics prepared. But keep the topics short and meaningful.

Does agile development have its problems? it does. Are they enough for us to revert to waterfall methodology? Please no!

As with everything in software development, we should use what is best and works for us. Is it SCRUM, KANBAN, or others? It is up to you to find out. One way or the other, it is the company standards that we need to follow to whom we do represent. This is not a “Stop Whining” post, I’m only stating that companies and the deliveries are the main focus. Only then comes the developers, that are the engines and the powerhouse of a software development company and as all engines they need to be fueled, oiled and clean to work properly. This is where the interpretation of these frameworks needs to be contextualized to the reality of each team. We don’t need an agile 2.0, each team needs to tune the agile that fits best for them.

Fixing the Generic Repository

2024-02-20T00:00:00+00:00

It has already been 4 years since my last post and almost 8 since I was actually writing on this blog, but I keep seeing developers using bad practices that are advertised as good and clean. The most obvious of them is the Generic Repository.

The Generic Repository is a pretty popular method of abstracting the data layer from the core business, and my personal opinion that why it got so popular is because it is easy and requires quite low code. You only implement the generic class, then all entities specific repository instances derive by generic instantiation, without any extra implementation. More so, there are innumerous implementations on the internet that can be easily adapted. The problem is that all of them have an issue while querying data. Every entity is queried differently. I’ve seen a few different flavours out there.

The most gross one is this public IEnumerable<TTentiry> GetAll(). It doesn’t accept any parameter and it returns an IEnumerable<TEntity>. Would you use this method to query a table with millions of rows? For example, we have a table with Books, would you use this to find a book by title, or a list of books by author? Using this method to query any kind of information results in loading into memory all the rows of the table and only then filtering the rows. This would prove extremely inefficient. You’re not making use of the query features of your database, and that is one of the main reasons they exist in the first place.

A close second is public IQueryable<TTEntity> GetAll(). Difference is that now we can use linq to query the data. But now I wonder… What is the value this layer generates if I can query whatever from my business? What prevents a developer from making a query that doesn’t have the right indexes on the database? We’re completely leaking the queries to the business layer where it is quite easy to make mistakes that will result in poor inefficient queries (and then we’ll just blame Entity Framework for performance issues).

Another flavour, and this one diverges a bit from the Generic Repository, is to have specific repository classes that derive from the Generic one just to be able to create specific query methods adapted to its entity. This sacrifices the “no-code” approach but addresses the last two issues that were just mentioned. The only problem of this flavour is scalability. How would we still handle all the different ways to query books?

/* ... */
PageOfBooks GetBooksByAuthor(string author, int page, int pageSize);
PageOfBooks GetBooksByYear(itn year, int page, int pageSize);
PageOfBooks GetBooksByGenre(string genre, int page, int pageSize);
Book GetBooksISBN(string isbn);
/* ... */

For each query, we need to create a new method. Everytime we need to change/add a query, we are violating the Open/Closed principle since we’re always modifying the entire Repository class.

I’ve promised a fix, and no, it isn’t just a click bait. The way I propose to fix the Generic Repository is Query Encapsulation. By this, each query is written in its own class and our repository would have just a single method like TOut Query<TOut>(IQuery<TEntity, TOut> query);. Queries would be encapsulated just as this:

public class GetBooksByAuthorQuery(string author, int page, int pageSize) : IQuery<PageOfBooks, BookEntity>
{
    public override PageOfBooks Execute(IDbSet<BookEntity> set)
    {
        return set
            .Where(book => book.Author == author)
            .Skip((page - 1) * pageSize)
            .Take(pageSize)
            .ToPage<PageOfBooks>()
    }
}

Using it on the business layer is simple as creating the query instance and passing it to the repository instance.

var booksByAuthor = repository.Query(new GetBooksByAuthorQuery(author, 1, 10));

A last note on why use Repositories and EntityFramework (or other kinds of ORM/micro-ORMs). I don’t have any. I simply don’t see a point. I could understand the point where developers want to be agnostic of Entity Framework to be able to replace it with other things in the future, but in 20 years, I never saw that happening. Like I said before, chances are, the business code would need to be modified as well. A change like that will always require a great investment because we would need to at least re-write an entire layer from scratch and we all know technical debt is always the least prioritisation because it doesn’t generate money.

My advice is keep it simple, and if you’re already using EntityFramework, you don’t need to abstract it.

If you don’t feel up to writing your own queries, and don’t feel up to reinventing the wheel, take a look at Ardalis Specification nuget package. It follows a little different signature as this examples, but is way neater. It’s pretty cool!

Replacing the Data Layer

2020-01-20T00:00:00+00:00

For the first time in my professional career, I’ve made the exercise of changing the data backing store of an application. In this case, I’ve changed from a document database to a relational database. The goal was simple… Change the data layer of the application without changing anything else. The application followed a typical onion design of presentation, domain and data layers. This last one by the form of the very well known Repository pattern.

public interface IBookRepository
{
    Guid GetById(Guid bookId);
    IEnumerable<Book> GetAll(string name);
    void Save(Book book);
    void Delete(Book book);
}

There is not much to be told about the Repository itself. Is as plain as it can be. A method to get an item, a method to search items, a save and another to delete items. The important bits to hold are the interface objects for input and output of the interface, those are domain objects. Remember the onion design I said before?

Ok, enough talk, on to the action. I got to say, the big part of the things went smoothly. Simple domain objects were mapped from one repository implementation to the new one without hassle. The real problem rose with more complex domain objects. For example, a Book has a list of Authors. Since we were using a document database, that was really a no brainer. The authors list was denormalized and stored along side the book data. Every update on the authors was simply persisted by saving the entire document on an atomic operation.

But now on the relational database, each Author has its own row on a dedicated table. The million dollar question, how do we know which authors were changed? From the DDD purity, the Domain objects should not have persistence details, so long for tracking changes. This means we need a way to keep sync between what we have on database and our domain object that was passed.

One solution is to eagerly load the inner authors list from our database entity and enumerate each domain author from input to check if already exists on the database, if no then add it. Then we need to enumerate the list from our database and check if all of them are still present on the domain list, if no, then remove it. And don’t forget to update the ones in the middle if needed.

public void Save(Book entity)
{
    //get the existing Book from the Database, eagerly loading the Authors 
    var book = await dbContext.Books
        .Include(x => x.Authors)
        .FirstOrDefaultAsync(x => x.Id == entity.Id);

    if (book == null)
    {
        book = new Model.Book
        {
            Id = entity.Id,
        };

        dbContext.Bundle.Add(book);
    }

    //Update the entity Properties
    book.Name = entity.Name;


    //check for Authors added
    foreach (var domainAuthor in entity.Authors)
    {
        var author = book.Authors
            .FirstOrDefault(x => x.Id == domainAuthor.Id);

        if (author == null)
        {
            //load the existing Author entity from Database
            author = dbContext.Authors.FirstOrDefault(x => x.Id == domainAuthor.Id);
            book.Authors.Add(author);
        }
    }

    //check for authors removed
    var existingAuthors = book.Authors.ToList(); //create a copy list in order to allow us to remove items while enumerating
    foreach (var existingAuthor in existingAuthors)
    {
        var author = entity.Authors.FirstOrDefault(x => x.Id == existingAuthor.Id);
        if (author == null)
        {
            book.Authors.Remove(existingAuthor);
        }
    }


    await dbContext.SaveChanges();
}

In this case we are using EF inside the repositories, but the same would apply using another ORM or even micro-ORM as there are no changes tracking.

Would this happen if the application was built using a relational database in the first place? I seriously doubt it. Chances are that either specific methods were created on the repository to add those entries on the inner list, or that the domain object would be replaced by the entity object itself. I think it’s safe to say, that no matter what, our choice of the backing store still affects the way we design an application, from top to bottom. Of course we do our best to create that layer of abstraction, but we’re only humans. We can only predict based on what we know.

If it was my design, I’m pretty positive that no repositories would be seen anywhere. I wouldn’t use the Onion Layers design. I would create vertical slices on the domain and mapping directly the data objects on the application. I wouldn’t care for database layer abstraction because I really want to use what each database features serve me best. Would this exercise be as “easy” as it was using that model? No, I guess not. But again, in more than 15 years, this is the first time I actually did it, and as of now, it is more academic than professional.

Single Entry, Single Exit... No wonder it's still single...

2016-01-24T00:00:00+00:00

During an internal code review, I asked a colleague of mine why was he using a variable inside a method just to hold the result of it and return it in the end. Why not just return immediately as soon as it knows the output? He told me that he was trying to follow the “Single Entry, Single Exit Principle (SESE)”.

My eyes rolled a bit… What was he talking about? The answer was a google away. In short, a routine should start only on one single location and exit on another single location.

Well… Let’s see the entry first… On modern iterative programming languages (like C#), all methods always start on the first line of the method declaration. This is almost true, since weird stuff can be accomplished with the goto statement but I can’t recall the last time I have ever used it (on c#) or even see it being used outside decompiled code. So.. I think we can assume the we’re using single entry by default.

Single Exit… this is where the issues arise. This mean that we can only use return once? This should have been called “Single Return” (k). Does this means we can’t use a throw new Exception("bla bla bla")? The issue I have with this, is that I like to return early. I like to keep the level of nesting and indentation inside a method to the minimum.

Take a look at this example. A simple routine (for demo proposes) to calculate the length of any string except for spaces until it finds the work stop.

public int CalculateStringLength(string text)
{
	int length = 0;
	if(!string.IsNullOrEmpty(text))
	{
		var indexOfStop = text.IndexOf("stop");
		if(indexOfStop > 0)
		{
			text = text.Substring(0, indexOfStop);
		}
		if(indexOfStop != 0)
		{
			text = text.Replace(" ", "");
			length = text.Length;
		}
	}
	return length;
}

I see there 2 level of indentation. This was a dummy method just built only for this example. A real one could have pretty worst levels of nesting…

Let’s see how the same exercise would look without the single “return” rule.

public int CalculateStringLength(string text)
{
	if(string.IsNullOrEmpty(text))
	{
		return 0;
	}
	var indexOfStop = text.IndexOf("stop");
	if(indexOfStop == 0)
	{
		return 0;
	}
	if(indexOfStop > 0)
	{
		text = text.Substring(0, indexOfStop);
	}
	var textWihoutSpaces = text.Replace(" ", "");
	return textWihoutSpaces.Length;
}

Less indentation improves readability and maintainability. In this example we kept the nesting level only to 1.

So I guess this is why this Principle is still Single after all these years. Maybe it should create a profile on Match dot com.

Breaking Enum

2015-10-24T00:00:00+00:00

Breaking changes comes in many ways.

Typically we’re used to think of breaking changes when we change method signatures, classes, return types or public interfaces. We as programmers know that we should avoid breaking changes because that can affect every little piece of code that depends on the changed part.

While they must be avoided, they’re normal part of code evolution. New features sometimes need big changes. Semantic versioning give us some helper lines in order to manage expectations on the library consumers whether a new version might include or not breaking changes. Usually only on major versions increments, a library can include them, but even so, they’re should be avoided. There’s always some “disappointment” when we have to re-write our code that was already working in order to adapt to a new library version.

Enough of talking…

Imagine that we have an application that consists on a core library (named BreakingEnum.Common.Dll on this example), another helper library that can be a third party plugin (named BreakingEnum.Helper.Dll) and a frontend application, that puts it all in motion. All of this components are only referenced by the libraries and not by source, as they are provided by different providers.

On BreakingEnum.Common.Dll for this example, only this enum if defined:

public enum StatusEnum
{
    Active,
    Disabled
}

The fact that Active is defined first, is indeed arguable, but it was only to better explain this example. It has no impact, it would only require me to write this example in a different way. On the BreakingEnum.Helper.dll, the library that simulates a third party plugin, It only consists on a method that accept a string and check if the value matches the active state.

public class StatusHelper
{
    public static bool IsActive(string statusText)
    {
        var status = (StatusEnum) Enum.Parse(typeof (StatusEnum), statusText);
        var isActive = status == StatusEnum.Active;
        return isActive;
    }
}

Seems simple enough… For the last part, the application that sets everything in motion:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Application: {0}", typeof(Program).Assembly.GetName().Version);
        Console.WriteLine("Helper: {0}", typeof(StatusHelper).Assembly.GetName().Version);
        Console.WriteLine("Common: {0}", typeof(StatusEnum).Assembly.GetName().Version);

        var status = StatusHelper.IsActive("Active");
        if (status)
        {
            Console.WriteLine("Status is active");
        }
        else
        {
            Console.WriteLine("Status is Disabled");
        }
    }
}

The first few lines, are only used to display the versions used by each assembly to demonstration only. The remaining logic is pretty simple, so I think there isn’t a need to explain it.

Running the application give us the result we’re expecting.

“Status is active”, and all assemblies are version 1.0. All is working, life is good, etc…

But then…

A new version of the core, changed the enum to:

public enum StatusEnum
{
    Unknown,
    Active,
    Disabled
}

At first sight, it seems a pretty inoffensive change, because we’re using enums and not magic numbers, so everything should be ok. Since the Helper library is a 3rd party, it doesn’t get update at the same rhythm as the core. Let’s try our application again.

“Status is Disabled”. Hum… that’s a bit unexpected…. Luckily, Helper is an open source project, so we can actually include the source on our solution and try to debug the issue. But even so, because it’s a third party, it references the Common by assembly, and the project wasn’t been upgraded yet. This means the project targets the Common v1.0.

Let’s look at the debug

So.. status has been parsed into Active. That is right. Next… The equality status == StatusEnum.Active is evaluated as true, ok.. Next… isActive is assigned with the result of the previous equality, that we already evaluated as true.. false!!!!!! What?!?!!?!

At first we thought this should be a Visual Studio bug, but even after restart the issue remained.

After hours of tries and research…

We finally discovered the issue. As it turns out, Visual Studio was right, as it uses the loaded version of the Common assembly at version 2.0. The Helper library was compiled against the version 1.0 of the Common assembly. The Enum.Parse() uses the run-time version of the Common, the 2.0. but the static part of the comparison StatusEnum.Active is compiled against the version 1.0. Being a constant, the value of that enum is used by the compiler and replaced inline. It doesn’t care about the literal part of the Enum (Enums are mostly ints by default, but you did knew that already, right?).

So basically, the run-time was comparing 1 == 0 and that is obviously false. You can se exacly that, if we cast the values to int like in the image below.

Conclusion

Changing the value of Enums, can be dangerous, especially when they’re used as constants, and let’s be real, everyone uses them as constants. I really don’t remember to see a static readonly enum anywhere (yes, using static readonly would not fall into this issue, because the value would be resolved at run-time other than at compile time.

So next time you need to change an Enum, either, set the values explicitly in order not to change the previous values, or don’t add any value before, just append new ones. Right @Umbraco? No more stuff like this. Let’s keep uneeded changes to a minimum.

You can find the entire code used by this example, and try it yourself here: https://github.com/kappy/BreakingEnum

Living in the edge between business and ux

2015-08-17T00:00:00+00:00

The line that dictates the separation of user experience (ux) and the business logic is not always clear. Remember me talking about multi layers applications? This one I will talk about some code that is near between the business and the presentation layer.

The scenario

Let’s start from the presentation. Here it’s where the ux is implemented. In this case, we’re doing a on/off button, specifically, a toggle button.

So.. How should we implement this… Let’s follow the dummy approach, no logic on the presentation. We have a toggle button, so let’s make a form to submit a toggle action.

Controller (UX):

public ActionResult Toggle()
{
    var statusModel = _myToggleService.GetStatus();
    return View(statusModel);
}

[HttpPost]
public ActionResult Toggle(FormCollection form)
{
    var statusModel = _myToggleService.Toggle();
    return View(statusModel);
}

View (UX):

@using(Html.BeginForm("Toggle")){
    <span>Status: @Model.State</span>
    <input type="submit" value="Toggle" />
}

Service (Business):

public StatusModel GetStatus()
{
    /* Get entity code (or anything else...)*/
    return new StatusModel { Status = entity.Status };
}

public StatusModel Toggle()
{
    /* Get entity code (or anything else...) */
    entity.Status = !entity.Status; //yes, it's a boolean
    /* Save entity (or anything else...) */
    return new StatusModel { Status = entity.Status };
}

Ok.. So the Code works.. We can make Unit tests for this and they’ll all pass. So what is the issue here? Why are we even talking about this?

The web is not a lonely place

You’re not alone on the web. When we design a product to be consumed on the web, intranet, or anyone’s browser, we take the chances of having more than one user/browser/tab open on the same form.

Imagine 2 users wanting to turn off the toggle switch. First one does indeed shut it down, but the next one turns it on again. Worst, the first user got the feedback of the button has been shut down, but the second gets the feedback that he just turn it on. What?

So what’s the solution? Instead of making the presentation dumb, why not make the actual business dumb? I mean, make the business layer explicitly set the desired state. For that, now we need to pass on the desired value through the form. This will result in the following:

Controller (UX):

public ActionResult Toggle()
{
    var statusModel = _myToggleService.GetStatus();
    return View(statusModel);
}

[HttpPost]
public ActionResult Toggle(bool state)
{
    var statusModel = _myToggleService.SetStatus(state);
    return View(statusModel);
}

View (UX):

@using(Html.BeginForm("Toggle")){
    <span>Status: @Model.State</span>
    <input type="hidden" name="state" value="@(!Model.State)" />
    <input type="submit" value="Toggle" />
}

Service (Business):

public StatusModel GetStatus()
{
    /* Get entity code (or anything else...)*/
    return new StatusModel { Status = entity.Status };
}

public StatusModel SetStatus(bool state)
{
    /* Get entity code (or anything else...) */
    entity.Status = state;
    /* Save entity (or anything else...) */
    return new StatusModel { Status = entity.Status };
}

So, everything still works, tests are green, business layer is dumb, stateless, and if a second user (for most unlikely to happen) tries to turn off, even if the first user already turn it off, it will stay off.

Conclusion..

This was a simple scenario, but similar issues happen every day. It’s not easy to separate what is presentation logic and what is in fact business logic. Not every if is business logic. The world is not back and white, there are some grey areas as well.

Entity Framework migrations and branches don't play well together

2015-08-04T00:00:00+00:00

My team and I, decided that was time to use Entity Framework Code first and their migrations. I used FluentMigrator to manage my migrations already a few times , so using the Entity Framework ones, shouldn’t be that hard. At the time of this writing we’re using version 6.1.3.

We were wrong… What we found isn’t pretty.

We use Bitbucket, feature branches and pull requests in our development cycle, and recently we just stumbled into the following scenario: Alice and Bob are developers, each one with a specific task. Bob needs to add a trucks feature, while Alice needs to add a Name into the user profile (of course names and tasks are entirely random and they represent only an example).

Both start they work from a HEAD 4fd9a75. Bob creates branch bob-feature, implements his changes (on this example just creates an entity named Truck), runs the command Add-Migration Trucks and commit the 2c430d4 - Added Trucks feature, containing the migration 201508011250003_Trucks.cs as follow.

public partial class Trucks : DbMigration
{
    public override void Up()
    {
        CreateTable(
            "dbo.Trucks",
            c => new
                {
                    Id = c.Int(nullable: false, identity: true),
                    Name = c.String(),
                })
            .PrimaryKey(t => t.Id);
        
    }
    
    public override void Down()
    {
        DropTable("dbo.Trucks");
    }
}

After the commit he creates the Pull Request for his feature.

Alice does a very similar procedure. Creates branch alice-feature, implements her changes (just adding a simple string Name property on the already existing ApplicationUser class), runs the command Add-Migration UserName and commit the 2659e15 - Added User Name feature containing the migration 201508011256120_User-Name.cs as follow.

public partial class UserName : DbMigration
{
    public override void Up()
    {
        AddColumn("dbo.AspNetUsers", "Name", c => c.String());
    }
    
    public override void Down()
    {
        DropColumn("dbo.AspNetUsers", "Name");
    }
}

Same applies, after the commit, she creates a Pull Request fo everyone to review.

TL TR: this is the repository status so far.

Both branches have a migration, both branches compile and run individually with their own local databases. Bob’s feature gets reviewed and is accepted, so the team merges it into main. Same goes for Alice’s feature.

Both features are done, merged, finished. Until….. Kelly (another developer) tries to run the project from the HEAD on main, and she complains the application can’t run.

Pending changes? but all my changes are there…

Digging into it

Why does this error appear? Microsoft saves a model snapshot with every migration it takes (is the Source field on the migration resource file). Since the last two migrations were created on different branches, they don’t know each other. So, the latest migration snapshot (the one from Alice, but could be the other way around) doesn’t know the migration before already exceuted and inserted its content. The Update-Database checks exactly that. In this particular case, it thinks the Truck entity isn’t created, and tries to add it. It fails because that entity already exists.

Even if you do what the message says, and run Add-Migration againd, it will create a new migration creating the Truck entity again.

This is not a new issue, in fact, it has been like this from the start. There is even a MSDN Article about this behavior and workarounds for it. But the solutions are simply unacceptable. To have manual process steps every time we merge migrations, is a No-Go! Why don’t they singe they’re selves to reproduce all the migrations code and compare it to the actual model, I don’t know. They should only execute the code and nothing else.

The more I use Entity Framework migrations, the more I like FluentMigrator.

Update 2015-08-06 - There is light at the end of the tunnel

Thanks you Jared Dykstra, for pointing me in the direction of this annoucement from one of the developers from EF. On Entity Framework 7 (at the time of writing still in beta6), this is addressed and the snapshot will only be used for reference. Source of the announcement here.

CQS on MVC - Crosscutting Concerns

2015-07-18T00:00:00+00:00

On my last post I vaguely introduced how CQS and a middle sized MVC project was match that fits like a glove.

In this post I will dig a little deeper and I’ll explain how we can add behavior whit the minimal impact.

The problem

I’m building a web application for my wife (source here) that can be multi-user. That means that the information and the permissions might vary by the logged on user.

Since most operations are user contextual, I need to access the logged user, and so, I found myself repeating the following lines of code, depending if I was on a Controller context or not:

Thread.CurrentPrincipal.Identity.GetUserIdAsGuid();
User.Identity.GetUserIdAsGuid();

Usually when I need to repeat something too many times, I always try to find a better way, because eventually, you might need to change it…

Who’s the responsibility?

So, now that we identified the problem, who should set the current user id? The controller should set it on the request message before call the mediator or should we get it from the Thread.CurrentPrincipal inside the Handler?

While the last one is not wrong, doing it like so, would imply that the handler would run on the same executing thread as the web request. I like to keep my handlers the most stateless I can. Remember I said migrating this to a service bus would be a breeze? Keeping your handler stateless, is the first step to achieve that.

Does this mean I opted out for the first choice? Almost… I opted out to pass the current user id on the request message, but like I said before, I don’t want to keep writing the same code, all over on every controller action.

DRY

We connect the handlers that encapsulate our business logic, and the controller with a mediator. That mediator relies on the IoC container to resolve the handler that is responsible to execute the request message. What if we could decorate our handlers with another that inspected the request message and somehow we informed that decorator to inject the current user id into a specific property? Hum… That might do the trick…

I thought also to make this also based on property name conventions, but I decided that it would restrain me in the future.

I started out to create an attribute that I could set up on the property I want.

[AttributeUsage(AttributeTargets.Property)]
public class CurrentUserIdAttribute : Attribute
{
}

And a request message example using it would be:

public class GetUserStatistiscsOverview : IAsyncRequest<IEnumerable<EarningsPerGameViewModel>>
{
    [CurrentUserId]
    public Guid UserId { get; set; }
}

So far so good… But now we need to decorate our handlers with this

public class CurrentUserAsyncHandlerWrapper<TRequest, TResponse> : IAsyncRequestHandler<TRequest, TResponse> where TRequest : IAsyncRequest<TResponse>
{
    private readonly IAsyncRequestHandler<TRequest, TResponse> _innerHander;

    public CurrentUserAsyncHandlerWrapper(IAsyncRequestHandler<TRequest, TResponse> innerHandler)
    {
        _innerHander = innerHandler;
    }

    public async Task<TResponse> Handle(TRequest message)
    {
        var messageType = typeof (TRequest);
        var userProperties = messageType.GetProperties()
            .Where(prop => prop.IsDefined(typeof(CurrentUserIdAttribute), false));
        foreach (var userProperty in userProperties)
        {
            userProperty.SetValue(message, Thread.CurrentPrincipal.Identity.GetUserIdAsGuid());
        }
        return await _innerHander.Handle(message);
    }
}

Yes, we’re using reflection to get the properties of the request message type. It’s not the fastest code ever written, it has room for improvement, but for my requirements, it does the job entirely.

The glue

This is going to deviate from the topic a bit, but putting this all together really depends on your IoC container of choice. In this application I’m using Autofac, and I find it very hard to configure decorators (comparing to other IoCs). Because of that on this project I used a slightly modified version of an helper that I found from James Thurley on this SO answer.

public static class AutofacBuilderExtensions
{
    public static void RegisterHandlers(this ContainerBuilder builder, Type handlerType, params Type[] decorators)
    {
        RegisterHandlers(builder, handlerType);
        for (var i = 0; i < decorators.Length; i++)
        {
            RegisterGenericDecorator(
                builder,
                decorators[i],
                handlerType,
                fromKeyType: i == 0 ? handlerType : decorators[i - 1],
                isTheLast: i == decorators.Length - 1);
        }
    }

    private static void RegisterHandlers(ContainerBuilder builder, Type handlerType)
    {
        builder.RegisterAssemblyTypes(Assembly.GetExecutingAssembly())
            .As(t => t.GetInterfaces()
                    .Where(v => v.IsClosedTypeOf(handlerType))
                    .Select(v => new KeyedService(handlerType.Name, v)))
            .InstancePerRequest();
    }

    private static void RegisterGenericDecorator(ContainerBuilder builder, Type decoratorType, Type decoratedServiceType, Type fromKeyType, bool isTheLast)
    {
        var result = builder.RegisterGenericDecorator(decoratorType, decoratedServiceType, fromKeyType.Name);
        if (!isTheLast)
        {
            result.Keyed(decoratorType.Name, decoratedServiceType);
        }
    }
}

With this in place, It allows me to register my handlers by simply calling:

builder.RegisterHandlers(typeof(IRequestHandler<,>), typeof(CurrentUserHandlerWrapper<,>));
builder.RegisterHandlers(typeof(IAsyncRequestHandler<,>), typeof(CurrentUserAsyncHandlerWrapper<,>));

Final considerations

This is a great way to insert crosscutting behavior without impacting the business logic or code already written. This can also be applyed to insert auditing, logging, cross boundaries transactions, validation, etc..

CQS on middle sized MVC projects

2015-07-10T00:00:00+00:00

In my previous post I’ve introduced it by saying it exists multiple flavors on the Business Layer of multiple layer architecture.

On this post, I want to talk a bit on CQS applied on a MVC project. Also related is the Command Pattern.

The CQS pattern is simple, any operation on the application is either a Read or a Write, but never both. Applying this concept on a MVC application, it means that every interaction with the site can be defined on either queries or commands. A command is any operation that results on changing data (a POST), and query is everything that retrieves data from the server but doesn’t change it (a GET).

So, when a request arrives on the web application, either a command or a query (both are requests) are created, then they get delivered to a Mediator. This mediator based on the current request, will find the proper handler to execute it.

See it in Action

Here’s an example of a controller using CQS:

public class DrawsController : Controller
{
    private readonly IMediator _mediator;
    private const int ItemsPerPage = 10;

    public DrawsController(IMediator mediator)
    {
        _mediator = mediator;
    }

    // GET: Draws
    public async Task<ActionResult> Index(GetDraws query)
    {
        query.ItemsPerPage = ItemsPerPage;
        var results = await _mediator.SendAsync(query);
        return View(results);
    }
    
    /* Other actions below*/
}

In this example, every time we get a request on /Draws/Index we issue a GetDraws query. The Mediator will then lookup an handler that handle the GetDraws request. This is an example of such an handler:

public class GetDrawsHandler : IAsyncRequestHandler<GetDraws,Paged<Draw>>
{
    private readonly ApplicationDbContext _db;

    public GetDrawsHandler(ApplicationDbContext db)
    {
        _db = db;
    }

    public async Task<Paged<Draw>> Handle(GetDraws message)
    {
        var basequery = _db.Draws.Where(d => d.UserId == message.UserId);
        if (message.GameId != null)
        {
            basequery = basequery.Where(d => d.GameId == message.GameId);
        }
        if (message.Date != null)
        {
            basequery = basequery.Where(d => DbFunctions.TruncateTime(d.Date) == message.Date);
        }

        var total = await basequery.CountAsync();
        var draws = await basequery.Include(d => d.Game).Include(d => d.User)
            .OrderByDescending(d => d.Date)
            .Skip((message.Page - 1) * message.ItemsPerPage)
            .Take(message.ItemsPerPage)
            .ToListAsync();

        return new Paged<Draw>
        {
            Items = draws,
            ItemsTotalCount = total,
            ItemsPerPage = message.ItemsPerPage,
            CurrentPage = message.Page
        };
    }

For the command side if the requests, everything is similar, simply in this case, we don’t need the result value. Here’s an example of the /Draws/Edit POST action:

    [HttpPost]
    [ValidateAntiForgeryToken]
    public async Task<ActionResult> Edit([Bind(Include = "Id,Date,Cost,Award")] Draw draw)
    {
        if (ModelState.IsValid)
        {
            await _mediator.SendAsync(new EditDraw { Draw = draw });
            return RedirectToAction("Index");
        }
        return View(draw);
    }

And the handler for EditDraw is like this:

public class EditDrawHandler : IAsyncRequestHandler<EditDraw, Unit>
{
    private readonly ApplicationDbContext _db;

    public EditDrawHandler(ApplicationDbContext db)
    {
        _db = db;
    }

    public async Task<Unit> Handle(EditDraw message)
    {
        var drawEntry = await _db.Draws.FindAsync(message.Draw.Id);
        drawEntry.Date = message.Draw.Date;
        drawEntry.Cost = message.Draw.Cost;
        drawEntry.Award = message.Draw.Award;

        await _db.SaveChangesAsync();
        return Unit.Value;
    }
}

The Unit type is a replacement for void, because we don’t need a return value.

Why Middle sized?

Well, I really think that middle sized applications really shine using this architecture. Applying this architecture on smaller applications, is too much of architecture. For simple and demo applications, I really don’t have nothing against using the ORM or equivalent directly on the controller.

For larger applications, this architecture still shines, but we can evolve this into a more traditional CQRS architecture, with different data-sources for read and writes and using event sourcing for notifications.

Final Considerations

Conceptually, this pattern is very straight forward. Every-time we receive a request, we bind it (using the MVC model binder, or manually) to a request message, and then the mediator delivers it to the corresponding handler that is responsible to execute the request.

This architecture applied, keeps the web project business clean, and with only one dependency, the mediator. This last bullet for me is a big winner. Many time I see multiple dependencies on a controller, that aren’t used on the many actions.

Because this is a message based communication, this is very scalable. Moving from a simple Mediator to a Service Bus, is a breeze. This is also very easy to apply cross-cutting behavior like logging, security or validation, using decorators on the handlers. But I’ll show you a bit more on this on my next post CQS on MVC - Crosscutting Concerns.

A severe case of Servicitis

2015-06-27T00:00:00+00:00

Before we go any further, let me start by stating two important thigs:

Disambiguation from “Cervicitis” or any other disease;
The title choice was only to be catchy, no pun intended there.

Now that we put this out of the way, let’s start the article.

##Introduction A very common 3 (or 4) layer architecture is defined by having a Presentation, Business and Data Access layers.

Particularly in the Business layer, also exist multiple flavors… Managers, Services, Beans, Handlers, etc… All of them have in common:

They should encapsulate the business logic of the application;
They shouldn’t leak implementations details;
They shouldn’t leak data entities.

Blá blá blá… Enough of introductions…

#The Problem

Lets imagine the follow scenario. We have a web application on which the business applications is defined by services. Each entity has a service (similar to Entity Services, which many consider an Anti-pattern, but that concept has a different scope than this (SOA), and clearly out of this post scope).

On to the point already… We have a CustomerService and a ShopService. We need to create a new operation to retrieve the shop the customer as chosen as it’s favorite. Where would you create that operation? Into the CustomerService or ShopService? The truth is, there’s no right answer on this. Both of them are correct. The only catch is, whatever your choice was, it needs to be coherent on the application, and thus, on the team, for further new operations.

In order to get an order for the Customer would you create it on the OrderService or on the CustomerService? I’m almost sure that you’ll chose the OrderService on this case. And the answer is the same as above. Both of them are right, as long as you keep coherence.

##Let’s look into the anatomy of the operation

We need to create a new operation to retrieve the shop the customer as chosen as its favorite

Input: Customer
Output: A shop
Operation: From a list of shops, select the shop the customer has as favorite.

So, what is the most preeminent entity here, shop or a customer? My option is shop, what’s yours?

What I see here is that I’m looking up a shop, and my customer is just a filter on my query. If I would write this down on Sql, this would be my query:

SELECT shop.Id, Shop.Name
FROM shop
JOIN customer ON shop.Id = customer.favoriteShopId

My conclusion? I would definitely put it on the ShopService because that is in fact the Entity I’m looking for. The customer here is just a query parameter. But like I said, I don’t consider the other option wrong, as long as that option is chosen coherently on all the application.

Unit tests beyond Test Driven Development

2015-06-18T00:00:00+00:00

I’m not really a fan of Test Driven Development (TDD). All that write a failing test before the feature, then implement it until the test passes, gives me a feeling of the amount of hours spent on writing code without need.

That said, it doesn’t mean I don’t like Unit tests. Quite the opposite. I like them very much, and I think they’re a very useful tool on your belt.

This is an example of how I sometimes use Unit tests on my day to day.

In one of those pesky summer days that almost resemble winter, for being so cold and rainy, I received a bug report saying that the user wasn’t unable to subscribe his child named “João” that only has 2 months old, in order to start receiving the newsletter with baby stuff promotions. He tried with different browsers, but ended up always on the error page.

Well, say no more… Let’s put it to the test (pun intended).

[TestClass]
public class BugReport_102
{
    /* 
     * User complains that he cannot subscribe his child with name "João" and age 0 
     */

    [TestMethod]
    public void CanInvokeSubscribeWithBugReportData()
    {
        var userController = new UserController();
        var result = userController.Subscribe("joao@email.org", "João", 0) as ViewResult;
        Assert.AreEqual("Subscribe", result.ViewName);
    }
}

Pretty simple test. We just invoke our controller with the request data. For demo only, we don’t need to mock any dependency. That would fall out the scope of this article.

There you go. Bug report replicated successfully. Now that we replicated the problem, it’s easier to pin point the problem. Apparently, someone (the virtual me) forgot to double check the age when performing some rate per age.

Done. Bug fixed. the best thing is that if you keep the test, you’re ensuring that you don’t have regressions on this particular bug.

Some may argue that this is very similar to TDD. I honestly disagree, mainly in the point that I don’t like to write code without knowing if ever will going to be used.

Is TDD a bad thing?

Of course not. It’s a very good way to minimize bug within your application, and so, keeping the quality very high.

It’s only my personal opinion that sometimes feels like we’re writing lots of code, for simple stuff.

One might argue that writing the test before is cheaper than writing it afterwards, and maybe they’re right…

Repository the right way

2015-06-05T00:00:00+00:00

Repository the right way

So.. You just finished creating your super data access layer, a whole lot of repositories on top of Entity Framework, in order to make it testable. All tests are green, everything works, QA goes as expected, and life is good…

Until… You reach production… You start seeing that your queries doesn’t perform well on large datasets, and you start reviewing and debugging your code again… All the indexes are fine on Sql Server; The query performs great when you make the query on Sql Management Studio; You add some sql logging to your DbContext to check it out…

public class HomeController : Controller
{
    public readonly CustomerRepository _repository;
    public readonly StringBuilder _logBuffer;
    public HomeController()
    {
        var context = new RepoDataContext();
        _repository = new CustomerRepository(context);
        _logBuffer = new StringBuilder();
        context.Database.Log = _logBuffer.AppendLine;
    }
    public ActionResult ThisActionDoesntPerformWell()
    {
        _repository.Get(c => c.Name == "Kappy");
        ViewBag.Sql = _logBuffer.ToString();
        return View("Index");
    }
}

You get this output:

Opened connection at 05/06/2015 23:04:11 +01:00
SELECT 
    [Extent1].[Id] AS [Id], 
    [Extent1].[Name] AS [Name]
    FROM [dbo].[Customers] AS [Extent1]
-- Executing at 05/06/2015 23:04:11 +01:00
-- Completed in 48 ms with result: SqlDataReader
Closed connection at 05/06/2015 23:04:11 +01:00

But… How can that be.. The action is selection ALL the records from the Customers table. Your thoughts are like: “But the predicate c => c.Name == "Kappy" is there!! I Know it’s there!!”. You start digging it deeply….

public class CustomerRepository
{
    private readonly RepoDataContext _context;

    public CustomerRepository(RepoDataContext context)
    {
        _context = context;
    }

    public Customer[] Get(Func<Customer, bool> predicate)
    {
        return _context.Customers
            .Where(predicate)
            .ToArray();
    }
}

Everything seems fine, no compiler errors, no compiler warnings… The world is ending, your boss starts to pressure you, because all this is already in Production. The client starts calling your office reporting this issue.

Did you already spotted the error?

You probably did. Yup, it’s the line public Customer[] Get(Func<Customer, bool> predicate) of the CustomerRepository. By replacing that line with public Customer[] Get(Expression<Func<Customer, bool>> predicate) everything works again as expected.

Opened connection at 05/06/2015 23:20:03 +01:00
SELECT 
    [Extent1].[Id] AS [Id], 
    [Extent1].[Name] AS [Name]
    FROM [dbo].[Customers] AS [Extent1]
    WHERE N'Kappy' = [Extent1].[Name]
-- Executing at 05/06/2015 23:20:03 +01:00
-- Completed in 30 ms with result: SqlDataReader
Closed connection at 05/06/2015 23:20:03 +01:00

The Why

There is a big difference between a Func<Customer, bool> and a Expression<Func<Customer, bool>>, despite both can be assigned as c => c.Name == "Kappy", the first one is the same as declaring inline the following function:

bool Predicate(Customer c)
{
    return c.Name == "Kappy";
}

The later, it doesn’t create the function (unless explicitly call .Compile()) but represents an expression graph of all the sentences (more on this later).

Another difference is that, the first case, we’re not invoking the intended Linq extension IQueriable<T> Where(this IQueriable<T> queriable, Expression<Func<Customer, bool>> predicate) but instead we’re invoking this one: IEnumerable<T> Where(this IEnumerable<T> enumerable, Func<Customer, bool> predicate). With that in the clear, it comes very obvious that all the records are being pulled from the database, no matter what query would be there on the predicate.

More on the Expression<>

Like I said before, an Expression<Func<Customer, bool>> is not the same as the Func<Customer, bool>. It doesn’t actually execute lambda c => c.Name == "Kappy", like the Func does. Instead, it creates an expression graph composed of miltiple other smaller expressions that in hole represent the lambda that was specified. Let’s take a look at the composition of our example: In red, you can see the left and right part of the expression, both of them Expressions of themselves. In yellow, you can the type of operation that is being applied to both sides. It’s with this information that EntityFramework, and other ORM’s build the Sql statements for you.

Why I love micro-Orms

2014-10-07T00:00:00+00:00

This is the main reason for me to not use Ado.Net directly anymore. With micro-Orms I can write this legacy code:

    public IEnumerable<User> GetUsersWithInvalidAttempts(int minAttempts)
    {
        using (var conn = CreateConnection())
        {
            var query = "select Id, Name, LastLogin, Active from users where InvalidLoginAttempts >= @minAttempts";
            var command = new SqlCommand(query, conn) {CommandType = CommandType.Text};
            command.Parameters.Add(new SqlParameter("minAttempts", minAttempts));

            using (var reader = command.ExecuteReader())
            {
                while (reader.Read())
                {
                    yield return new User
                                 {
                                     Id = reader.GetInt32(reader.GetOrdinal("Id")),
                                     Name = reader.GetString(reader.GetOrdinal("Name")),
                                     LastLogin = reader.IsDBNull(reader.GetOrdinal("LastLogin")) ? 
                                        (DateTime?)null : 
                                        reader.GetDateTime(reader.GetOrdinal("LastLogin")),
                                     Active = reader.GetBoolean(reader.GetOrdinal("Active"))
                                 };
                }
            }
        }
    }

Into this:

    public IEnumerable<User> GetUsersWithInvalidAttempts(int minAttempts)
    {
        using (var conn = CreateConnection())
        {
            var query = "select Id, Name, LastLogin, Active from users where InvalidLoginAttempts >= @minAttempts";
            return conn.Query<User>(query, new {minAttempts}, commandType: CommandType.Text);
        }
    }

It just works…. In this case I was using Dapper.Net but others are pretty similar.

Just for clarity, this is the class User:

public class User
{
    public int Id { get; set; }
    public string Name { get; set; }
    public DateTime? LastLogin { get; set; }
    public bool Active { get; set; }
}

Repositories! Repo what??

2014-09-18T00:00:00+00:00

Lately, the Repository pattern has become the conversion topic almost every day, mostly because nowadays it’s a so popular pattern, and most people get admired on the fact that I prefer to avoid it, in favor to expose the ORM to the service layer. So I thought about publicly exposing my why, so I can remit here and not repeat myself every time.

For introduction to the Repository, check here. This post will also be an extension from what I’ve already said on my previous post.

Let’s get to the point. My first problem with Repositories is that they hide, and often blocks you the use of advanced features of the actual ORM. Future queries, cascade deletes, batch updates, etc…

I’ve seen two different strategies with repositories. The most radical one, it’s a full closed box, where they implement every query that needs to be issued. They main advantage of this method is that there’s a single place where all queries sit, at the cost of filling each repository with methods for each different type.

public interface ICarRepository : IRepository
{
    Car GetById(int id);
    Car GetBySerialNumber(string serialNumber);
    IEnumerable<Car> GetRedCars();
    IEnumerable<Car> GetRedCarsInPortugal();
    IEnumerable<Car> GetClientOrderCars(int clientId);
    /* And other....  */
}

The other way, is build one generic, fits all repository. This is also the most common example you can find on the internet. I usually find in this way some flaws:

Exposing the Collection-Like as IEnumerable: although this defers the query, this blocks you the ability to make joins with entities, or even perform aggregations and other linq extensions.

Exposing a SaveChanges() or internally call the orm SaveChanges on adding/updating items: It violates your unit of work, and blocks you the ability to save different entities on the same transaction.

  public void CreateAndAddItemToShoppingCart(object newItem)
  {
      var cart = new ShoppingCart();
      _shoppingCartRepository.Add(cart);
      _shoppingCartRepository.SaveChanges();
      newItem.CartId = cart.Id;
      _shoppingCarRowtRepository.Add(newItem);
      _shoppingCarRowtRepository.SaveChanges();
  }
    
  public void AddItemToShoppingCart(int cartId, object newItem)
  {
      var cart = _shoppingCartRepository.GetCart(cartId);
      _shoppingCartRowRepository.Add(cart, newItem);
      cart.Total += newItem.Cost;
      _shoppingCartRepository.UpdateCart(cart);
  }

Just abstracting method by method the Orm: This is just a proxy and it really don’t add anything to the table.

So…. Let’s analyse this the other way around. What’s the reasons I ear this pattern being defended…

I can mock and unit test my code. - Well, you can do that too without Repositories. If your goal is to test your queries, you can mock the Orm.
It add a security layer on my junior developers to protect them from doing something wrong. - Are you sure it’s not the other way around? If your junior developers don’t know what’s on inside that black box, how can you be sure they’re fetching the data right?
I can replace my Orm if I don’t like it. - Sure you can, but if you are doing that, you’ll need to refactor all your repositories, which would take the same cost of changing the Orm itself on the services.
Repositories abstract my data access layer. - Yes, but the Orm too. So you have a double data access layer. Good for you.

Ok, so what do you use then? Queries and Commands! But I’ll explain that a little bit better on my next post.

Wanted dead or alive: Middleware (part III)

2014-05-28T00:00:00+00:00

This is a series of blog posts that focuses on faulty middleware services invocations, and how can we reduce the impact of those faults.

Part I

Part II

Part III (this)

The full source code can be located here

##Improving performance Let’s face it. If we throw a ball to a solid wall, it will ricochet every time. So why bother throw it all again, and again, and again. The same way, most services that outputs data that isn’t changing every second (an employee name, address, status, etc..) can be safely cached. This way we surely are less error prone to our FaultyService and we should gain performance because we’ll use the service less.

##Do we need to change? No we do not. We already made the required changes when we first introduced the interceptor. Nothing keeps us from adding a new piece between and let that piece decide if we need to really invoke the service or we already know the result and output it right away.

To simulate a cache, I created a simple cache provider.

public interface ICacheProvider
{
    bool TryGet(object key, out object value);
    void Set(object key, object value);
}

I also created a PoorMansCacheProvider that only stores the values on an Hastable. Please do note that this cache isn’t production quality. It’s just an academic proof of concept.

Here’s the cache interceptor:

public class CacheInterceptor : IInterceptor
{
    private readonly ICacheProvider _cacheProvider;

    public CacheInterceptor(ICacheProvider cacheProvider)
    {
        _cacheProvider = cacheProvider;
    }

    public void Intercept(IInvocation invocation)
    {
        var arguments = invocation.Request.Arguments;
        var methodName = invocation.Request.Method.Name;
        // create an identifier for the cache key
        var key = methodName + "_" + string.Join("", arguments.Select(a => a ?? ""));  
        object value;
        if (_cacheProvider.TryGet(key, out value))
        {
            invocation.ReturnValue = value;
            return;
        }
        
        invocation.Proceed();

        _cacheProvider.Set(key, invocation.ReturnValue);
    }
}

On it, I’m just generating a key that allows me to differentiate action invocations (even so, in this example we only got 1) and also differentiate invocations by parameters. If the key exists, we return the value we got on store. If not, we proceed the chain of execution and save the result. Could it be simpler?

##Setup Most of the hard work is done, but a still a detail on the setup of our interceptors.

public class Module : NinjectModule
{
    public override void Load()
    {
        Kernel.Bind<StatsCounter>().ToConstant(new StatsCounter());
        Kernel.Bind<ICacheProvider>().ToConstant(new PoorMansCacheProvider());
        var binding = Kernel.Bind<INaiveClient>().To<NaiveClient>();
        binding.Intercept().With<CacheInterceptor>().InOrder(1);
        binding.Intercept().With<RetryInterceptor>().InOrder(2);
    }
}

Note the InOrder extension. That’s the way we setup the execution order of the interceptors.

##Show me the results! We invoked the service 1000 times in 1.6 secods. That is a huge performance increase. As for the resulting numbers: 0.15% error rate. We’re increasing performance and in the process we increased the resilience. By not needing to invoke the service so many times, we did increased the resilience by a significant order of magnitude. Notice the Execution Success: 30. This 30 is the different invocations we have. For the most sharp reader, we’re invoking the service like this:

client.GetMyDate(DateTime.Today.AddDays(i % 30));

This causes the cache to store 30 different values, hence the 30 success invocations. To achieve those 30 success we needed to call the service 45 times. 1 time out of those 1000, we couldn’t get any response on any of the tries before the result was cached.

##Series conclusion I don’t know about you, but this is a really neat way of improving an application performance. The awesome part of this series is that we didn’t have to mess around the client implementation. We extracted it an interface, but that’s about it, and it should be already your way to do things.

Program to interfaces, not implementations

Dependency Injection and Interception are indeed tools that you should have always on your pocket.

The full source code can be located here

Wanted dead or alive: Middleware (part II)

2014-05-28T00:00:00+00:00

This is a series of blog posts that focuses on faulty middleware services invocations, and how can we reduce the impact of those faults.

Part I

Part II (this)

Part III

The full source code can be located here

Try Again, and Again, and Again…

If the error is in fact transient, we can avoid it if we retry the invocation. The other way around, if the erros is caused by missdata that we send erroneously, then retrying won’t do us any good. This is where we distinguish application errors or comunication errors. For simplicity sake we’ll just consider the later one.

Ok, so let’s get the party started. We need to repeate the service invocation if we got any exception. One way to do it, is to code the repeate logic on the client implementation. But what if we get more services? What if we get more actions to call? No… This is not the way to go. What if we could put a transparent piece between the code that invokes the client and the client itself? Well, we actually can. It’s called an interceptor!

To use an interceptor, first let’s call a friend of mine. Dependency Injection. For the sake of lenght of this post, I won’t detail what DI is. Just Google it. Also, for this example, I’ll be using NInject and NInject.Extensions.Interception.LinFu packages for handling DI and Interception, but you can easly switch to the DI container of your own choosing. The important here is how the pieces work togheter, not what glues them.

##Show me the code already!

Ok, ok.. Let’s refactor our NaiveClient, and extract interfaces on the public methods. Then we bind the interfaces and the implementations. Oh wait, did I said interfaces? Right, for the interception work the best, we need to work with interfaces and not the concrete types themselfs. But that is already your common practice, right?

So, here’s the glue:

public class Module : NinjectModule
{
    public override void Load()
    {
        Kernel.Bind<StatsCounter>().ToConstant(new StatsCounter());
        Kernel.Bind<INaiveClient>().To<NaiveClient>().Intercept().With<RetryInterceptor>();
    }
}

What we’re telling here is that INaiveClient is served by the type NaiveClient but has a RetryInterceptor between. Here’s the retry:

public class RetryInterceptor : IInterceptor
{
    private readonly StatsCounter _counter;
    private const int Tries = 3;

    public RetryInterceptor(StatsCounter counter)
    {
        _counter = counter;
    }

    public void Intercept(IInvocation invocation)
    {
        var tryNumber = 0;
        do
        {
            try
            {
                invocation.Proceed();
                return;
            }
            catch (Exception ex)
            {
                tryNumber++;
                if (tryNumber == Tries)
                {
                    throw;
                }
            }
        } while (true); 
    }
}

In simple terms, it’s invocating the invocation.Proceed() until it succeeds or we reach the maximum tries. The invocation.Proceed() is the magic here. From the docs:

Continues the invocation, either by invoking the next interceptor in the chain, or if there are no more interceptors, calling the target method.

We need also change our test program, to use the magic DI.

class Program
{
    const int TimesToInvoke = 1000;

    static void Main(string[] args)
    {
        var kernel = new StandardKernel();
        kernel.Load(new Module());

        var counter = kernel.Get<StatsCounter>();
        var client = kernel.Get<INaiveClient>();
        counter.Stopwatch.Start();
        for (var i = 0; i < TimesToInvoke; i++)
        {
            try
            {
                client.GetMyDate(DateTime.Today.AddDays(i % 30));
                counter.TotalSuccess++;
            }
            catch (Exception ex)
            {
                counter.TotalError++;
            }
        }
        counter.Stopwatch.Stop();

        counter.PrintStats();
        Console.WriteLine("Press any key to exit");
        Console.ReadKey();
    }
}

Take notice that we only work with INaiveClient. We don’t have any direct reference to the RetryInterceptor. The for cycle is the same, no changes what so ever. This time we completed the 1000 invocations in 50 seconds, but we had only 28 failures. That’s 2,8% error rate (again, trust my math). It’s a really nicer number than 32,1% error rate from our last example, and we “didn’t change anything” on our client (only extracted an interface out of it, that should already be a practice). We just inserted a middle piece that extends the default behavior. The higher time can be easily justified by retries that were made. More 417 invocations were made to accomplish those retries. Note also the Execution Fail number is a bit higher. That’s also because we’re doing more invocations.

##Does it get better than that? This is a great improvement. We traded of some performance to gain some resilience. But can we improve this better, and of course, without changing any implementation? Of course we can. On my next post, I’ll add some cache layer, again by using interception, and not changing anything on the client and the retries components.

Part I

Part II (this)

Part III

The full source code can be located here

Wanted dead or alive: Middleware (part I)

2014-05-28T00:00:00+00:00

This is a series of blog posts that focuses on faulty middleware services invocations, and how can we reduce the impact of those faults.

Part I (this)

Part II

Part III

The full source code can be located here

Introduction

Who never invoked a middleware may throw the first stone here. I call it a middleware any piece of software that runs anywhere and anyhow, and was made by any. I don’t need to know how it’s made, just that I need to invoke a service and it gives me an answer, sometimes…

Let’s focus on that sometimes.

Sometimes a new version is being deployed and the service is down.
Sometimes there is a bug and the service blow up.
Sometimes the server is so overloaded, the service times out.
Sometimes we lose network and we can’t reach the service.
Well… there’s probably like 1001 reasons why some service fails giving us a proper response.

Can we do something about it?

Sure we can. Most of the errors are transient. Ok, we failed the invocation, shall we try again? This is the right answer. Let’s try again, up to a number of tries. If the problem was random and/or caused by a spike, invoking the service more than once can improve us some success rate. NOTE: please keep in mind that one of the best practices for distributed systems is to allow message retries without error.

A naive example

Let’s simulate a faulty service:

public class Service
{
    private static readonly Random Rand = new Random();

    public string GetMyDate(DateTime dateTime)
    {
        //Some work
        Thread.Sleep(2);
        
        if (Rand.NextDouble() <= .3)
        {
            throw new Exception("Fault!");
        }
        return string.Format("My date is {0}", dateTime);
    }
}

This fake service will randomly blow up 30% of its invocations. This isn’t perfect, but it suits our needs. Let’s look at the client how will consume this service.

public class NaiveClient
{
    private StatsCounter _counter;

    public NaiveClient(StatsCounter counter)
    {
        _counter = counter;
    }

    public string GetMyDate(DateTime date)
    {
        var faultService = new Service();
        try
        {
            _counter.TotalExecutions++;
            var mydate = faultService.GetMyDate(date);
            _counter.ExecutionSuccess++;
            return mydate;
        }
        catch (Exception)
        {                
            _counter.ExecutionError++;
            throw;
        }
    }
}

We’re passing a StatsCounter object on the constructor just to get some stats on the end of our demo. This client only invokes the services and keeps track of every execution, and which resulted in success or error.

And finally, here is our test program. Let’s simulate 1000 executions. It’s really not a big number, but it’s enough to get us some data.

class Program
{
    const int TimesToInvoke = 1000;

    static void Main(string[] args)
    {
        var counter = new StatsCounter();
        var client = new NaiveClient(counter);
        counter.Stopwatch.Start();
        for (var i = 0; i < TimesToInvoke; i++)
        {
            try
            {
                client.GetMyDate(DateTime.Today.AddDays(i % 30));
                counter.TotalSuccess++;
            }
            catch (Exception ex)
            {
                counter.TotalError++;
            }
        }
        counter.Stopwatch.Stop();
        counter.PrintStats();
        Console.WriteLine("Press any key to exit");
        Console.ReadKey();
    }
}

In approximately 29 seconds we invoked 1000 times the Service.GetMyDate and got 321 errors. It’s like 32,1% errors for those who know simple math (for those who don’t just take my word for it). That was the expected result. Next post let’s dive on how we can improve these results.

Part I (this)

Part II

Part III

The full source code can be located here

IEnumerable Bullying

2014-05-21T00:00:00+00:00

I know I didn’t post anything for quite a while. I’m sorry for that, but I’m having a few events that absorb out my time entirely.

I found this little pearl lying around on a past project that I was called out to save from disaster. Some classes and properties were renamed to protect the business value of the code, but the algorithm remains the same.

It’s so wrong at so many levels I can’t even know where to begin. At least some points are worthy to point out like:

Why flatten the list in the first place? Having 10 properties named the same, doesn’t make sence. What if tomorrow the business scenario changed from 10 questions to 100?
Ok, It really needed to flatten. Shouldn’t Reflection be an option? With refection just add an extra property with the right index and it should be filled with the right value.
Why on earth enumerating the results over and over again? How many times GetEnumerator() and MoveNext() are called? The right way only 1+10 times per item were needed, but this way we have like 1+(10 * n) times per item. If this isn’t abusing the IEnumerable, I don’t know what is.
What really kicked me off my seat was the last statement list.AsQueriable(). Slick….

To log or not to log...

2013-12-15T00:00:00+00:00

A common mistake I’ve seen sometimes is mixing up logging and auditing. Here’s an example:

The first public method, Log_Action, was used to register actions (like logons, logoffs, external services actions, etc…). This is a clear audit related action.

The third public method, Log_Error, was used to register application errors in the database. This is a clear log related action.

The second method, Log_DB, was used to register saves, updates and deletes on every object in the database. This is where the confusion rises. For debugging, having a record of every entity change is great, but the real reason for this action is for auditing. A clear use case for this last action is an entity change history.

Logging to the database is a bloat of information. It costs application resources, when it shouldn’t, and it should be transparent. Logging is mainly a diagnosing tool and it should always be part of the solution, not part of the problem.

Can you see any parameter of type Exception on the Log_Error method? Nop… There isn’t any. The only information we register, is a simple description text…

A little tip: Logging should be transparent. If we remove the logging from the source code, the application should run the same way. Having 358 coded references to the concrete type isn’t probably the best way to do it.

Code with value

2013-12-11T00:00:00+00:00

Not a long time ago, I was called to assist and manage an already ongoing project. This is a small excerpt of a member of one call in the business layer:

Well… Despite the obvious indentation and naming/code style, this small method has a lot of issues.

It shouldn’t dispose objects that we don’t own. This applies to transaction and erm parameters;
Its disposing erm and creating a new instance, but erm parameter isn’t set as ref or out parameter, so the calling code only see the already disposed instance;
Isn’t c# a object oriented programming language? then why so many parameters on the method signature;
Some parameters are only needed for logging. Why do we need to take them for a ride on every call? Infrastructure!!!!!
catch statement is cacthing every exception and soaking the result as null. If we don’t do anything with it (only logging) then why are we catching in the first place? It’s halfway to get a NullReferenceException in the layer above.
The cherry on the top of the cake: we’re returning an IQueriable. Why are we catching exceptions when we’re only returning the query, not the query execution?

Conclusion: there isn’t absolutely no value what so ever in this method. The only thing that it does is to call the layer below. This kind of infrastructure only gets in the way, it doesn’t help the programmer or in fact, anyone.

Pair Programming

2013-11-07T00:00:00+00:00

I really love pair programming. Today I found out that my cats love it too…

N Tier applications - Back then and now

2013-11-03T00:00:00+00:00

N tier applications

N tier applications are a generally adopted design pattern to address the separation of concerns, and application reuse. In a nutshell: every tier has a responsibility, and all tiers are disposed vertically and each one only depends on the tier immediately below.

The number of tiers is variable, but the most used is the 4 tier application:

Presentation (Web, rich client, web service)
Business (validations, transformations)
Data Access (Data i/O)
Data (Database persistence)

Presentation Layer

Since the 2000’s, web applications and web services dominated the world. Let’s face it. It’s easy to install, it’s easy to upgrade, it runs on pretty well know ports and protocols. Today, more and more web application appear. HTTP is here, it works, it’s easy (kinda…). MVC, REST, Web API, Sockets, are they keywords today.

Business

Probably the most important layer on the application. Business rules, data validation and entity manipulation happens here. A few new kids on the block appeared here. DDD (Domain driven design), Rx (Reactive), maybe a few others.

Data Access

Here happened the biggest boom. For years the only way to do data access was ADO.Net. Now, ORMs rise to the occasion. Entity Framework, NHibernate, Linq to Sql, etc… Even micro ORMs (OMs - Object Mappers) have dethroned ADO.Net. It doesn’t make sence any more to use Commands, DataReaders, DataSets…

One common mistake I usually see is to create a Data Access tier using an ORM. What’s the purpose to it? Why have a super framework to create the queries, execute them and get results and relations, database independant, and hide all that inside a box? How can we get the most of it? You can’t really. In my vision, the ORM ‘IS’ the Data Access. Features like linq, hql, criterias, are the glue that ORMs give to the business layer. I’m not saying that business layers should be making the queries and updates to the database, but there are other players that we can use to store and reuse queries.

Data

Well… Databases will be databases. Not much to say here. Only NoSql databases are winning some share here. The world isn’t a set of element relations. It’s more like composition of elements.

My Conslusions

A lot has changed in these years, and we have to adapt. Doing Ado.Net directly nowadays is unthinkable. There are tools and frameworks to help us. We don’t need to reinvent the wheel every project. That isn’t the way to be productive. In the end, it’s one of your major goals.

Code Style - I went to war... and I lost...

2013-09-30T00:00:00+00:00

Code Style - I went to war… and I lost…

My post today will be about indenting style in C#. There are essentially two styles that were broadly adopted:

K&R
Allman

The most notorious difference between the two is where to put the curly braces. K&R opens the curly brace in the same line of the block.

if (started) {
    stop();
} else {
    start();
}

Allman instead, always opens a curly brace in a new line.

if (started)
{
    stop();
}
else
{
    start();
}

As you can see, it’s evident the difference in the number of total lines and most important vertical space.

I went to war…

I’m a clear fan of K&R style. I think it has a clear distinct block structure and doesn’t waste vertical space. Why vertical space you say… A good monitor has a lot of vertical lines. Almost too many to be tracking each and every one of it. Well… That’s true. The problem is, from your company, you almost never get what you want. Especially on computer hardware.

On my actual company, I got the lowest (on lacking of better word) computer. It’s an i5 with 4Gb ram (shared with gpu), and a stunning display with 1366x768 resolution. Visual Studio takes about 1 full minute to load (without any solution). It’s clearly not a developer machine.

That said, I still find important not to waste space. Even well written production software it’s not written on 5 or 6 lines of code. If you still waste lines (an else block uses 3 lines…) screen space falls off quite fast.

And I lost…

A long time ago, I had an idea to use github to analyze a few open source projects to check the trend of indent styles used. As pointed out by Phill Haack, a recent project did just that.

The numbers speak for themselves.

Allman style: 88%
K&R style: 11%
Other 1%

As it turns out, vertical space is a thing of the past. At least for someone… not me tho.

Typed vs Untyped languages

2013-08-26T00:00:00+00:00

Today my thoughts are for typed versus untyped languages. That said, I know that untyped languages has many advantages, but I really hate to be able to set potatoes to onions. Look at this sample code on Javascript:

    
    var onion = { name: 'onion', numberOfLayers: 5 };
    document.write("<p>" + onion.name + "</p>");
    
    onion = { name: 'potato', isFried: false };
    document.write("<p>" + onion.name + "</p>");
    
    onion = 0;
    document.write("<p>" + onion + "</p>");

Live Feedle

This is a perfectly valid code. The most obvious matter on this type of assignments is that, we’re assigning completely different things to the same variable witch causes the variable to lose all its meaning. Meaningful variables are the number one rule thumb on my book of trades. This is also one of the most common issues on code that I review.

This is why I love typed languages:

Can you see that thin red line? That’s the IDE telling you: You’re doing it wrong!

Markdown on Visual Studio 2013

2013-07-31T00:00:00+00:00

Today I got a nice surprise. I was updating a class on one of my OSS projects and I opened the README.md file to write the version history. I had just changed the README to README.md to take advantage of the Markdown styling.

This was my surprise: Markdown rendering inside Visual Studio. Why was I editing markdown all this time on Notepad++? Why!?!?!?!

At first I thought this was a VS feature. Nop, It isn’t. This feature is from Web Essentials plugin.

Anyway, this little surprise just made my day. (That and a new issue posted by someone with interest in the ar-SA locale on my library. This only shows it’s beeing used by someone. I really love when new Issues arrive.).

Dynamic Type Discovery

2013-07-22T00:00:00+00:00

On one of my open source projects (check you my github profile) I use some dynamic type discovery and loading. I need these because I want to load every type that implements an interface or has a specified attribute. I also need to be able to load types that reference my asssembly (not known to my assembly).

I started by enumerating all types on the appdomain:

var types = AppDomain.CurrentDomain.GetAssemblies()
    .ToList()
    .SelectMany(a => a.GetTypes());

I released my library using this approach. I worked. nice…

Few days after, I received 2 new issues: You library is blowing out with my application.

[ReflectionTypeLoadException: Unable to load one or more of the requested types.     Retrieve the LoaderExceptions property for more information.]
System.Reflection.RuntimeModule.GetTypes(RuntimeModule module) +0
System.Reflection.RuntimeModule.GetTypes() +4
System.Reflection.Assembly.GetTypes() +78

Hum… ok.. fine, But it really works on my machine… Let’s digg a bit more about ReflectionTypeLoadException shall we. ReflectionTypeLoadException Class Reference.

There is also another array of exceptions (LoaderExceptions property). This exception array represents the exceptions that were thrown by the class loader. The holes in the class array line up with the exceptions.

Hum…. yup, the problem wasn’t on my library but on some missing dependencies of the application, but because the library tries to enumerate all the types on the AppDomain, it burst in flames.

So, what can we do? Not much really, just work with the good eggs and ignore the others.

private static Type[] GetTypesFromAssemblySafe(Assembly assembly) {
    try {
        return assembly.GetTypes();
    } catch {
        return new Type[] {};
    }
}

var types = AppDomain.CurrentDomain.GetAssemblies()
    .ToList()
    .SelectMany(GetTypesFromAssemblySafe);

Bottom line… Even if your code is working nice, tested and feels good, on a disfunctional context, it can be a bomb ready to explode.

Code Metrics Alarm

2013-07-14T00:00:00+00:00

When you’re building any kind of software development project, there are some red flags that you should be aware of. One of them is the number of logical lines of code (Source lines of code) on a single class.

This is one of those cases: 3k LLoC on a single class is way too much…

Lets dig a little bit more into this class. Lets build a simple project to analyse the class stats.

    class Program
    {
        static void Main(string[] args)
        {
            var type = typeof (ConfigurationDBReadHandler);
            Console.WriteLine("Stats of {0}", type.Name);
            Console.WriteLine("Number of properties {0}", type.GetProperties().Count());
            Console.WriteLine("Number of methods {0}", type.GetMethods().Count());
            Console.WriteLine("Number of fields {0}", type.GetFields().Count());
            Console.ReadLine();
        }
    }

Here’s the output: Stats of ConfigurationDBReadHandler Number of properties 3 Number of methods 160 Number of fields 0

Without starting picking on the name of the class, the reasons behind those 160 methods it’s because this was supposed to be a data access class. A single data access class for the entire project. All entities were loaded and saved from this class. It was madness to find a single entity related methods. A direct violation of SRP.

Unit Of Work

2013-07-03T00:00:00+00:00

This time I want to indroduce my approach on the Unit Of Work pattern (Unit of Work). If you’re reading this and thinking EF or Nhibernate, then don’t waste your time. DbContext and ISession are already implementations of Unit of Work.

I use this aproach mostly with Micro ORM frameworks, like Dapper, Simple.Data, etc… Are you still doing Ado.Net directly? In what year do you live in?

    public class UnitOfWork : IDisposable
    {
        private IDbConnection connection;
        private IDbTransaction transaction;

        public UnitOfWork(string connectionString, bool requireTransaction = true)
        {
            connection = GetConnection(connectionString);
            connection.Open();
            if (requireTransaction)
            {
                BeginTransaction();
            }
        }

        public void BeginTransaction()
        {
            if (transaction != null)
            {
                return;
            }
            transaction = connection.BeginTransaction();
        }

        public void CommitChanges()
        {
            if (transaction != null)
            {
                transaction.Commit();
            }
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected void Dispose(bool disposing)
        {
            if (disposing)
            {
                if (transaction != null)
                {
                    transaction.Dispose();
                    transaction = null;
                }
                if (connection != null)
                {
                    connection.Dispose();
                    connection = null;
                }
            }
        }

        ~UnitOfWork()
        {
            Dispose(false);
        }
    }

This is the base code. Now we have a few different options:

Add public properties of Connection and Transaction and let the outside world to manage them.
Add protected properties of Connection and Transaction and derive the class to a data access specific class.
Add to it the public interface of your favorite micro Orm library.

For option #1, I really don’t like it. It most certainly will lead to micro managing connection and transactions… For option #2 and #3, it’s really up to you. You can even mix and match them.

I use option #3. Why? because is simpler, and I mostly just use Dapper on my projects. It’s a matter of personal preference.

This is what I end up with:

    public class UnitOfWork : IDisposable
    {
        private IDbConnection connection;
        private IDbTransaction transaction;

        public UnitOfWork(string connectionString, bool requireTransaction = true)
        {
            connection = GetConnection(connectionString);
            connection.Open();
            if (requireTransaction)
            {
                BeginTransaction();
            }
        }

        public void BeginTransaction()
        {
            if (transaction != null)
            {
                return;
            }
            transaction = connection.BeginTransaction();
        }

        public void CommitChanges()
        {
            if (transaction != null)
            {
                transaction.Commit();
            }
        }

        public int Execute(string query, object param = null, CommandType commandType = CommandType.Text)
        {
            return connection.Execute(query, param, transaction, commandType: commandType);
        }

        public IEnumerable<T> Query<T>(string query, object param = null, CommandType commandType = CommandType.Text)
        {
            return connection.Query<T>(query, param, transaction, commandType: commandType);
        }

        public IEnumerable<dynamic> Query(string query, object param = null, CommandType commandType = CommandType.Text)
        {
            return connection.Query(query, param, transaction, commandType: commandType);
        }

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected void Dispose(bool disposing)
        {
            if (disposing)
            {
                if (transaction != null)
                {
                    transaction.Dispose();
                    transaction = null;
                }
                if (connection != null)
                {
                    connection.Dispose();
                    connection = null;
                }
            }
        }

        ~UnitOfWork()
        {
            Dispose(false);
        }
    }

If True Else

2013-07-02T00:00:00+00:00

If True Else….

This was an old project where I has asked to contribute. It was a mixed of c# and vb.net. The most suprisingly piece of code I saw was this:

You might wander why… I know I do. The reason behind it was simple. Far far away there was a variable named whatever. Some day, after a change request, it was removed, so instead on refactoring code, ctrl.F - Replace whatever to True - Replace All….

What do you get? a bunch of rubish…