Wednesday, July 9, 2014

npm install ENOENT errors

I've been working on a project that is built using NodeJS. Our development environment is a bit of a hybrid. Most of the developers are developing on Macs, with one exception. Our build server is on Windows and our site is running in Windows Azure. For the most part developing NodeJS on a Mac is quite straightforward. All the tooling is readily available and tends to work quite well. Not so much on Windows. So when our build server starting to run into issues installing the requestify package, we chalked it up to something not liking Windows.

We were getting ENOENT errors where npm was complaining that it couldn't find files that it was supposed to download on the file system. Since the error was happening installing a dependency of a dependency, we thought maybe it was a long paths issue. However, we couldn't even get it to install running in the root directory. We decided to use a work around to switch to a different version of that particular dependency.

Weeks later I ran into the same issue, but this time on my Mac. I found a solution by running npm cache clean before running npm install again. I tried it on the Windows build server and it started working again.

TL;DR;
If you're seeing ENOENT errors when running npm install try running npm cache clean and try again.

Monday, December 23, 2013

Thursday, November 14, 2013

Simple TypeMock Auto-Mocking container

TDD is a great methodology for simplifying your code and focusing on delivering business value. When doing TDD, we always want to strive to make it as friction-free as possible. This means making writing tests and, more importantly, refactoring as painless as we can. One of the ways we can do that is to use an Auto-Mocking container.

A very common reason for tests to change is because we added a new dependency to our SUT and we have to modify all of our previous tests to use this new dependency. One way to avoid this is to use an object that will build your SUT and inject all appropriate dependencies as mocked objects. So your tests go from looking like this:

dependency1 = new Mock<IDependency1>();
dependency2 = new Mock<IDependency2>();
sut = new MyClass(dependency1, dependency2);

to this:

mocker = new Automocker();
sut = mocker.CreateSut<MyClass>();
dependency1 = mocker.GetMock<IDependency1>();
dependency2 = mocker.GetMock<IDependency2>();

We can now add another dependency without having to modify this test (assuming that this dependency is not used in this test).

Most of the popular mocking frameworks that are in use today (Moq, Rhino.Mocks, NSubstitute, etc) all have 1 or more Auto-Mocking frameworks that can be downloaded via NuGet. On our current project, we are using TypeMock, which is a very powerful, commercial grade, mocking tool. It can mock out almost anything and is used quite extensively in the SharePoint development world. There is no Auto-Mocking container for it, so I decided to write one.

An Auto-Mocking container needs to be able to do 2 things. Generate a SUT object with all dependencies mocked out and be able to retrieve the dependencies that were injected. TypeMock uses the following method to generate a mock object

Isolate.Fake.Instance<IDependency>();

Since it uses a static generic method, there is no way to call this method with a Type object without using reflection. So we will use reflection to call this method. We can get the MethodInfo object for this method in the following way:

TypeMockInstance = Isolate.Fake.GetType()
        .GetMethods()
        .Where(m => m.Name == "Instance")
        .Select(m => new
             {
              Method = m,
              Params = m.GetParameters(),
              Args = m.GetGenericArguments()
             })
        .Where(x => x.Params.Length == 0
           && x.Args.Length == 1)
        .Select(x => x.Method)
        .First();

and we would use it like so:

TypeMockInstance.MakeGenericMethod(type).Invoke(Isolate.Fake, null);

When creating our SUT we need to use reflection to get the types of the constructor parameters and use the method above to create them. In order to retrieve them at a later point we need to create a registry, which will simply be a Dictionary<Type, object>; In the case of multiple constructors we will pick the largest constructor. Here's what our CreateSut method looks like:

public T CreateSut<T>()
{
 var type = typeof (T);
 var constructors = type.GetConstructors(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance);
 //get largest constructor
 ConstructorInfo largestConstructor = null;
 foreach (var constructor in constructors)
 {
  if (largestConstructor == null ||
   largestConstructor.GetParameters().Length < constructor.GetParameters().Length)
  {
   largestConstructor = constructor;
  }
 }
 //generate mocks and build up parameter list
 var parameters = new List<object>();
 foreach (var parameterInfo in largestConstructor.GetParameters())
 {
  if (!_typeRegistry.ContainsKey(parameterInfo.ParameterType))
  {
   _typeRegistry[parameterInfo.ParameterType] = GetMockInstance(parameterInfo.ParameterType);
  }
  parameters.Add(_typeRegistry[parameterInfo.ParameterType]);
 }
 return (T) largestConstructor.Invoke(parameters.ToArray());
}

To get our generated mock objects we can grab them from our dictionary after our SUT has been created. The full source for the Automocker can be found on Github.

Tuesday, June 18, 2013

Implementing an external service lookup cache

We have a form in our web application that takes an address. The field is an auto-complete field and as the user types their address, the form shows them matching addresses. There is a web service that is used in this organization to help with address lookup. This service takes a string (partial address) and returns a list of addresses that start with that particular string, up to a maximum of 1000 addresses. I wanted to implement a cache of this look up and it turned out to be an interesting little problem. How do we cache the addresses and more importantly, how can we figure out if we should hit the server or the cache?

An initial thought was that we could store all addresses from the service in a HashSet to avoid duplicates. We could check if there are any items in the cache that start with the search string and if so, we can return those items. Here's what this lookup class might look like.

public class AddressLookup
{
 public static AddressLookup Instance { get; private set; }

 private ISet<AddressType> Addresses
 {
  get
  {
   lock (this)
   {
    if (HttpRuntime.Cache["Addresses"] == null)
    {
     HttpRuntime.Cache["Addresses"] = new HashSet<AddressType>();
    }
    return (ISet<AddressType>)HttpRuntime.Cache["Addresses"];

   }
  }
 }

 static AddressLookup()
 {
  Instance = new AddressLookup();
 }

 public IEnumerable<AddressType> GetByPartialAddress(string searchAddress)
 {
  lock (this)
  {
   searchAddress = searchAddress.ToUpper();
   var cachedAddresses = Addresses.Where(address => address.FormattedAddress.StartsWith(searchAddress, StringComparison.InvariantCultureIgnoreCase)).ToArray();
   if (cachedAddresses.Length > 0)
   {
    return cachedAddresses;
   }

   var addresses = new AddressManager().RetrieveByPartialAddress(searchAddress);
   Addresses.UnionWith(addresses);
   return addresses;
  }
 }
}

This idea works, but consider a worst case performance issue of traversing a large collection of values, not finding the value and then making a call to the web service after all. Not to mention the issue of having an incomplete cache of values due to the web service only returning the top 1000 results. For an illustration of the issue consider a web service that only returns the top 2 results.

Total data set1, 12, 122
Uncached results for search string "1"1, 12
Uncached results for search string "12"12, 122
Cached results for search string "12" after searching string "1"12

To solve this issue I decided in addition to caching the address list, I would also cache the searches. This will fix both issues, the first by improving cache check performance, the second by returning a better result set from the cache. Below is the implementation:

public class AddressLookup
{
 public static AddressLookup Instance { get; private set; }

 private ISet<string> Searches
 {
  get
  {
   lock (this)
   {
    if (HttpRuntime.Cache["AddressSearches"] == null)
    {
     HttpRuntime.Cache["AddressSearches"] = new HashSet<string>();
    }
    return (ISet<string>)HttpRuntime.Cache["AddressSearches"];
   }
  }
 }

 private ISet<AddressType> Addresses
 {
  get
  {
   lock (this)
   {
    if (HttpRuntime.Cache["Addresses"] == null)
    {
     HttpRuntime.Cache["Addresses"] = new HashSet<AddressType>();
    }
    return (ISet<AddressType>)HttpRuntime.Cache["Addresses"];

   }
  }
 }

 static AddressLookup()
 {
  Instance = new AddressLookup();
 }

 public IEnumerable<AddressType> GetByPartialAddress(string searchAddress)
 {
  lock (this)
  {
   searchAddress = searchAddress.ToUpper();
   if (Searches.Contains(searchAddress))
   {
    return Addresses.Where(address => address.FormattedAddress.StartsWith(searchAddress, StringComparison.InvariantCultureIgnoreCase));
   }

   Searches.Add(searchAddress);
   var addresses = new AddressManager().RetrieveByPartialAddress(searchAddress);
   Addresses.UnionWith(addresses);
   return addresses;
  }
 }
}

The next potential for improvement would be to implement a rolling cache. One that only keeps the n most recent searches. I haven't taken this implementation that far, but a potential implementation might be to associate the search results on a per search basis in a dictionary, rather than keeping a global address list. Let me know how you might implement a rolling cache or if you have any improvement suggestions with my implementation.

Thursday, February 28, 2013

How I feel as a developer

Here's a blog with a bunch of animated gifs illustrating what it's like to be a developer.

Monday, October 1, 2012

Node.js, the next web development platform?

Node.js has been gaining steam over the last little while, thanks to videos like this, showing how Node.js can handle 1000 concurrent requests. The development community has grown by leaps and bounds and new packages are constantly being added to NPM. In addition to this, Microsoft has thrown its weight behind it and has provided support for Node.js development in Windows Azure and their WebMatrix IDE. The selling features sound pretty good compared to existing frameworks and architectures:
  • Better memory efficiency under high loads compared to traditional thread based concurrency model
  • No locks and as a result no dead locks
  • HTTP is a first class protocol, this means it's a good foundation for web libraries and frameworks
  • Developers can write code in one language on the client and server
In order to jump right in and start learning Node.js I decided to write an adapter for the jugglingdb ORM. This is an ORM which is bundled in the very Ruby on Rails like framework Railway.js. The adapter was written to integrate into Windows Azure Table Storage and is available in github and on npm.

From this experience I've learned several things, the most important of which is that Node.js is not ready for primetime. While it's great for doing simple endpoints, complex applications seem to be very hard to troubleshoot and debug. This is due to the use of async callbacks all over the place. While this allows high concurrency, debugging becomes a muddled mess. Here is an example of the getTable function in the TableService class, provided by Microsoft.

TableService.prototype.getTable = function (table, optionsOrCallback, callback) {
  var options = null;
  if (typeof optionsOrCallback === 'function' && !callback) {
    callback = optionsOrCallback;
  } else {
    options = optionsOrCallback;
  }

  validateTableName(table);
  validateCallback(callback);

  var webResource = WebResource.get("Tables('" + table + "')");
  webResource.addOptionalHeader(HeaderConstants.CONTENT_TYPE, 'application/atom+xml;charset="utf-8"');
  webResource.addOptionalHeader(HeaderConstants.CONTENT_LENGTH, 0);

  var processResponseCallback = function (responseObject, next) {
    responseObject.tableResponse = null;
    if (!responseObject.error) {
      responseObject.tableResponse = TableResult.parse(responseObject.response.body);
    }

    var finalCallback = function (returnObject) {
      callback(returnObject.error, returnObject.tableResponse, returnObject.response);
    };

    next(responseObject, finalCallback);
  };

  this._performRequestExtended(webResource, null, options, processResponseCallback);
};

Not only is this hard to read, it's not clear why all of these layers of callbacks are even necessary. This could just be this one library, however, it also seems that Node.js is solving a problem that isn't there for a vast majority of people. The Railways.js and Express frameworks are great, however, if the problem you are trying to solve is concurrency, you are unlikely to use a heavyweight framework. You are much more likely to build a highly specialized and tuned end point, using low level platform constructs.

That is not to say that it's a lost cause. Projects like TypeScript are seeking to make Node.js development much more scalable. Tools like Jetbrains WebStorm with the Node.js plugin, provide a slew of  functionality for JavaScript development that you may be used to in other development environments. This includes refactoring, code completion, a nodeunit test runner, and debugging

Would I use Node.js on my next project? Potentially for small endpoints that may need high concurrency and throughput. However, I am interested in seeing where the development community takes the concept of JavaScript on the server. Perhaps at some point this will become a viable alternative to the existing platforms and frameworks.

Friday, April 27, 2012

Using Sqlite to test database interaction with NHibernate

In a previous post, Stubbing Out NHibernate's ISession, I talked about creating a mock class in order to stub out QueryOver interaction in your classes. In this post I'd like to talk about another approach to take: using an in-memory Sqlite database to simulate real data interaction. The upside to using an in-memory Sqlite database, is that you can quickly test to make sure if your query will actually work as you expect it to work. There are a couple of downsides, however. One is that you're using Sqlite as opposed to whatever production grade database you will be using in deployment. The second, is that set-ups for your tests can get pretty complex, as you may potentially need to persist a lot of entities.

The first thing you'll need to do, is to download Sqlite and the appropriate .NET wrapper. These can be found here and here respectively. Next you'll need to create some sort NHibernate session manager. This class should be able to create an in-memory Sqlite database, create the schema, and open an ISession. Here's  what mine looks like

public class NHibernateSqliteSessionFactory
{
 private static readonly NHibernateSqliteSessionFactory _instance = new NHibernateSqliteSessionFactory();
 private static Configuration _configuration;
 private readonly ISessionFactory _sessionFactory;

 private NHibernateSqliteSessionFactory()
 {
  var model = AutoMap.AssemblyOf<Entity>(new GasOverBitumenConfiguration())
   .UseOverridesFromAssemblyOf<NHibernateSessionFactory>()
   .Conventions.Add(
    DefaultLazy.Always(),
    DynamicUpdate.AlwaysTrue(),
    DynamicInsert.AlwaysTrue())
   .Conventions.AddFromAssemblyOf<Entity>();

  FluentConfiguration fluentConfig = Fluently.Configure()
   .Database(
    new SQLiteConfiguration()
     .InMemory()
     .UseReflectionOptimizer()
   )
   .ProxyFactoryFactory("NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle")
   .Mappings(m =>
        {
         m.AutoMappings.Add(model);
         m.FluentMappings.AddFromAssemblyOf<Entity>();
        })
   .ExposeConfiguration(c => _configuration = c);

  _sessionFactory = fluentConfig.BuildSessionFactory();
 }

 public static NHibernateSqliteSessionFactory Instance
 {
  get { return _instance; }
 }

 public ISession OpenSession()
 {
  var session = _sessionFactory.OpenSession();
  new SchemaExport(_configuration).Execute(false, true, false, session.Connection, new StringWriter());
  session.BeginTransaction();
  return session;
 }
}

Note that we are using FluentNhibernate to create the configuration. This can be just as easily accomplished with either traditional or loquacious configuration. Furthermore the OpenSession method creates a session and sets up the database schema. When the session is closed, the in-memory database is gone.

It might be a good idea to also persist several entities in your database, such that your unit tests are not required to have large set up methods. Your test can then simply open the session, do what is being tested and verify that the result is correct. In your clean up step, it's a good idea to dispose the session.

There you have it. An easy way to set up an in-memory Sqlite database for NHibernate.