August 23, 2016
Everyday, we find ourselves facing more and more projects with more and more diverse technology stacks. So how do we make sure everyone has the right versions of the right tools for the right project?
The obvious first step is to manage as many dependencies on the project level as possible. And while we’ve moved all dependency management into Virtual Machines for many environments, this effort is still ongoing for Visual Studio builds. So we decided to figure out how many of the dependencies we’d usually install globally could be managed through Visual Studio 2015. The goal was to be able to get the code, open the solution and build successfully including code style checking and automated tests for both the C# code as well as for the front end code. There were two key things we had to do to achieve this:
Let’s take a look at the details of how we did it:
Npm is a package manager for Node.js modules which we use to manage the dependencies of our front end build tools and actual project code. It’s one layer where we can manage a lot of otherwise global dependencies locally per project. Most tutorials and examples suggest that Node.js and npm are installed globally, but as we just saw, that doesn't work well in an environment with dozens of different projects. We also use npm as a task runner, so our code style checking, unit testing and build commands are all initialized through npm. An example would be to execute npm test to start the unit tests.
Visual Studio 2015 comes with its own versions of Node.js and npm, but they are quite outdated - at least in VS2015 Update 2. So we decided to install them with NuGet. NuGet is the package manager for the Microsoft development platform including .NET. Built into Visual Studio, it allows us to track dependencies per project in a solution.
To make sure that we’re using the locally installed version when we run the npm command, we have to use the alias Visual Studio adds in the project's .bin folder. So at the top of any script we run from the project folder, we temporarily add the .bin folder to the PATH environment variable:
The local version of Node.js and npm will then take precedence over any globally installed version. A small batch script that can run any npm command you'd like locally would look like this:
REM Forward any arguments to npm
CMD /C npm %*
If this script is called npm-local.cmd, you would now run npm-local test instead of npm test.
We also use NuGet to install other dependencies like StyleCop for C# code.
As developers, we want to spend as little time as possible on project setup. Minimizing global dependencies can help a lot with this. The entire process becomes more predictable and less error-prone. And the more time we can avoid a long list of manual setup steps, the more time we can spend solving real problems.
The second part of ensuring Visual Studio builds our front end is to use the predefined BeforeBuild build target. You'll find this at the end of a default .csproj file.
This can be set up individually per project in a solution. We’d recommend setting up a project just for the front end so a build doesn't slow down a backend developer's builds or vice versa.
We set up a separate batch file to run all the build steps and referenced it in the build target. As far as we can tell, there’s no difference running this in the BeforeBuild or AfterBuild target when the project only contains front end code. We create a NuGet package from the front end build output and use that in other projects within the solution.
Our BeforeBuild build target looks something like this:
<Exec WorkingDirectory="$(ProjectDir)" Command="post-build.cmd $(Configuration)" ConsoleToMSBuild="true">
<Output TaskParameter="ConsoleOutput" PropertyName="OutputOfExec" />
It boils down to this: Run
from the project directory and make sure that the output is reported back to Visual Studio.
We pass along the Configuration so we can run different commands for debugging compared to release builds. You can find a full list of macros available on MSDN.
Important note: If you change front end code outside of Visual Studio, because you prefer a text editor like Sublime Text or Visual Studio Code to do your front end development, you’ll have to run a Rebuild on the project, because Visual Studio might not detect any changes on Build and won't trigger the build events.
Another issue we came across when we switched to a local instance of npm was that our Continuous Integration server had no global cache for it. Npm can take quite a while to install dependencies, so the npm cache saves some of that - but only when there is a global or at least shared instance of the cache.
The first thing we considered was reconfiguring npm, but even with a shared cache, it didn’t give us the performance we wanted. Others have recommended zipping up the node_modules folder containing all the installed dependencies after the build and then retrieving/extracting it again before the next build, but that took almost as long as installing from scratch, because it contains A LOT of files. Then we looked into symlinking the folder, but on Windows you need admin rights to create a symlink.
In the end, we came to the conclusion that the path of least resistance and the best performance was to simply move the entire folder to a project specific temporary (e.g. /tmp/project_name/node_modules) folder on the CI server after each build and retrieving it at the beginning of the next. Then we just run npm prune and npm install to make sure we don't have unnecessary or missing dependencies and we're good to go. Now every build spends about 15 seconds on ensuring that we have the right node modules instead of minutes.
Another thing that can save time is to disable the progress option of npm by running npm set progress=false before your build.
When we have a big project and a group of developers pushes commits constantly, it's important that our builds are fast so we can iterate quickly. Before we deploy, we need to be able to fix the last bugs and test on our test environment as quickly as possible.
We’re busy further encapsulating our projects with tools like Vagrant and Docker and automating as many things as we can so we can spend less time on setup and waiting for builds and focus on building great experiences for users.
Posted in continuous integration, web development, node.js