Presence Insights JavaScript tooling: From GruntJS to homebrew

Datetime:2016-08-23 02:45:16          Topic: Grunt           Share

On Presence Insights , I was responsible for the development workflow of a team of developers and managing their deliveries in the form of nearly 30 microservices and 30 npm modules. This involvement ranged from managing branches in GitHub to building out automation software for a deployment pipeline.  One of the biggest aspects of my job was finding, or creating, the best tools to get the job done for my team and ensuring that they were all usable by anyone that was on the team or could join the team in the future.

Presence Insights is built on a Node.js microservice architecture that consists of over 60 repositories, a mix of services, and private node modules. Finding tooling that made working with all of these individual projects in a composite way was a challenge that my team fought for a period of time, and these are some of the lessons we learned.

GruntJS

GruntJS is a task runner that one member of the team had some experience with. The small team prototyping Presence Insights jumped in and got to work, quickly creating a project that allowed cloning and issuing a set of commands against the small but growing collection of Presence Insights repositories. However, we lacked a clear understanding of the complications and requirements of a workspace supporting a variety of individual projects that need to both act on their own as well as be managed in unison.

Over time, needs for the workspace arose, and as developers with varying knowledge levels of the Grunt patterns and “gotchas” supplied those demands, the Gruntfile grew into a mix between poorly implemented tasks used for workspace manipulation and poorly implemented tools actually used for building the services. The Gruntfile was powered by an auxillary collection of home grown Grunt tasks and various automation scripts that ended up being bolted on to the workspace due to convenience in the early stages when speed was important and we had more to do than experience in the area. This contributed to a tight coupling between builds and the workspace, on top of the ever growing and unwieldy library attached to our Gruntfile . The file itself grew to over 650 lines and was complex in its own right while mostly being a front for a larger set of more complex and more convoluted libraries.

Aside from maintainability issues, an annoyance brought on by Grunt was that almost all of our tooling that we had written ourselves was in the form of Grunt tasks, so none of them were of any value independently. While installing grunt-cli isn’t a huge pain, adding a Gruntfile to a project where you want to do something trivial that you’ve automated, such as package a simple JavaScript project into an artifact, is an annoyance and contributes to the noise in the root of these projects.

We additionally felt the Grunt way of doing things was bloated and not straightforward. Grunt’s idea behind not duplicating code is by using various configuration objects, but these objects tend to end up duplicating themselves anyway:

batch_git_clone: {
  servicesSoft: {
    options: {
      repos: SERVICES,
      overWrite: false
    }
  },
  servicesHard: {
    options: {
      repos: SERVICES,
      overWrite: true
    }
  },
  libsSoft: {
    options: {
      repos: LIBS,
      overWrite: false
    }
  },
  libsHard: {
    options: {
      repos: LIBS,
      overWrite: true
    }
  },
  project: {
    options: {
      repos: SERVICES,
      overWrite: false,
      project: '<%= grunt.task.current.args[0] %>',
      branch: '<%= grunt.task.current.args[2] %>'
    }
  }
}

The options are trivial: either we would like to clone and overwrite or don’t overwrite. Through Grunt, this becomes two different configuration objects for each type of project we would like to clone. Collecting argument parameters in an attempt to make some of this configuration intelligent (for instance, specifying hard or soft on the command line and only having one configuration object) is not intuitive and argument parsing being used inside of the configuration objects requires templates.

Overall, the amount of value in GruntJS that was Grunt-specific knowledge combined with factors such as the mix between logic being in both the Gruntfile and in other additional tasks/scripts helped us make the decision that GruntJS was not the right tool for Presence Insights.

On the hunt for a new tool, we considered our requirements and what we didn’t like about GruntJS.

Needs:

  • Be able to script repeatable tasks that can be operated against a single or collection of JavaScript repositories. This can range from updating package.json and running git status to packaging a service and publishing a collection of related libraries.
  • Simple build dependency that does not require cloning a bulky workspace project for each build or automation task. GruntJS was a global Node CLI on top of installing 15+ additional node modules and packaging our custom grunt tasks.
  • Simplicity in design and implementation such that anyone with Node experience would be reasonably comfortable jumping in.

We looked at some open source options, but there had already been a rumbling brewing for the entire last quarter of 2015, one that claimed we already had the one-size-fits-all build tool that we all needed…

npm

Inspired by reading a plethora of articles about how to use npm as a build tool , it wasn’t long before we examined it seriously in the wake of dropping Grunt. The added bonus was that the development team was at least passingly familiar with npm as a script runner.

What was grunt setup and grunt pull became npm run clone and npm run pull , in addition to those commands taking traditional style argument flags such as --force and --branch , for example. Behind these npm run scripts were simple shelljs and commander Node.js CLIs. shelljs handled the heavy lifting of running all of the commands to build and manage the workspace, while commander provided an easy to understand framework for contributors and abstracting away argument parsing.

Here is an example package.json from this time period.

The relevant bits:

"scripts": {
  "lint": "eslint .",
  "force-clone": "npm run clone-libs -- -F; npm run clone-services -- -F; npm run clone-tools -- -F",
  "clone": "npm run clone-libs; npm run clone-services; npm run clone-tools",
  "pull": "npm run pull-libs; npm run pull-services; npm run pull-tools",
  "pull-libs": "gitall pull -f libs.yml",
  "pull-services": "gitall pull -f services.yml",
  "pull-tools": "gitall pull -f tools.yml",
  "clone-libs": "gitall clone -f libs.yml",
  "clone-services": "gitall clone -f services.yml",
  "clone-tools": "gitall clone -f tools.yml",
  "checkout-libs": "gitall checkout -f libs.yml -b",
  "checkout-services": "gitall checkout -f services.yml -b",
  "checkout-tools": "gitall checkout -f tools.yml -b",
  "start": "server start -r routes.yml",
  "link": "aperture link",
  "bulk": "aperture bulk --",
  "link-services": "bash scripts/link-all.sh services",
  "link-libs": "bash scripts/link-all.sh libs",
  "link-tools": "bash scripts/link-all.sh tools",
  "postinstall": "git submodule update --init --recursive"
},

The argument passing to npm (via -- ) gave some members of the development team trouble at first, but everyone was fast adopters. The string passed is simply tacked on to the script called, so more complex bash parameter handling is not possible. This made it impossible to create composite commands such as npm run clone for things like npm run checkout because there was no way to handle the argument given and pass it to all three of npm run checkout-tools , npm run checkout-libs , and npm run checkout-services .

For the time being, some scripts were still in the workspace project itself (link-*) and some were full blown private node modules that exported CLIs (gitall and server) instead of being Grunt tasks. This meant we could now install any of those tools globally and run them in any project or against any project without needing to add additional configuration and JavaScript files to the project.

The components of the legacy Gruntfile that were related to builds had been torn out and provided through a separate node module, @pidev/package-service, which assuaged the coupling problems between the workspace and the services that had been getting worse with time. During this time, coverage and other commands were added to the npm run scripts of the individual project package.json s.

As new functionality was required and libraries were split up, npm run travis-publish joined npm run package-service as a command that was required in each service and library. The script used to run the publishing command was also put into a private node module @pidev/publish. Private node modules allowed us to manage an ecosystem of over 60 repositories on an individual basis uniformly via npm run scripts because each project could simply install that tool as a devDependency. If for some reason a project was not ready or needed to be supported by a legacy routine, locking into an old version would be all that’s required.

Comparatively, npm was a much better fit for Presence Insights than Grunt. We still had issues with it, as our scripts ballooned in number, most of which simply being aliases for a CLI we had written. To add to the awkwardness, putting a legitimate CLI with multiple commands behind npm was not a good user experience. Consider that to get the help for any CLI, you’d be typing npm run cli -- help instead of just cli help . Where at one point, we were gaining clarity and efficiency because using npm run scripts enforced a consistent interface for interacting with any of our npm projects, we ended up with so many that any time we wanted to do anything, we were checking what the exact syntax was of one of our 15 scripts and typing much more than necessary.

Because the tools we had been writing to run through npm were already mostly full blown CLIs in their own right, we saw less and less of a point over time in putting some of them behind npm.

pi and other home-rolled CLIs

“pi” was a Node CLI written using commander + shelljs + inquirer + chalk that was designed to be the multipurpose workspace tool that we had been hunting for.The rich feature set of community projects such as chalk and inquirer made it possible for us to quickly and easily spit out high quality CLIs with features like stdout highlighting and prompts without having to learn an opinionated framework. Individual CLIs that had no ties to other programs such as npm or Grunt for their functionality allowed Presence Insights to finally have truly reusable, robust, more easily maintainable tools.

After initializing for the first time with your GHE token and specifying a workspace, pi would save that information for future requests against Github Enterprise or other workspace scoped commands. When pi initialized a workspace, it crawled the Presence Insights GHE org looking for projects that fell into patterns defined by a .yaml file included in pi. The .yaml file simply mapped repository prefix slugs to project types. If a repository started with piservice- it was cloned into the services folder in the workspace and the piservice- was removed, and so on for pidev- and pilibs-. Otherwise, the repository was ignored and not cloned into the workspace.

pi removed the idea of a workspace project, because the workspace it creates and manages is never persisted in source control and is completely dynamic based on the contents of the GHE org. Previously, we had defined all of the projects in the workspace repository, and that had caused our on-boarding process for new services or libraries to be complex and error-prone. With self discovery through naming patterns, all a developer had to do was name their repository correctly and they would immediately join the workspace ecosystem.

After having used both GruntJS and npm-as-a-build-solution for more than three months at a time, we can safely say that we prefer rolling our scripts into reusable CLIs and relying on npm as a task runner when it makes the most sense.

Learn more about these tools and IBM





About List