My week of 5/26 at npm

This week, I’m at @jsconf all week. I may or may not get anything else done, but I’m not investing in a checklist this week.

Last week, I:

  • Did npm’s release, which thank’s to @othiym23‘s excellent documentation went almost flawlessly.
  • Fix a bunch more tests in #7774 (the npm@3/multi-stage branch test fixing ticket). I also found an error in rolling back optional dependency failures in dependency configurations. (Eg, deps that are both optional to one dep and non-optional to another, especially when more layers of indirection are involved.)

My week of 5/18 at npm

This week…

  1. #7774 npm@3 tests, I really want to get a lot of them done.
  2. If things go as planned, I’ll also be doing the release this week.

Last week I…

  • #8055 #8052 Documented a new module, a git repo manager which will be used in resolving these tickets. It’s shelved for the time being while I push on getting npm@3 out the door.
  • #7774 Got about a third of the tests passing
  • Interviewed four candidates for a position as a coworker on the CLI team. Everyone was amazing, and if we had the resources we’d hire them all. We’re doing additional interviews with the wider npm team this week, but eeee, we’re close to having more folks on our team!
  • I talked at the Boston Ember.js meetup (pics, video) about what’s new in npm@3. This was a repeat from a talk I gave earlier at a local Node.js meetup, but with a few updates.

My week of 5/11 at npm

I’m going back to finishing tests in the multi-stage branch, but I have one carry-over I want to get done. As always, this’ll be updated as we go over on the npm roadmap.

  1. #7774 Getting tests passing for multi-stage and npm@3
  2. #8055 #8052 Isolating git support in npm and making update and outdated checking work well– I started this last week and will be making a separate github repo for it shortly. It will likely be shelved after Monday, while I work on tests in multi-stage.
  3. Pluuus we’re going to start interviewing CLI candidates…

Last week I…

  • Got married!

  • Started work on #8055 #8052 Isolating git support in npm and making update and outdated checking work well

My week of 5/4 at npm

This week, I’m away for most of it getting married. Or rather, I got married on Sunday and now I’m recovering. 😉

Last week I worked on:

  1. I’m working through the various high priority git issues– specifically:
    1. #7202 Race conditions w/ multiple git deps pointing at the same repo but with different committishes. Fixed and test in progress.
    2. #7994 Upon reflection, was actually behaving as intended.
  2. Meeting with @stefanpenner to discuss ember-cli coordination

The week of 4/27 at npm

This week, Forrest and I are swapping our focuses. He’s gonna dig into the multi-stage test suite and I’m gonna dig into outstanding git issues and handling pull requests. This is a short week for me– I’m not around on Friday, busy getting married and all. 😉

  1. I’m working through the various high priority git issues– I’ve not dug in detail yet so I’m not sure how many we’ll get fixed this week. But some of them.
  2. Meeting with @stefanpenner to discuss ember-cli coordination
  3. Manage pull requests, which basically involves reviewing and getting pull requests into a shape we can accept them. Plus the purely mechanical act of merging them. 😜

Last week, again involved a bunch of off-the-roadmap work:

  1. #7774 (partial) Got node test/run.js working in multi-stage
  2. #6942 Write action validator that checks permissions.
  3. 2b51fce Eliminate stray progress bar display cruft
  4. 844e72a Rewrote shrinkwrap to not use npm ls. And other shrinkwrap improvements:
    1. 999be86 Don’t install missing modules from the shrinkwrap when explicitly installing modules (eg npm install foo)
    2. 19e9d6b @thefourtheye gave us a fix to get fixing test/run.js working, which meant making the target directory if it doesn’t exist when installing globally.
    3. 8369df9 Update fetch-package-metadata’s tarball streamer, which is used to read shrinkwraps, to support ungzipped tars and plain JS files, just like cache/tar.
  5. Fixes to read-package-tree’s link marking:
    1. e3d3721 Folders should only be marked as links if they’re symlinked deeper than the top of the tree.
    2. 38207e7 The path attribute we put on nodes should be based on the path passed in to read-package-tree, not off it’s actual symlink-free name.

The week of 4/20 at npm

As usual, this is posted and updated live in the npm roadmap wiki.

  1. #7774 Getting tests passing for multi-stage and npm@3 (ongoing)
  2. #6932 Make npm install --link work
  3. #6942 Write action validator that checks permissions.

These are all carry overs from last week- and really the first needs breaking out further, at least for the purposes of todo lists. Last week was busy– I was in Oakland visiting the mother ship, and I got a bunch of things done including a slew not originally planned:

  1. #5090 npm install doesn’t update changed git url dependencies.
  2. #6926 Reorder installation steps such that if A depends on B, we don’t run A’s lifecycles till B’s have run first.
  3. #6934 Implemented linked directory semantics.
  4. Make global installs/uninstalls not waste time looking at other packages in the global tree
  5. Fix outstanding dedupe bugs
  6. #5693 Not installing deps of deps from shrinkwrap
  7. #5779 Make npm-shrinkwrap idempotent. This means both deep sorting AND consistent _from fields.
  8. Fix outstanding shrinkwrap bugs. Specifically: A) warnings during shrinkwraps left progress bar cruft on screen and B) npm install modulename when you had a shrinkwrap would remove any dev deps (and any other deps not in the shrinkwrap).

Open Work Stuff

So, working on open source full time means working in open, but it can still be remarkably difficult to create public visibility.

We’re now maintaining our current task lists for npm publicly on the roadmap wiki page. We refresh the lists each week on Monday and update them throughout the week as we complete things and/or add new things to our lists. You can see mine here.

Publish-Only Shrinkwrap

Edit: It’s worth pointing out that freezing versions in libraries is not generally recommended (though this is a matter of community contention). Further publishing anything to the public registry with a shrinkwrap causes problems for people with private registry mirrors as the shrinkwrap encodes the public registry url. Obviously, if your organization has decided to freeze its libraries then publishing shrinkwraps to your own private registry like npme or as private modules is quite alright.

Some people also feel that shrinkwraps MUST be checked into git, so that you can recreate published archives from the git repository. Some do not. Obviously you need to fit things to your workflow and your release practices.

npm’s shrinkwrap feature provides a key benefit when it comes to supporting published code– it fixes all of the dependency versions for all of your dependencies AND THEIR dependencies. This means that you can be sure that your end users are using exactly the same software as you are.

But shrinkwrap can also be frustrating to use because as long as the npm-shrinkwrap.json file is laying around npm won’t behave the same as it usually does.

What this does, is only include the npm-shrinkwrap.json in the published artifact. This means that your development will get newer versions of your dependencies to test, while your users will get exactly what you tested with at the time you published.

How To

Dev diary…

I kinda fell off the wagon there when it comes to dev diaries, but let’s see if we can’t get this going again. Since last time, I gave up trying to extend npmlog indirectly– its heavy use of bind and the way that ansi integrates led to this. I spent more time fiddling with it as I’d had an incorrect model of bind. Somewhere I got the idea that you could rebind, but no, once bound a function will only ever be called with that object, no matter what you do to it. It turns out that these two really are semantically equivalent (though the bind version is MUCH slower to define AND to call):



Is semantically equivalent to:

So yeah, I ended up integrating my progress bar changes directly into npmlog. It’s currently in the version in the multi-stage branch. I do plan to pull that out into its own module to handle rendering, at which point I’ll look again and see if it can use any of the existing progress bar modules. What I did do was factor completion tracking out entirely. Your application interacts with the completion tracker which feeds completion data to your progress bar. This is currently living in [an unreleased module], which’ll see release before the npmlog updates see release. I’m quite happy about having factored completion tracking separately from the progress bar, as unusual as this might be. We actually have a lot of different kinds of things we need to count toward the overall install completion, from discrete tasks to downloads.

I also learned that node-gyp executes during the install lifecycle step, which does make sense as it replaced explicitly putting things in that step. As a result of this, and also just generally thinking about things that could reasonably happen in this step, I’ve moved it and postinstall till after finalize, so they now run in the final destination, not in staging.

I also learned that calls to node-gyp (and probably other lifecycle calls) go to stdout and bypass the log system entirely. This should really be fed into the logging system so that it can be suppressed or not and it can not muck up the progress bar.

I got distracted yak shaving and wrote [documentation for my bespoke dotfiles].

Wednesday/Thursday Oct-29/30 Dev Diary

These days kind of ran together due to a weird schedule, involving in part sleeping off the programming binge from earlier. Like most days it started off with support, emails and finding orphaned slack results…

  • I rewrote my commit history be be more sane, since people are starting to make noises about building on my branch. Having people build on my branch is … awkward because rewriting history is pretty core to how I work– I often make commits of experiments, then throw them out and replace them, then later flatten all that down. But I can make it work. The main thing I had to change was moving install to msinstall and restoring the old install. This makes my branch usable as real npm while we’re developing.
  • I started looking at failing tests now… which given that the replacement install command is temporarily named something different should be none of them. =D
  • Decided to spend some time with the progress bar…
    • I tried a few approaches to integrating with the logger and after a few false starts, settled on wrapping it, which thus far seems to be working ok. I’m not totally happy with how it factors into existing code, but I don’t yet see another option that’d integrate cleanly. =/
    • Still, I do actually have a percentage meter now, so plugging it into an actual progress bar should be pretty easy. There are still some integration issues with logging (due to its “I’m a giant magic global object” thing =D) This all took longer then I’d’ve liked.

Whatever fills my mind…