Shrinkwrap NPM dependencies

[jorbin]

We have our dependencies pinned pretty well, however our dependencies dependencies sometimes change in such a way that our code is impacted. #30736 highlighted this. In order to prevent this and similar problems in the future, we should use ​npm shrinkwrap to lock down the exact versions that we want to have installed.

[jorbin]

One potential pitfall is that this could have is that we could get out of date pretty quickly. I think what we should do is have a tracking ticket every release for updating the dependencies. We can then close this out during (or at the start of ) beta. I'm open to other ideas though to help ensure we don't end up too out of date.

[swissspidy]

I think what we should do is have a tracking ticket every release for updating the dependencies. We can then close this out during (or at the start of ) beta.

+1 for going this route.

[ocean90]

We had the same issue during 4.3 again, @nacin fixed it by copying an old node_modules to the build server I think.
As soon as we branch we should add a shrinkwrap file to the branch.

[nacin]

We have two opportunities to shrinkwrap a branch. One, as the branch is created with a copy. (No.) Two, after the fact — but might that means the build will have run, which will result in a mess.

I've again copied over the node_modules from trunk to branches/4.4 on the build server. The next commit should clean things up. I can also update the build script to copy over node_modules whenever a branch is created.

I'd like us to shrinkwrap branches for sure. Can we enforce that through a unit test somehow?

I don't love the idea of maintaining a shrinkwrapped file in trunk, as it'd probably get pretty tedious. On the other hand, I don't love that our builds are only reproducible on a single build server.

[nacin]

I tried to run npm shrinkwrap on 4.4 so I could upload the json here and so it could be committed to the branch. It errored out. Here's npm-debug.log:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'shrinkwrap' ]
2 info using npm@1.3.24
3 info using node@v0.10.25
4 verbose node symlink /usr/local/bin/node
5 error Error: Problems were encountered
5 error Please correct and try again.
5 error extraneous: autoprefixer-core@5.2.1 develop.svn.wordpress.org/branches/4.4/node_modules/autoprefixer-core
5 error     at shrinkwrap_ (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/shrinkwrap.js:29:15)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/shrinkwrap.js:23:5
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/ls.js:43:30
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:119:5
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
6 error If you need help, you may report this *entire* log,
6 error including the npm and node versions, at:
6 error     <http://github.com/isaacs/npm/issues>
7 error System Linux 3.2.0-4-amd64
8 error command "/usr/local/bin/node" "/usr/local/bin/npm" "shrinkwrap"
9 error cwd /home/svn/checkouts/develop.svn.wordpress.org/branches/4.4
10 error node -v v0.10.25
11 error npm -v 1.3.24
12 verbose exit [ 1, true ]

[nacin]

Based on [33129], I removed node_modules/autoprefixer-core. Now I'm seeing this on npm install:

# npm install
npm WARN package.json WordPress@4.4.1 No README data
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/meow/node_modules/read-pkg-up/node_modules/find-up requires path-exists@'^2.0.0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/path-exists,
npm WARN unmet dependency which is version 1.0.0
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/meow/node_modules/read-pkg-up/node_modules/read-pkg requires normalize-package-data@'^2.3.2' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/unique-stream/node_modules/through2-filter/node_modules/through2 requires xtend@'~4.0.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/imagemin-gifsicle/node_modules/through2 requires readable-stream@'>=1.0.33-1 <1.1.0-0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.4
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/imagemin-optipng/node_modules/through2 requires readable-stream@'>=1.0.33-1 <1.1.0-0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.4

It returns exit code 0, which is why the build continues to work, but shrinkwrap understandably fails.

I've uploaded the build server's node_modules here for debugging.

[nacin]

Check that, uploaded to Cloudup instead: ​https://cloudup.com/c8vxc1C4PY0.

[netweb]

@nacin Backporting r35859 via #35003 to the 4.4 build branch _should_ fix up those errors:

Also we should be using at least NodeJS v0.12.x as PostCSS requires this v0.12.x or greater, and WordPress includes two node modules that use PostCSS, Autoprefixer and RTLCSS.

I'd be tempted to actually recommend that we bump up our minimum supported NodeJS version to v4.x LTS per the ​LTS announcment, this brings back compatibility of IO.JS and NodeJS branches and gives us a somewhat more dependable NodeJS base that module owners are aware of for maintaining backcompat for this NodeJS version.

[netweb]

Related: #35105 Specifiy minimum node and npm version in package.json

[jorbin]

I was able to shrinkwrap the dev dependencies on 4.4 without an issue. @nacin - do you want me to commit that to the 4.4 branch? And then we can commit trunk's to trunk as well.

[jorbin]

When I attempt to build with the modules that @nacin uploaded above, it gives me an error. I have 0.12.6 installed on this box.

Loading "imagemin.js" tasks...ERROR
>> Error: Cannot find module 'clone'
Loading "sass.js" tasks...ERROR
>> Error: The `libsass` binding was not found in /home/jorbin/wp/node_modules/grunt-sass/node_modules/node-sass/vendor/linux-ia32-14/binding.node
>> This usually happens because your node version has changed.
>> Run `npm rebuild node-sass` to build the binding for your current node version.

[nacin]

On a fresh npm install, the output of grunt build diffs clean on ​https://core.svn.wordpress.org/branches/4.4. This means that, for the moment, we won't generate churn by doing a fresh npm install and committing the resulting npm shrinkwrap.

@jorbin, want to commit a shrinkwrap file to branches/4.4 directly? Per our discussion, we should leave it out of trunk for now.

[nacin]

Okay, I've generated the following npm-shrinkwrap.json using the build server. Also note the build server currently runs node 0.10.25 and npm 1.3.24. Both can change upon request.

[ocean90]

Tested npm-shrinkwrap.json​ with node v5.3.0 and npm 3.3.12. npm install ran without any issues. Same for grunt build.

When comparing the build files of wp-admin and wp-includes with current core.svn I have one changed file: wp-admin/js/customize-controls.min.js. Diff: 30787.diff.png​
But I get this for trunk too, so it's unrelated to the shrinkwrap file.

[jorbin]

In 36165:

Add npm-shrinkwrap.json for 4.4

By shrinkwraping our dependencies, the same versions of everything will be installed no matter what rules the dependency package.json has specificed.

See #30787
props nacin, ocean90, netweb

[jorbin]

@nacin Can you confirm that the build server is able to do it's thing?

[netweb]

Replying to nacin:

Okay, I've generated the following npm-shrinkwrap.json using the build server. Also note the build server currently runs node 0.10.25 and npm 1.3.24. Both can change upon request.

Upgrading the build server to use NodeJS v0.12.x should happen at some time per comment:9, I believe per the original ​upstream issue only a notice will be shown, functionality shouldn't be affected.

Not that I saw the above notice in my tests with NodeJS v0.10.33 I did get a warning for the grunt-sass dependencies:

• grunt-sass -> node-sass -> request -> hoek
• WARN engine hoek@2.16.3: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.33","npm":"3.5.1"})

Revisiting the upgrading of NodeJS toward the end of the 4.5 cycle when the npm-shrinkwrap.json will be updated sounds like a plan when some more in depth testing/analysis can happen as currently functionality does not appear to be an issue despite the NodeJS version warning notices ¯\_(ツ)_/¯

[jorbin]

In 37186:

Add npm-shrinkwrap.json to 4.5

By shrinkwraping our dependencies, the same versions of everything will be installed no matter what rules the dependency package.json has specificed.

See #30787.
Fixes #36498.