WordPress.org

Make WordPress Core

Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#30787 closed enhancement (fixed)

Shrinkwrap NPM dependencies

Reported by: jorbin Owned by: nacin
Milestone: 4.4.1 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: 2nd-opinion commit
Focuses: Cc:
PR Number:

Description

We have our dependencies pinned pretty well, however our dependencies dependencies sometimes change in such a way that our code is impacted. #30736 highlighted this. In order to prevent this and similar problems in the future, we should use npm shrinkwrap to lock down the exact versions that we want to have installed.

Attachments (2)

npm-shrinkwrap.json (1.1 MB) - added by nacin 4 years ago.
30787.diff.png (28.0 KB) - added by ocean90 4 years ago.

Download all attachments as: .zip

Change History (26)

#1 @jorbin
5 years ago

  • Keywords 2nd-opinion added

One potential pitfall is that this could have is that we could get out of date pretty quickly. I think what we should do is have a tracking ticket every release for updating the dependencies. We can then close this out during (or at the start of ) beta. I'm open to other ideas though to help ensure we don't end up too out of date.

This ticket was mentioned in Slack in #core by ocean90. View the logs.


5 years ago

#3 @swissspidy
4 years ago

  • Keywords close added

I think what we should do is have a tracking ticket every release for updating the dependencies. We can then close this out during (or at the start of ) beta.

+1 for going this route.

#4 @ocean90
4 years ago

  • Keywords close removed
  • Milestone changed from Awaiting Review to 4.4.1

We had the same issue during 4.3 again, @nacin fixed it by copying an old node_modules to the build server I think.
As soon as we branch we should add a shrinkwrap file to the branch.

#5 @nacin
4 years ago

We have two opportunities to shrinkwrap a branch. One, as the branch is created with a copy. (No.) Two, after the fact — but might that means the build will have run, which will result in a mess.

I've again copied over the node_modules from trunk to branches/4.4 on the build server. The next commit should clean things up. I can also update the build script to copy over node_modules whenever a branch is created.

I'd like us to shrinkwrap branches for sure. Can we enforce that through a unit test somehow?

I don't love the idea of maintaining a shrinkwrapped file in trunk, as it'd probably get pretty tedious. On the other hand, I don't love that our builds are only reproducible on a single build server.

#6 @nacin
4 years ago

I tried to run npm shrinkwrap on 4.4 so I could upload the json here and so it could be committed to the branch. It errored out. Here's npm-debug.log:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'shrinkwrap' ]
2 info using npm@1.3.24
3 info using node@v0.10.25
4 verbose node symlink /usr/local/bin/node
5 error Error: Problems were encountered
5 error Please correct and try again.
5 error extraneous: autoprefixer-core@5.2.1 develop.svn.wordpress.org/branches/4.4/node_modules/autoprefixer-core
5 error     at shrinkwrap_ (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/shrinkwrap.js:29:15)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/shrinkwrap.js:23:5
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/lib/ls.js:43:30
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:119:5
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
5 error     at /usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/read-installed/read-installed.js:240:14
5 error     at cb (/usr/local/node-v0.10.25-linux-x64/lib/node_modules/npm/node_modules/slide/lib/async-map.js:48:11)
6 error If you need help, you may report this *entire* log,
6 error including the npm and node versions, at:
6 error     <http://github.com/isaacs/npm/issues>
7 error System Linux 3.2.0-4-amd64
8 error command "/usr/local/bin/node" "/usr/local/bin/npm" "shrinkwrap"
9 error cwd /home/svn/checkouts/develop.svn.wordpress.org/branches/4.4
10 error node -v v0.10.25
11 error npm -v 1.3.24
12 verbose exit [ 1, true ]

#7 @nacin
4 years ago

Based on [33129], I removed node_modules/autoprefixer-core. Now I'm seeing this on npm install:

# npm install
npm WARN package.json WordPress@4.4.1 No README data
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/meow/node_modules/read-pkg-up/node_modules/find-up requires path-exists@'^2.0.0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/path-exists,
npm WARN unmet dependency which is version 1.0.0
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/meow/node_modules/read-pkg-up/node_modules/read-pkg requires normalize-package-data@'^2.3.2' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/vinyl-fs/node_modules/glob-stream/node_modules/unique-stream/node_modules/through2-filter/node_modules/through2 requires xtend@'~4.0.0' but will load
npm WARN unmet dependency undefined,
npm WARN unmet dependency which is version undefined
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/imagemin-gifsicle/node_modules/through2 requires readable-stream@'>=1.0.33-1 <1.1.0-0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.4
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/imagemin-optipng/node_modules/through2 requires readable-stream@'>=1.0.33-1 <1.1.0-0' but will load
npm WARN unmet dependency develop.svn.wordpress.org/branches/4.4/node_modules/grunt-contrib-imagemin/node_modules/imagemin/node_modules/readable-stream,
npm WARN unmet dependency which is version 2.0.4

It returns exit code 0, which is why the build continues to work, but shrinkwrap understandably fails.

I've uploaded the build server's node_modules here for debugging.

#8 @nacin
4 years ago

Check that, uploaded to Cloudup instead: https://cloudup.com/c8vxc1C4PY0.

#9 follow-up: @netweb
4 years ago

@nacin Backporting r35859 via #35003 to the 4.4 build branch _should_ fix up those errors:

Also we should be using at least NodeJS v0.12.x as PostCSS requires this v0.12.x or greater, and WordPress includes two node modules that use PostCSS, Autoprefixer and RTLCSS.

I'd be tempted to actually recommend that we bump up our minimum supported NodeJS version to v4.x LTS per the LTS announcment, this brings back compatibility of IO.JS and NodeJS branches and gives us a somewhat more dependable NodeJS base that module owners are aware of for maintaining backcompat for this NodeJS version.

#10 in reply to: ↑ 9 @netweb
4 years ago

Related: #35105 Specifiy minimum node and npm version in package.json

#11 @jorbin
4 years ago

I was able to shrinkwrap the dev dependencies on 4.4 without an issue. @nacin - do you want me to commit that to the 4.4 branch? And then we can commit trunk's to trunk as well.

#12 @jorbin
4 years ago

  • Owner set to nacin
  • Status changed from new to assigned

This ticket was mentioned in Slack in #core by jorbin. View the logs.


4 years ago

#14 @jorbin
4 years ago

When I attempt to build with the modules that @nacin uploaded above, it gives me an error. I have 0.12.6 installed on this box.

Loading "imagemin.js" tasks...ERROR
>> Error: Cannot find module 'clone'
Loading "sass.js" tasks...ERROR
>> Error: The `libsass` binding was not found in /home/jorbin/wp/node_modules/grunt-sass/node_modules/node-sass/vendor/linux-ia32-14/binding.node
>> This usually happens because your node version has changed.
>> Run `npm rebuild node-sass` to build the binding for your current node version.

#15 @nacin
4 years ago

On a fresh npm install, the output of grunt build diffs clean on https://core.svn.wordpress.org/branches/4.4. This means that, for the moment, we won't generate churn by doing a fresh npm install and committing the resulting npm shrinkwrap.

@jorbin, want to commit a shrinkwrap file to branches/4.4 directly? Per our discussion, we should leave it out of trunk for now.

#16 @nacin
4 years ago

  • Owner changed from nacin to jorbin

@nacin
4 years ago

#17 follow-up: @nacin
4 years ago

Okay, I've generated the following npm-shrinkwrap.json using the build server. Also note the build server currently runs node 0.10.25 and npm 1.3.24. Both can change upon request.

@ocean90
4 years ago

#18 @ocean90
4 years ago

  • Keywords commit added

Tested npm-shrinkwrap.json with node v5.3.0 and npm 3.3.12. npm install ran without any issues. Same for grunt build.

When comparing the build files of wp-admin and wp-includes with current core.svn I have one changed file: wp-admin/js/customize-controls.min.js. Diff: 30787.diff.png
But I get this for trunk too, so it's unrelated to the shrinkwrap file.

#19 @jorbin
4 years ago

In 36165:

Add npm-shrinkwrap.json for 4.4

By shrinkwraping our dependencies, the same versions of everything will be installed no matter what rules the dependency package.json has specificed.

See #30787
props nacin, ocean90, netweb

#20 @jorbin
4 years ago

  • Owner changed from jorbin to nacin

@nacin Can you confirm that the build server is able to do it's thing?

#21 in reply to: ↑ 17 @netweb
4 years ago

Replying to nacin:

Okay, I've generated the following npm-shrinkwrap.json using the build server. Also note the build server currently runs node 0.10.25 and npm 1.3.24. Both can change upon request.

Upgrading the build server to use NodeJS v0.12.x should happen at some time per comment:9, I believe per the original upstream issue only a notice will be shown, functionality shouldn't be affected.

Not that I saw the above notice in my tests with NodeJS v0.10.33 I did get a warning for the grunt-sass dependencies:

  • grunt-sass -> node-sass -> request -> hoek
  • WARN engine hoek@2.16.3: wanted: {"node":">=0.10.40"} (current: {"node":"0.10.33","npm":"3.5.1"})

Revisiting the upgrading of NodeJS toward the end of the 4.5 cycle when the npm-shrinkwrap.json will be updated sounds like a plan when some more in depth testing/analysis can happen as currently functionality does not appear to be an issue despite the NodeJS version warning notices ¯\_(ツ)_/¯

This ticket was mentioned in Slack in #core by jorbin. View the logs.


4 years ago

#23 @ocean90
4 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

#24 @jorbin
4 years ago

In 37186:

Add npm-shrinkwrap.json to 4.5

By shrinkwraping our dependencies, the same versions of everything will be installed no matter what rules the dependency package.json has specificed.

See #30787.
Fixes #36498.

Note: See TracTickets for help on using tickets.