Make WordPress Core

Opened 3 years ago

Closed 23 months ago

Last modified 19 months ago

#52356 closed defect (bug) (fixed)

npm install does not work on wordpress-trunk on new Mac M1's

Reported by: justinahinon's profile justinahinon Owned by: johnbillion's profile johnbillion
Milestone: 6.0 Priority: normal
Severity: normal Version: 5.7
Component: Build/Test Tools Keywords: has-patch dev-reviewed needs-testing fixed-major
Focuses: Cc:

Description

I've been trying to run a development environment to contribute to WordPress on my new MacBook M1 for some time now, but I go through a lot of mistakes every time.

The first one is that some libraries used by WordPress Core are not yet supported on the ARM architectures of the new Macs (I guess).

For example, running npm install from a folder with WordPress Trunk returns this error:

Error
    at /Users/segbedji/wp/wordpress-develop/node_modules/grunt-contrib-qunit/node_modules/puppeteer/lib/BrowserFetcher.js:112:19
    at FSReqCallback.oncomplete (fs.js:183:21)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! puppeteer@4.0.1 install: `node install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the puppeteer@4.0.1 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

After some research, the puppeteer package has several incompatibilities with the Macs M1, and many issues have already been reported on their repo: https://github.com/puppeteer/puppeteer/issues/6622, https://github.com/puppeteer/puppeteer/issues/6634.

Some solutions have been proposed on the issues. Like this one (https://github.com/puppeteer/puppeteer/issues/6622#issuecomment-759840596) that doesn't work on my end.

But it looks like there is an official fix here https://github.com/puppeteer/puppeteer/commit/9a8479a52a7d8b51690b0732b2a10816cd1b8aef. So I guess upgrading the package version on our end could fix the issue.

Change History (41)

#1 @justinahinon
3 years ago

I may have found a workaround for this.

So, it seems to me that the versions of Node from 14 onwards are "universal". So you need to install a lower version of Node (branch 12) from a terminal emulated with Rosetta 2. Preferably install this version with NVM.

Then from this terminal, run npm install, and it should work correctly.

#2 follow-up: @desrosj
3 years ago

  • Component changed from External Libraries to Build/Test Tools
  • Keywords reporter-feedback added

Hey @justinahinon,

Do you recall which version of NodeJS you were using when running npm install originally? Core runs on 14.x, so that should be the version you use. As of a few weeks ago, all devDependencies are updated to the latest versions (except Webpack related ones, which are at the highest version that supports the version of Webpack trunk uses).

If you're still having issues and are able to narrow down which packages are problematic, we can look at updating them!

#3 in reply to: ↑ 2 @justinahinon
3 years ago

Replying to desrosj:

Do you recall which version of NodeJS you were using when running npm install originally? Core runs on 14.x, so that should be the version you use. As of a few weeks ago, all devDependencies are updated to the latest versions (except Webpack related ones, which are at the highest version that supports the version of Webpack trunk uses).

I was using Node LTS 14.15

#4 @afragen
3 years ago

@justinahinon node v15.x compiles and installs under arm64 (Apple Silicon). Here's what I did.

nvm is installed via plugin zsh-nvm for oh-my-zsh!

You will need Xcode Command Line Tools installed.
xcode-select --install, may need sudo

  1. Open a terminal session.
  2. Run nvm install 15 and allow it time to compile/build.
  3. Run nvm use 15
  4. Run nvm alias default 15, so every shell session uses this version of node by default

After the above, you can open a new terminal session where wordpress-develop is installed.

  1. export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true , this avoids a current error. https://github.com/puppeteer/puppeteer/issues/6622
  2. npm install, you can also start over by rm -rf node_modules && npm install
  3. npm run build

If you're looking to run the npm run env:xxxx instructions you will need to install version of Docker for Apple Silicon, https://docs.docker.com/docker-for-mac/apple-m1/. Note that this Docker app does not automatically update you must update yourself.

If you want to run the above Docker version you will need to modify the .env to use mariadb as the database using latest version.

Last edited 3 years ago by afragen (previous) (diff)

#6 @talldanwp
2 years ago

I've also encountered this after getting a new laptop, but a curious thing is that there's no such problem in the Gutenberg project.

For some background, Gutenberg only uses the puppeteer-core package. That package doesn't install Chromium when running npm install, instead chromium is installed when running the test command:
https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/scripts/test-e2e.js#L31-L33

This is distinct from the puppeteer package, which installs chromium whenever running npm install (which can be a frustrating waste of time).

My first thought was that perhaps core should also use puppeteer-core, but then I was surprised to find it already does. It uses the same tooling as Gutenberg.

Looking into the dependencies a bit further, it looks like the issue is that the grunt-contrib-qunit package has puppeteer as a dependency, and an old version at that. This is what's causing the issue. Versions of puppeteer are locked to specific chromium versions, so it's likely that the version of chromium that project depends on doesn't support the M1.

IMO, any fix should be made upstream to grunt-contrib-qunit, or perhaps WordPress should look into an alternative to that project that doesn't cause chromium to be installed when running npm install.

As a workaround you can set the PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true environment variable, which allows npm install to finish, but running tests won't work. Technically you should be able to install chromium independently (brew install chromium) and also set PUPPETEER_EXECUTABLE_PATH, but I haven't been able to get that working.

edit: actually, I do know the reason for the last thing I mentioned. e2e tests are using puppeteer-core and that doesn't support the environment variables!

Last edited 2 years ago by talldanwp (previous) (diff)

#7 @antonvlasenko
2 years ago

Solution that worked for me when I got my M1 Mac a couple of months ago:

  1. brew install chromium
  2. Go to ~/.zshrc file and add the following lines:
    export PUPPETEER_EXECUTABLE_PATH=`which chromium`
    export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1
    export PATH="/opt/homebrew/opt/node@14/bin:$PATH"
    

/opt/homebrew/opt/node@14/binpath must match the actual node version installed on your Mac).

  1. Log out of your user account and log in again.
  2. Try to install and/or build Gutenberg.
Last edited 2 years ago by antonvlasenko (previous) (diff)

#8 @afragen
2 years ago

Also for Apple Silicon you might need to add the following to your docker-compose.yml under the mysql section so that the correct image of mysql will be used.

mysql:
  platform: linux/amd64


Last edited 2 years ago by afragen (previous) (diff)

#9 @gziolo
2 years ago

The issue with npm install got fixed with [53069]. There is the remaining issue with running npm run env because of some issues with MySQL.

#10 @gziolo
2 years ago

For the issue with MySQL, I followed the following article:
https://www.jesusamieiro.com/installing-wordpress-develop-on-m1-chip/

I executed docker pull --platform linux/x86_64 mysql:5.7 which is probably similar to what @afragen shared but it doesn't require changes to the docker-compose.yml file.

The other way to fix it is to change the database to MariaDB v10 as suggested by @youknowriad.

#11 @afragen
2 years ago

@gziolo you are correct. I have actually tried all 3 methods. They all work.

I don't know if it's possible for the docker-compose.yml` to do a check for a the processor and add the correct platform automatically?

This ticket was mentioned in PR #2650 on WordPress/wordpress-develop by johnbillion.


23 months ago
#12

  • Keywords has-patch added

#13 @johnbillion
23 months ago

  • Keywords dev-feedback added; reporter-feedback removed

I've opened a PR at https://github.com/WordPress/wordpress-develop/pull/2650 which uses the https://hub.docker.com/r/amd64/mysql/ and https://hub.docker.com/r/amd64/mariadb/ images on arm64 machines instead of using the --platform property. Tested and working for me on an M1 MacBook Pro, would appreciate testing from others.

afragen commented on PR #2650:


23 months ago
#14

I've tested this PR and it works on my M1 Mac. Thanks @johnbillion

#15 @afragen
23 months ago

  • Keywords dev-reviewed added; dev-feedback removed

I have tested @johnbillion’s PR and it works great.

#16 @aristath
23 months ago

I had the same issue a while back before seeing this ticket, and switching to MariaDB worked perfectly fine. The changes I made are https://github.com/WordPress/wordpress-develop/compare/trunk...aristath:try/env/mariadb?expand=1 but using the patch from above works just as well.

#17 @SergeyBiryukov
23 months ago

  • Milestone changed from Awaiting Review to 6.1

#18 @johnbillion
23 months ago

Yeah switching to MariaDB works too but I'd like to leave MySQL in place so the choice remains, and so that testing can be performed on both.

@justinahinon @talldanwp @antonvlasenko @gziolo Could you give the PR a try on your M1s please?

#19 @desrosj
23 months ago

#54814 was marked as a duplicate.

This ticket was mentioned in Slack in #core by johnbillion. View the logs.


23 months ago

#21 @costdev
23 months ago

  • Keywords needs-testing added

This ticket was discussed in the bug scrub. Adding needs-testing so that this ticket can move forward.

#22 @johnbillion
23 months ago

If we could get a few more thumbs on on this change we can pull it into 6.0. Any testers both with M1s and without would be appreciated.

#23 @aristath
23 months ago

Tested once again and confirmed 👍

gziolo commented on PR #2650:


23 months ago
#24

npm run env:start works great.

When I run npm run env:install I see the following error:

Status: Downloaded newer image for wordpressdevelop/cli:latest
Creating wordpress-develop-svn_cli_run ... done
ERROR 2002 (HY000): Can't connect to MySQL server on 'mysql' (115)
ERROR: 1
child_process.js:866
    throw err;
    ^

Error: Command failed: docker-compose run --rm cli config create --dbname=wordpress_develop --dbuser=root --dbpass=password --dbhost=mysql --path=/var/www/src --force
    at checkExecSyncError (child_process.js:790:11)
    at execSync (child_process.js:863:15)
    at wp_cli (/Users/gziolo/Projects/wordpress-develop-svn/tools/local-env/scripts/install.js:52:2)
    at Object.<anonymous> (/Users/gziolo/Projects/wordpress-develop-svn/tools/local-env/scripts/install.js:14:1)

Am I missing anything?

johnbillion commented on PR #2650:


23 months ago
#25

@gziolo Thanks for testing, that sounds like you're encountering https://core.trac.wordpress.org/ticket/50342 which is a bug unrelated to this change. If you run npm run env:install again it should work.

#26 @SergeyBiryukov
23 months ago

  • Milestone changed from 6.1 to 6.0

Moving to 6.0 per comment:22, as testing seems to confirm that the PR works as expected.

Last edited 23 months ago by SergeyBiryukov (previous) (diff)

gziolo commented on PR #2650:


23 months ago
#27

@gziolo Thanks for testing, that sounds like you're encountering https://core.trac.wordpress.org/ticket/50342 which is a bug unrelated to this change. If you run npm run env:install again it should work.

I can confirm it resolves the issue. All good in that case. I'm super excited about this patch, thank you so much 🎉

#28 @johnbillion
23 months ago

  • Owner set to johnbillion
  • Resolution set to fixed
  • Status changed from new to closed

In 53358:

Build/Test Tools: Allow the local development environment to run on Apple M1 machines without requiring local configuration changes.

The amd64/mysql and amd64/mariadb official images from Docker are compatible with x86 running on the ARM64v8 architecture of M1 machines.

Props afragen, justinahinon, desrosj, talldanwp, antonvlasenko, gziolo, aristath, johnbillion

Fixes #52356

#29 @afragen
23 months ago

@johnbillion does this need to be reopened and backported to 6.0?

#30 @johnbillion
23 months ago

  • Keywords fixed-major added
  • Resolution fixed deleted
  • Status changed from closed to reopened

You're too quick for me :-)

#31 @johnbillion
23 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 53359:

Build/Test Tools: Allow the local development environment to run on Apple M1 machines without requiring local configuration changes.

The amd64/mysql and amd64/mariadb official images from Docker are compatible with x86 running on the ARM64v8 architecture of M1 machines.

Merges [53358] to the 6.0 branch.

Fixes #52356

This ticket was mentioned in Slack in #core by sergey. View the logs.


23 months ago

This ticket was mentioned in PR #2700 on WordPress/wordpress-develop by johnbillion.


20 months ago
#35

Usage of the amd64/mysql and amd64/mariadb official images from Docker was introduced in https://core.trac.wordpress.org/ticket/52356 as they are compatible with an x64 container running on a host machine using ARM64 architecture (ie. Apple M1 machines).

However they're also compatible with an x64 host machine which should mean they can be used by default instead of only when the host uses ARM64. This allows some image juggling to be removed.

## Testing

  • Ensure Docker Desktop is running if necessary
  • While still on your current branch (eg. trunk) run npm run env:stop
  • Checkout the branch for this PR
  • Run npm run env:start
  • Run some tests that use the database container, eg. npm run test:php

## Todo:

  • [ ] Test on x64 Intel Mac
  • [x] Test on ARM64 M1 Mac
  • [x] Test on x64 Linux (GitHub Actions)
  • [ ] Test on x64 Windows

ockham commented on PR #2700:


19 months ago
#36

FWIW, _without_ this patch, I cannot run the dev environment locally on my M1 Pro. npm run env:start fails with

no matching manifest for linux/arm64/v8 in the manifest list entries

when attempting to install the mysql image. (Related Slack convo.)

If you're on Apple Silicon, you can try to reproduce this issue on trunk by running npm run env:reset followed by npm run env:start.

michalczaplinski commented on PR #2700:


19 months ago
#37

I've had the same exact issue as @ockham and it was fixed by cherry-picking 34d9b9c (this PR) on top of trunk.

gmovr commented on PR #2700:


19 months ago
#38

Tested on x64 Intel Mac (stats below):
https://i0.wp.com/user-images.githubusercontent.com/52675688/188884375-f3b850f1-dc7a-4f8a-8506-f0e32b6f1099.png

{{{sh
$ node -v
14.17.0

$ docker -v
Docker Version 20.10.17

}}}

Docker Desktop v 4.12.0
Compose v 2.20.2

  • npm run env:reset Just in case
  • gh pr checkout 2700
  • npm install
  • npm run build:dev
  • npm run env:start
  • npm run env:install
  • npm run test:php

Tests pass 👌

OK, but incomplete, skipped, or risky tests!
Tests: 13829, Assertions: 45228, Skipped: 59, Risky: 30.

withinboredom commented on PR #2700:


19 months ago
#39

  • Tested in Windows WSL (ubuntu base): Success
  • Tested in Windows 11 native: Success
  • Tested in Linux native, no docker desktop (Pop! OS 22.04): Success

johnbillion commented on PR #2700:


19 months ago
#40

Thanks @gmovr @withinboredom 👍 👍

Note: See TracTickets for help on using tickets.