WordPress.org

Make WordPress Core

Opened 11 months ago

Last modified 3 months ago

#49274 new enhancement

Grunt copy:files should ignore node_modules

Reported by: iandunn Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

I've gotten this error several times when run grunt watch, and several others have too:

Maximum call stack size exceeded

It happens because there's a node_modules folder somewhere in src/, and they typically contain too many files for Grunt to handle.

Configuring copy:files to always ignore node_modules folders seems like it'd remove that friction for people.

Attachments (3)

49274.1.diff (567 bytes) - added by iandunn 11 months ago.
Adds node_module ignore to _watch task.
49274.2.diff (638 bytes) - added by iandunn 3 months ago.
1.diff + ignoring build and build-* folders
49274.3.diff (381 bytes) - added by iandunn 3 months ago.
different approach, ignore all of wp-content

Download all attachments as: .zip

Change History (13)

@iandunn
11 months ago

Adds node_module ignore to _watch task.

#1 @iandunn
11 months ago

49274.1.diff works for me, but could use some more eyes/testing.

  1. Clone several plugins that depend on many NPM packages into src/wp-content/plugins (e.g., gutenberg, yoast, coblocks, etc)
  2. Run npm install in each of their directories
  3. Go up to src and run npm run dev
  4. Change some files in src. If there are enough files inside plugins/**/node_modules, then you'll get this error from the watch task:
Running "_watch" task
Waiting...
Warning: Maximum call stack size exceeded
  1. Cancel the npm run dev task
  2. Apply the patch
  3. Restart npm run dev, and change some files.

This ticket was mentioned in Slack in #core by iandunn. View the logs.


11 months ago

#3 @iandunn
4 months ago

It looks like the build, build-module, build-types, etc folders inside plugins/gutenberg also cause this problem. We could exclude those explicitly, or maybe do that for all plugins? There may be some where it's necessary to watch build folder, though?

This is helpful for finding the folders with the most files:

for i in `find src -type d `; do echo `ls -a $i | wc -l` $i; done | sort -n

@iandunn
3 months ago

1.diff + ignoring build and build-* folders

@iandunn
3 months ago

different approach, ignore all of wp-content

#4 @iandunn
3 months ago

49274.2.diff adds additional ignore entries for build and build-* folders, which fixes the max call stack error with Gutenberg (comment:3). The watch task takes ~25 seconds to run, though, so it's still unusable.

49274.3.diff takes a more radical approach, and ignores all of wp-content. This solves the speed issue, but has the (somewhat) unintended consequence of preventing mu-plugins, plugins, and themes from being copied from src to build.

I often need to test plugins against trunk, especially Gutenberg, so that seems like a reasonable use case. People could create & build those things directly in WP's build folder, but it doesn't seem like build is really a safe location for anything that isn't in src. I often rm -rf build when troubleshooting.

Does anyone have any thoughts?

#5 @iandunn
3 months ago

Part of the problem is that too many files are being watched, but another part might be that sometimes there are two watcher tasks running simultaneously.

If grunt watch and grunt watch:phpunit are both running simultaneously, then that'd probably make the problem much worse.

The Handbook says that npm run grunt watch -- --phpunit --group={testgroup} will run the normal watch tasks and watch:phpunit simultaneously, but that isn't working for me. Apparently I added that note, but don't remember the details. It's also possible some things have changed in the meantime, and the docs are out of date.

This ticket was mentioned in Slack in #core by iandunn. View the logs.


3 months ago

#7 @iandunn
3 months ago

Even without running grunt watch:phpunit, npm run watch still takes 10s per cycle on my system, when using 49274.2.diff.

For 49274.3.diff, one potential solution to the syncing problem might be to symlink build/wp-content to src/wp-content, or at least the mu-plugins/ and plugins/ folders, since Core doesn't have anything in those. I'll test that for awhile and see if I notice any problems.

#8 @iandunn
3 months ago

So far, the only problem I've run into with symlinking wp-content is that these files are sometimes deleted during build and watch tasks:

src/wp-content/index.php
src/wp-content/plugins/hello.php
src/wp-content/plugins/index.php
src/wp-content/themes/*

One potential solution would be to adjust Gruntfile.js to avoid that, but I haven't dug into that, and any unintended consequences it'd have.

For now I'm just symlinking everything inside wp-content except those files. It seems to work fine at first glance, but I'll keep working with it for a few days.

Related: #44492, where symlinking was tried for most of the src/ folder, but that didn't work out because of problems with Windows support.

cc @netweb, @omarreiss, @SergeyBiryukov

#9 @netweb
3 months ago

Related: #43731Use Webpack + NPM scripts to build all the things

It might be a good opportunity explore an alternate file watcher, and move away from the Grunt task and it’s limitations

Adding a native npm watch script and then wrapping a back-compact Grunt task

Facebook’s Watchman comes to mind, Windows is still officially beta, but appears to be well tested.

#10 @iandunn
3 months ago

#50412 is also related, since having two watchers running appears to be part of the problem.

Note: See TracTickets for help on using tickets.