WordPress.org

Make WordPress Core

Opened 13 months ago

Last modified 3 months ago

#23912 new enhancement

Add Composer package description

Reported by: Rarst Owned by:
Milestone: Future Release Priority: normal
Severity: trivial Version: 3.5
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description

WordPress, as software download, lacks machine-readable source of meta information about it. For PHP projects de-facto standard for such is Composer via composer.json file in project root.

While WP currently doesn't use or need Composer dependency functionality, it will help provide information to developers and improve WP usage as dependency in projects that make use of Composer.

Suggested composer.json draft:

{
	"name"        : "wordpress/wordpress",
	"description" : "WordPress is web software you can use to create a beautiful website or blog.",
	"keywords"    : ["blog", "cms"],
	"homepage"    : "http://wordpress.org/",
	"license"     : "GPL-2.0+",
	"authors"     : [
		{
			"name"    : "WordPress Community",
			"homepage": "http://wordpress.org/about/"
		}
	],
	"support"     : {
		"issues": "http://core.trac.wordpress.org/",
		"forum" : "http://wordpress.org/support/",
		"wiki"  : "http://codex.wordpress.org/",
		"irc"   : "irc://irc.freenode.net/wordpress",
		"source": "http://core.trac.wordpress.org/browser"
	},
	"require"     : {
		"php": ">=5.2.4"
	}
}

Attachments (7)

installer.patch (5.9 KB) - added by johnpbloch 8 months ago.
WordPress Core Installer
composer.patch (970 bytes) - added by johnpbloch 8 months ago.
Draft composer.json patch
composer.2.patch (970 bytes) - added by johnpbloch 8 months ago.
composer.3.patch (6.6 KB) - added by johnpbloch 8 months ago.
Comprehensive patch with installer and composer.json
composer.4.patch (1.1 KB) - added by johnpbloch 7 months ago.
ext-mysql and ext-curl as suggestions
composer.5.patch (1.1 KB) - added by johnpbloch 7 months ago.
Increase version constraints
23912.6.patch (2.2 KB) - added by bpetty 6 months ago.

Download all attachments as: .zip

Change History (96)

comment:1 SergeyBiryukov13 months ago

  • Version changed from trunk to 3.5

comment:2 retlehs12 months ago

  • Cc retlehs added

comment:3 F J Kaiser12 months ago

  • Cc 24-7@… added

comment:4 TJNowell11 months ago

  • Cc contact@… added

comment:5 slbmeh11 months ago

  • Cc steve@… added

comment:6 slbmeh11 months ago

Another nice to have associated with this is to add to packagist once the composer.json is in the repo.

comment:7 slbmeh11 months ago

  • Keywords dev-feedback added

comment:8 bpetty11 months ago

  • Cc bpetty added

comment:9 hakre11 months ago

  • Cc hanskrentel@… added

comment:10 follow-up: Rarst11 months ago

Let's limit this ticket to having composer.json at all for now.

Bolting it to Composer stack properly is more complex than that and would possibly need custom installer to correct path (composer/installers currently support WP themes and plugins, but not core itself explicitly).

comment:11 in reply to: ↑ 10 ; follow-up: bpetty11 months ago

Replying to Rarst:

Let's limit this ticket to having composer.json at all for now.

Bolting it to Composer stack properly is more complex than that and would possibly need custom installer to correct path (composer/installers currently support WP themes and plugins, but not core itself explicitly).

Agreed. Custom install paths for core should be managed by end users anyway.

The only other path we would arguably customize would be the vendor-dir in the case of the WP composer manifest being used as the root manifest in order to use a custom install path for 3rd party libs used directly by WordPress core. But that's only if WP started depending on Composer to pull in 3rd party deps (for development, they'd obviously be pre-bundled for zip distribution still), and even if that happened, WP doesn't have any reason not to use the default "vendor" path anyway.

comment:12 bpetty11 months ago

For reference, MediaWiki has done this recently too (just in February).

comment:13 in reply to: ↑ 11 ; follow-up: Rarst11 months ago

Replying to bpetty:

Agreed. Custom install paths for core should be managed by end users anyway.

The only other path we would arguably customize would be the vendor-dir

Not quite. The problem is that package needs to opt in to use composer/installers or otherwise Composer will not allow user to choose where core (used as dependency) goes and it will mandatory go into vendor dir, which does not quite fit WP core.

I've opened issue about adding custom type for core https://github.com/composer/installers/issues/77 but as above - this is probably conversation for another ticket and another WP release since this is not making it into 3.6

comment:14 in reply to: ↑ 13 bpetty11 months ago

Replying to Rarst:

Not quite. The problem is that package needs to opt in to use composer/installers or otherwise Composer will not allow user to choose where core (used as dependency) goes and it will mandatory go into vendor dir, which does not quite fit WP core.

When used as a dependency, yes. In fact, there is no "logical default location" for WordPress core at all. WordPress core isn't the only package with this issue - that goes for all PHP applications that need a docroot. The Composer issue you opened isn't likely to go anywhere because that's simply not how Composer (or composer/installers) is designed to work. There isn't a new installer type for every PHP application that requires a docroot.

However, when talking about the "vendor-dir" configuration, I wasn't talking about the package when used as a dependency. In that case, the "vendor-dir" configuration option is completely ignored by Composer, so it doesn't matter what it is. It's only used when the package is used as the root package, and there is some validity to customizing it for some applications, but like I said, it doesn't matter anyway because WordPress isn't one of those applications.

The "vendor-dir" setting (core composer) and the "type" setting (composer/installers) are two completely different settings with very different behaviors.

comment:15 Rarst11 months ago

In fact, there is no "logical default location" for WordPress core at all.

There are two possible locations:

  • site root
  • root subdirectory with arbitrary name

These are what code (as of recently in full) and documentation currently support.

The Composer issue you opened isn't likely to go anywhere because that's simply not how Composer (or composer/installers) is designed to work. There isn't a new installer type for every PHP application that requires a docroot.

composer/installers is designed precisely to satisfy path requirements of [legacy] PHP applications. Which it already does for WP extensions, I merely propose to codify it for core as well.

comment:16 slbmeh11 months ago

IMO composer/installers is intended to bring some dependency management support to applications that do not have native composer support. e.g. using the autoloader and working directly out of the defined vendor-dir.

A custom installer may be nice to have, but not necessary for inclusion on packagist to be pulled in as a dependency. I have a generic project, fancyguy/webroot-installer intended to allow installation of any package that must live in the document root that was built to allow me to install wordpress as a piece of larger applications.

Last edited 11 months ago by slbmeh (previous) (diff)

comment:17 nacin9 months ago

  • Milestone changed from Awaiting Review to 3.7
  • Type changed from feature request to enhancement

comment:18 Rarst9 months ago

Since this was dormant for a while, let me summarize explorations on core as Composer package since.

In a nutshell for proper package mechanics will need to decide for core if use wordpress-plugin type or different (from composer/installers) installer. If aiming for subdirectory support only (no in-root) custom installer will be trivial on top of composer/installers (WP support in it is literally 10 lines of code for example https://github.com/composer/installers/blob/master/src/Composer/Installers/WordPressInstaller.php )

comment:19 Rarst9 months ago

Draft updated with wordpress-plugin type (path of least resistance):

{
	"name"       : "wordpress/wordpress",
	"description": "WordPress is web software you can use to create a beautiful website or blog.",
	"keywords"   : ["blog", "cms"],
	"homepage"   : "http://wordpress.org/",
	"license"    : "GPL-2.0+",
	"authors"    : [
		{
			"name"    : "WordPress Community",
			"homepage": "http://wordpress.org/about/"
		}
	],
	"support"    : {
		"issues": "http://core.trac.wordpress.org/",
		"forum" : "http://wordpress.org/support/",
		"wiki"  : "http://codex.wordpress.org/",
		"irc"   : "irc://irc.freenode.net/wordpress",
		"source": "http://core.trac.wordpress.org/browser"
	},
	"type"       : "wordpress-plugin",
	"require"    : {
		"php"                : ">=5.2.4",
		"composer/installers": "~1.0"
	}
}

Then root package, requiring WordPress, would need to specify path for it in its composer.json:

"require"     : {
        "wordpress/wordpress"          : ">=3.6",
    },
"extra"       : {
        "installer-paths": {
            "./wp/": ["wordpress/wordpress"]
        }
    }

Also this probably drags de-facto requirement for Composer use to PHP 5.3+, but I can't imagine it being relevant since Composer itself requires 5.3 to work and earlier versions in requirements are used more informatively for non-Composer use, than practically.

Last edited 9 months ago by Rarst (previous) (diff)

comment:20 bpetty9 months ago

Back in comment:10, you were advocating keeping things simple just initially here without any extra dependencies on the Composer stack, or worrying about installer paths for now. That doesn't appear to be the case anymore.

I still feel like it would be a mistake adding a dependency on composer/installers right off the start, as well as abusing the "wordpress-plugin" type for a package it was not designed to be used for.

Using the "wordpress-plugin" type for core itself would end up requiring developers and admins using the package as a dependency to absolutely specify an installation path (even if it's fine in the default vendor path) since otherwise, it will be treated like a plugin and installed in "wp-content/plugins". Sure, this is easy to do, and I can't currently think of a situation where it wouldn't be possible to customize, but it still results in a terrible default behavior.

Honestly, anything requiring a webroot should never be used as a package dependency to begin with. It's always your base software you're installing, and as your Composer project root, it will always be in the root path of your specified installation directory (either manually installed there, or by using "composer create-project" in most situations). Defining a composer/installers "type" certainly isn't going to effect that at all, but it would download and install composer/installers even if it's not in use, and placing it within your webroot every time. Of course, if you're using this with plugins (and if you're using Composer, chances are pretty good that you are), it will require composer/installers anyway until WordPress supports traditional Composer packages itself (not just 3rd party libraries but also plugins), so probably no harm in either case. However, why make that decision in the base package? Hopefully at some point in the future, composer/installers is never required even for plugins and themes. The composer/installers package is only designed to be a transitional package, not for the purpose of building new architecture intentionally designed to be a crutch for all future use.

Anyway, the issues surrounding this decision are complex, which is exactly why I was agreeing with your first comment (comment:10) in regards to ignoring this for now, and just getting a basic composer package file in place with appropriate meta data, and appropriate PHP version dependency information. Can we at least agree on that, and push the composer/installers dependency back to a new ticket that can be evaluated when we also have the time to investigate possible solutions to including 3rd party composer libraries through the autoloader? We have to handle that eventually too, and it has some relevance to the composer/installers approach.

comment:21 Rarst9 months ago

Back in comment:10, you were advocating keeping things simple just initially here without any extra dependencies on the Composer stack, or worrying about installer paths for now. That doesn't appear to be the case anymore.

Yes, which was two months and a lot of exploration ago. That is still an option of course, but it fixes WP core to the vendor directory which is considerably limiting.

I still feel like it would be a mistake adding a dependency on composer/installers right off the start

Since any Composer-driven WP project will probably need composer/installers for every theme and plugin, it's not really out of line requirement.

Sure, this is easy to do, and I can't currently think of a situation where it wouldn't be possible to customize, but it still results in a terrible default behavior.

I don't disagree, I am just trying to lay out a all the options and come to one that would be reasonably workable both from perspective of Composer as tool and from perspective of core actually adopting it.

Honestly, anything requiring a webroot should never be used as a package dependency to begin with.

I am not aiming for webroot usage, I am aiming for subdirectory install (in WP terms) / package with custom path requirement (Composer terms). Subdir install is already de-facto preferable for any serious site, since it cleanly isolates core from all the other things.

Anyway, the issues surrounding this decision are complex, which is exactly why I was agreeing with your first comment (comment:10) in regards to ignoring this for now, and just getting a basic composer package file in place with appropriate meta data, and appropriate PHP version dependency information.

I could live with that, but my current opinion that it will not actually move WP usage with Composer forward. It's the very start of 3.7 cycle and this was put on agenda right away, I think some time to ponder it we have.

Can we at least agree on that, and push the composer/installers dependency back to a new ticket that can be evaluated when we also have the time to investigate possible solutions to including 3rd party composer libraries through the autoloader?

If I remember right autoloaders in core are not being considered as long as 5.2 compat is the target (argument is that SPL might be disabled).

In a nutshell I am trying to provide as much information and options as I can. It's up to core team to choose which level (if any) of Composer compatibility to set as target(s).

comment:22 johnpbloch9 months ago

I think it would be very much worth considering having a custom installer written specifically for core and released by WordPress (e.g. wordpress/wordpress-core-installer). This would get around the need to shoehorn core into the wordpress-plugin package type while still allowing us to set it as a wordpress-core type to get it into a custom directory other than vendor/wordpress/wordpress. This would let us control the install process as much as we need to. Also, should installation options ever change in the future (I don't know how, but I don't pretend to know how things will change years down the road), we could adapt as needed by just bumping the installer's version.

And of course, if someone wants to use WordPress as the web root package, thus sacrificing the ability to upgrade core with Composer, there is always the composer create-project wordpress/wordpress command.

comment:23 johnpbloch9 months ago

  • Cc johnpbloch@… added

comment:24 johnpbloch9 months ago

I put together a proof of concept for a core installer and put it up on github (https://github.com/johnpbloch/wordpress-core-installer). Using something similar to this would allow us to define a default subdirectory into which to install WordPress core. All we need to do is add a dependency to the core package for the custom installer and optionally set a custom default:

"require": {
    "johnpbloch/wordpress-core-installer": "~0.1"
},
"extra": {
    "wordpress-install-dir": "wp"
}

If the default location is acceptable for a project, the composer file only needs to declare core as a dependency. The rest is handled transparently. Otherwise, it's not too much extra to declare a custom path for core:

"require": {
    "wordpress/wordpress": "~3.7"
},
"extra": {
    "wordpress-install-dir": "httpdocs/wp"
}

And, of course, we would want to change the vendor to wordpress, etc.

johnpbloch8 months ago

WordPress Core Installer

comment:25 johnpbloch8 months ago

Ok, I've uploaded a patch to add the custom core installer to the main source repo. It's under the tools directory, since it is technically a separate package. It's also in the traditional trunk/tags directory pattern so that the repository itself can be used by Composer (or Packagist). The patch also includes a tagged version of 0.1.0.

Currently, this package conflicts with the composer/installers package (which would be used to ensure plugins and themes end up in the correct place) because of a bug in composer/installers, but I already have a patch submitted to fix it. I anticipate that my patch will be accepted soon.

If/when this package gets added to the repo, I think we should look into getting it added to the Packagist repository.

johnpbloch8 months ago

Draft composer.json patch

johnpbloch8 months ago

comment:26 johnpbloch8 months ago

I added a patch for the draft composer.json file too. I figure it's time to get it into the form of patches now (ignore the first one, I messed up on a version number...).

Regarding versions, we're going to want to put the version directly into the composer.json file, since the non-traditional structure of the repo won't let us infer tags any more. The version in the file in trunk should always be the next major version with -dev appended. The version in tags should always be the correct version for that tag directory, and the version in branches should always be that branch number with -dev appended. I'm not really sure how the versions will work with Packagist if we're going to add the core package there, but I'm sure we'll figure it out.

comment:27 follow-up: Rarst8 months ago

Regarding versions, we're going to want to put the version directly into the composer.json file

Probability of that going horribly wrong at some point is about 100% :)

since the non-traditional structure of the repo won't let us infer tags any more

Subversion paths are mappable http://getcomposer.org/doc/05-repositories.md#subversion-options , however if I got new things right we would want "old" repository with "built" results and tags and everything anyway.

Actually might want to ponder if it's preferable to point packages at SVN or GitHub mirror. Pointing to GitHub will provide HTTP downloads automagically with Composer.

One more thing I wanted to mention - are there any more platform dependencies we would want to require or suggest in composer.json? Like what images stuff needs currently, etc.

comment:28 in reply to: ↑ 27 johnpbloch8 months ago

since the non-traditional structure of the repo won't let us infer tags any more

Subversion paths are mappable http://getcomposer.org/doc/05-repositories.md#subversion-options , however if I got new things right we would want "old" repository with "built" results and tags and everything anyway.

This is a very good point. I'd (foolishly) forgotten that the old repo is going to be kept in sync and with the same (traditional) structure. We can just use that location.

Actually might want to ponder if it's preferable to point packages at SVN or GitHub mirror. Pointing to GitHub will provide HTTP downloads automagically with Composer.

But pointing to Github also makes dev checkouts take WAY longer because it has to clone the entire repo instead of just /trunk/. My personal satis build uses the core svn repos and it has built packages that are HTTP downloads. I'll have to look into how I'm doing that, though. It might be through explicit definition of the download package.

One more thing I wanted to mention - are there any more platform dependencies we would want to require or suggest in composer.json? Like what images stuff needs currently, etc.

I suppose adding "ext-mysql" to the require block is a good idea. Besides that and PHP 5.2.4, WordPress doesn't really have hard dependencies. IIRC, image manipulation fails gracefully in the absence of GD and Imagick. We could probably throw some of those in as dev-dependencies, though. Or we could add a suggest block. Here are the things that occurred to me that would go in there:

  1. ext-gd
  2. ext-imagick
  3. ext-curl

Anything else? No need to put SPL in there, since they're on 5.3 or higher if they're using Composer.

johnpbloch8 months ago

Comprehensive patch with installer and composer.json

comment:29 johnpbloch8 months ago

I just uploaded a new comprehensive patch with the installer and the main package's composer manifest.

I removed the 'version' line. As Rarst pointed out, only bad things were going to come from that idea. Aside from the fact that you can manually override default repo structure for composer, we can also just use the old repo at http://core.svn.wordpress.org as the package source.

Also, for software built to work with as many server configurations as possible, there isn't really anything useful that we can put in 'suggest'. For now, we can probably leave it out.

comment:30 follow-up: nacin8 months ago

Rather than tools/installer/tags/0.1.0/src/WordPress/Composer/Installer.php can we just do tools/composer/WordPress/Composer/Installer.php? (Optionally src in there if required?) Given that our own trunk will be branched and tagged, I'd rather not manage internal branches and tags for this.

Or, if this is its own self-standing thing, is it worth managing this at github.com/wordpress/composer-installer (or wordpress/composer?) instead?

Also, for software built to work with as many server configurations as possible, there isn't really anything useful that we can put in 'suggest'. For now, we can probably leave it out.

I would tend to disagree, actually. There are a number of things we can "suggest". Cursory examination produces ext-gd, ext-imagick, ext-curl (as you said), ext-json (maybe omit given current upstream licensing mess), ext-exif, ext-openssl, ext-iconv, ext-libxml, ext-hash, ext-mbstring, ext-simplexml, ext-zlib. Also, ext-mysql is missing from require.

The desires for robust functionality aside, I would not mind showing off to everyone "Hey look, we use all of these obvious things, but since PHP is, you know, we still work without them — beat that." It's an interesting case study in portability.

comment:31 in reply to: ↑ 30 johnpbloch8 months ago

Replying to nacin:

Rather than tools/installer/tags/0.1.0/src/WordPress/Composer/Installer.php can we just do tools/composer/WordPress/Composer/Installer.php? (Optionally src in there if required?) Given that our own trunk will be branched and tagged, I'd rather not manage internal branches and tags for this.

Or, if this is its own self-standing thing, is it worth managing this at github.com/wordpress/composer-installer (or wordpress/composer?) instead?

The installer is intended to be a self-contained package. Something on Github would work equally well. Personally, I think that would be a preferable solution to using this repository. I thought it best to start with a patch that included it here, though. :)

Also, for software built to work with as many server configurations as possible, there isn't really anything useful that we can put in 'suggest'. For now, we can probably leave it out.

I would tend to disagree, actually. There are a number of things we can "suggest". Cursory examination produces ext-gd, ext-imagick, ext-curl (as you said), ext-json (maybe omit given current upstream licensing mess), ext-exif, ext-openssl, ext-iconv, ext-libxml, ext-hash, ext-mbstring, ext-simplexml, ext-zlib.

Fair enough. Let's get a list together. Aside from the following:

ext-gd
ext-imagic
ext-curl
ext-json
ext-exif
ext-openssl
ext-iconv
ext-libxml
ext-hash
ext-mbstring
ext-simplexml
ext-zlib

we should also include

ext-apc
ext-xcache
ext-eaccelerator
ext-memcache
ext-memcached
ext-mysql
ext-mysqli
ext-PDO
ext-ssh2

Any others? I'm sure I'm missing something or misspelling something, or both.

Also, ext-mysql is missing from require.

That's intentional, now that it's deprecated in the latest version of PHP. The requirement can be handled by other extensions plus a drop-in plugin, so it strikes me as a suggestion rather than a requirement. Thoughts?

comment:32 Rarst8 months ago

Note that suggests (unlike requires) take not version, but human-readable description of why the thing is suggested http://getcomposer.org/doc/04-schema.md#suggest Which might take a little code auditing rabbit hole to do write up precisely (been there http://wordpress.stackexchange.com/a/42212/847 ).

Myself I am hesitant to suggest extensions that are functionally irrelevant. For example HTTP API works with or without curl, suggestion to have it is not important information. Extension listed should be those, without which functionality starts to fail. For example I don't think WP has extension-less image resize, right?

Also I think it should also be limited to extensions core actually uses natively - object caches, alternate DB drivers and such are implemented by extensions.

Last edited 8 months ago by Rarst (previous) (diff)

comment:33 christian10128 months ago

  • Cc christian1012 added

comment:34 nacin8 months ago

ext-mysql is still required by core. It should be listed in require until core no longer requires it out of the box. The deprecation was done for political reasons (which I encouraged), not because it fails to work.

Myself I am hesitant to suggest extensions that are functionally irrelevant. For example HTTP API works with or without curl, suggestion to have it is not important information. Extension listed should be those, without which functionality starts to fail. For example I don't think WP has extension-less image resize, right?

Well, the HTTP API continues to prefer curl. There *are* some things that cannot be done via streams/sockets as well. So it should definitely be suggested.

We do "extension-less image resize" using browser resize. So, no. ext-gd and ext-imagick should also both be suggested.

Also I think it should also be limited to extensions core actually uses natively - object caches, alternate DB drivers and such are implemented by extensions.

I agree.

comment:35 follow-up: Rarst8 months ago

ext-mysql is still required by core. It should be listed in require until core no longer requires it out of the box

require is a hard requirement, Composer will error on build if any of the requirements cannot be satisfied. I think ext-mysql as such hard requirement unnecessarily couples core package to presence of extension, while core does have very viable support for swappable database layer.

Hypothetically in environment where ext-mysql cannot be installed (hosting restriction for example) this will completely prevent core package from working.

There *are* some things that cannot be done via streams/sockets as well.

Any of such things actually exposed in HTTP API? Just curious. I don't think I ever did anything with API that was sensitive to the engine used.

comment:36 in reply to: ↑ 35 rmccue7 months ago

Replying to Rarst:

Any of such things actually exposed in HTTP API? Just curious. I don't think I ever did anything with API that was sensitive to the engine used.

cURL is much faster (we're talking orders of magnitude) and has better support for a lot of things, including proxies (we only support HTTP proxies, cURL supports basically every type). Off the top of my head, I can't think of any big things that sockets don't support that we use, but the performance difference is worth it alone.

comment:37 netweb7 months ago

  • Cc netweb added

comment:38 follow-up: nacin7 months ago

Okay, so, ext-mysql becomes a suggestion, and ext-curl should be a suggestion as well. Given that:

  • I need a patch that adds a composer.json file.
  • There should be a repository somewhere on GitHub that operates as a composer installer for WordPress core. At some point we could decide whether to transfer it to the WordPress account.

johnpbloch7 months ago

ext-mysql and ext-curl as suggestions

johnpbloch7 months ago

Increase version constraints

comment:39 in reply to: ↑ 38 johnpbloch7 months ago

Replying to nacin:

Okay, so, ext-mysql becomes a suggestion, and ext-curl should be a suggestion as well. Given that:

  • I need a patch that adds a composer.json file.

Done

  • There should be a repository somewhere on GitHub that operates as a composer installer for WordPress core. At some point we could decide whether to transfer it to the WordPress account.

Done? It's named johnpbloch/wordpress-core-installer. Should I add a new package instead?

comment:40 johnpbloch7 months ago

Also, this is probably a good time to discuss the text in the suggest entries from my last composer.json patch:

"ext-mysql": "A MySQL extension is required for WordPress to function",
"ext-curl": "cURL is the preferred extension for remote requests with the WordPress HTTP API"

comment:41 nacin7 months ago

  • Milestone changed from 3.7 to Future Release

There was a lengthy discussion on this in IRC last weekend. Consensus seemed to be:

  • Add a large list to suggest because pretty much everything proposed to be listed *does* enhance core in either functionality (e.g. exif) or performance (e.g. curl).
  • ext-mysql should be a requirement, because it is possible to have another package override this requirement, per johnpbloch.
  • The various caching and database drivers should not be in suggest at all. Or they can be, for situations where well-established plugins implement them (to me, this is just mysqli, apc, and memcache). I am not sure what the consensus was.

But, rmccue expressed overall reservations as to whether we should even add composer.json at the moment. I'm indifferent, and think it would be best for us to reign in what clearly got out of hand here to figure out A) what our goals are and B) what kind of impact this actually makes. (We need to figure out what this improves beyond just johnpbloch's workflow.) Because this is so edge, and there isn't yet a complete consensus or a patch, I'm punting it (at bpetty's advice).

I'm fine with this going in once rmccue, tierra, Rarst, and johnpbloch all agree on the approach.

comment:42 johnpbloch7 months ago

  • Cc johnpbloch@… removed

comment:43 Rarst7 months ago

  • Cc contact@… removed

comment:44 TJNowell7 months ago

  • Cc contact@… removed

comment:45 F J Kaiser7 months ago

  • Cc 24-7@… removed

comment:46 nacin7 months ago

To be clear, since I'm seeing a lot of unsubscribe activity here. As I said, I'm fine with this going in once rmccue, tierra, Rarst, and johnpbloch endorse a particular approach. I have no stake in this. My only original (and weak) objection was it'd be a new root file, and two months ago deveop.svn eliminated that as a problem (the proposal for it actually cited composer.json as something we could add). You are all very reasonable, smart people who generally agree on things, and all of you have positively talked about Composer in the past, so I don't think requesting a consensus was unrealistic or quixotic. It's not like I deliberately spiked the ticket by requiring that you must get Republicans and Democrats to agree on tax reform. This ticket isn't dead. Honestly, just tell me what to commit and I'm happy to commit it, even for 3.7.

On the other hand, Rarst, if you no longer think that WordPress should have a composer.json file, please close your ticket as wontfix. This isn't a request to close it if you don't think it is worth fighting for it anymore, only if you honestly don't think anything should be added.

comment:47 follow-up: Rarst7 months ago

I still consider adding composer.json convenient and beneficial change to WordPress core.

I will not however dedicate any more of my time to this ticket at this time. There are other Composer things for me to work on, where more practical and less core-coupled progress can be made.

comment:48 in reply to: ↑ 47 nacin7 months ago

Replying to Rarst:

I still consider adding composer.json convenient and beneficial change to WordPress core.

I will not however dedicate any more of my time to this ticket at this time. There are other Composer things for me to work on, where more practical and less core-coupled progress can be made.

Is there a patch attached to this ticket you think should be committed as-is?

comment:49 follow-up: Rarst7 months ago

Is there a patch attached to this ticket you think should be committed as-is?

composer.5.patch is nearly identical to what I use in practice at the moment.

Note that I am not sure about you implying committing it as non root file, since that defeats technical point of using composer.json to automagically create Composer packages for core via repo scan.

comment:50 in reply to: ↑ 49 nacin7 months ago

Replying to Rarst:

Is there a patch attached to this ticket you think should be committed as-is?

composer.5.patch is nearly identical to what I use in practice at the moment.

Thanks.

Note that I am not sure about you implying committing it as non root file, since that defeats technical point of using composer.json to automagically create Composer packages for core via repo scan.

Didn't mean to imply that at all. I was only saying that I have no outstanding objections to this ticket. My only original objection (6 months ago) was about cluttering the root of core.svn. (I was also trying to make it clear that my objection was not strong enough to make it a "blocker".) Now that we have develop.svn (which was helped along by a need for root files like composer.json), that objection no longer stands, as the directory structure is designed for root files like these.

comment:51 Rarst7 months ago

Now that we have develop.svn (which was helped along by a need for root files like composer.json), that objection no longer stands, as the directory structure is designed for root files like these.

I am not confident we are on same page here. What do you mean "designed for root files"?

For automated creation of packages versioning is derived from version control heads, which have composer.json:

  • trunk/composer.json turns trunk SVN folder into dev-trunk package version
  • tags/3.6/composer.json turns tags/3.6 SVN folder into 3.6.0.0 package version
  • and so on

I am not sure which distinction you imply between "old" and "new" repo for this purpose, since the desired result is same - composer.json present in root of WP core files.

comment:52 sc0ttkclark7 months ago

  • Cc lol@… added

comment:53 bpetty7 months ago

The automated package creation you're referring to works perfectly fine with the root of the develop repo @nacin is referring to. It would still be in trunk/composer.json, and tags/3.7/composer.json. In fact, it has to be right there for the develop repo. There just isn't much of a reason to include it in the document root (src) except that the existing "wordpress-plugin" installer type defines a default install location that expects wp-content at the root of a WordPress installation installed using the create-project command. That, and it would just save a bit more space and time on download sizes and speeds.

I actually believe that there's likely legitimate use cases for adding *two* composer.json package files: one in the root of develop, and one at the root of src - letting end users pick the appropriate package for their environment. It's just not ideal that we might be duplicating the package file contents. Maybe this could just be another build step though, copied directly from develop root over to the core repo (with package name change) so nothing needs to be maintained in both locations. This could leave us with both a "wordpress/core" package, and a "wordpress/core-release" package.

At the least though, adding one in develop (outside of the document root) to begin with offers everyone everything they need including our custom WordPress unit testing framework (useful for plugins using composer to automate their own plugin unit tests), even if they only need the packaged release version of WordPress without tests or additional tools. Also, while the existing wordpress-plugin installer type defines a default location of wp-content/plugins, all end users have a way of overriding that path easily already, so providing the second composer.json package file right away isn't important.

comment:54 TJNowell6 months ago

  • Cc contact@… added

bpetty6 months ago

comment:55 follow-up: bpetty6 months ago

Some notes about 23912.6.patch:

Since none of the proposed patches here even attempted to keep things simple without a dependency on not only composer/installer, but also a completely custom installer that would need to be maintained along with WordPress, I've pulled that out.

I've set the package type to "project" instead of the default "library". Composer actually doesn't treat this any different what-so-ever, however, they have reserved this "project" type for this type of package (as defined by Composer: "full fledged applications distributed as packages").

I've also moved ext-mysql suggestion over into 'requires' section as discussed.

And the main difference here is that I've run through all package suggestions as discussed here and in the IRC discussion (same discussion linked earlier), and painstakingly came up with incredibly clear and consise descriptions of why you would want to install those suggested packages (having checked usage of every extension individually throughout all of core).

Regarding individual package suggestions:

ext-hash: I left this out because we have a perfectly acceptable fallback. This would be in 'require' if we didn't have the fallback in compat.php, but there's no reason to suggest it. It's not any more efficient (performance is not a concern here), and it doesn't provide anything new that core uses even when it's available.

ext-zlib: I think this should probably just be in "require" (but I left it as 'suggest' in this patch), PclZip will just die() if it's not available (and the ZIP uploaders still don't even check if it's available first before offering an upload form). Also, it's used frequently in core for adding compression support for scripts, styles, RSS feeds, and the HTTP API. Both ZipArchive and PclZip use it during decompression of plugin, theme, language pack, and upgrade ZIPs.

ext-zip: This was never discussed anywhere, and not listed on anyone's WP-used extensions list, but ZipArchive is in fact used by the ZIP uploaders. PclZip is just the fallback if ZipArchive isn't available, but both still require ext-zlib. I'm still not sure if we should bother listing it though since even though I don't know how it performs in comparison to PclZip, it's rarely actually used, so performance isn't a big concern, and it doesn't provide any extra functionality that PclZip handles fine. It's in a similar situation to ext-hash.

Regarding XML package suggestions:

Core XML use is spread across 3 unique features: import/export, oEmbeds, and SimplePie RSS. ext-libxml is the base for all XML methods, and is required for any of the above. However, each of these has their own additional requirements on top of ext-libxml.

Import/export works best with either ext-simplexml or ext-xml, but will still work and fall back to an extremely rudimentary (and horribly inefficient) regex parsing method if neither are available.

oEmbeds absolutely require both ext-dom and ext-simplexml (this could be reduced to one *or* the other, but that's a patch for another day), or they won't work unless the specific oEmbed used is making use of JSON for their API.

SimplePie requires ext-xml, and won't work otherwise (correct me if I'm wrong Ryan).

So, we come up with a total of 4 suggested XML extensions for optimal core use: ext-dom, ext-libxml, ext-simplexml, and ext-xml. I think I've clearly summarized each in composer.json.

comment:56 TJNowell6 months ago

Can you explain how I would write a composer.json that would include WordPress and a plugin as a dependency using your composer.json? The composer installers dependency is pretty unavoidable unless you're also planning to teach WordPress to search the vendor folder for plugins and themes

comment:57 bpetty6 months ago

As Ryan has also expressed, there really is no context under which you would include WordPress as a package dependency through Composer. It's not a library, and none of it can even be loaded by the autoloader, which is how Composer dependencies are designed to be included without relying on known installation paths. This is why John's patches require custom hacks to the Composer installer package, it was not designed to be used this way.

The only context under which WordPress makes sense as a Composer package is as a "project" that can be installed using "create-project" with dependencies on additional plugins and themes added after the fact. Speaking of plugins and themes, the existing "wordpress-plugin", "wordpress-theme", and "wordpress-muplugin" installer types built into the official composer/installer and already being used by many plugins and themes actually work with the core package when used this way by default. They don't with any of John's patches without additional configuration required by end users for every single plugin and theme added.

In the end, WordPress is one of these legacy applications that was never designed to be used with Composer. It's not worth the uphill battle with custom Composer hacks to get it to work the way you want it to with WordPress as a dependency in it's current form. If WordPress wasn't a legacy application, it would provide a skeleton project that you would still install using "create-project" and add to your own VCS just as I've outlined above that would handle routing and dispatch, and autoload the rest of the core components of WordPress that it needed as libraries just like any other plugin. Maybe someday we'll get there years down the road, but if we use this installer hack, we won't because we won't have the forward compatibility to do it.

For now, we simply treat WordPress as a skeleton project because that's what it is until individual components can be split out into an autoloaded library, without any direct routing requirements, who's updates can be maintained either through WordPress or through Composer depending on user configuration. This doesn't lock us out of opportunities to take advantage of a more optimal architecture down the road.

comment:58 TJNowell6 months ago

WP Core is, and is used as a dependency, be it via composer, svn external, git submodule, or manually downloaded and put in place. It's a dependency of every WordPress website in existence by virtue of being a WordPress website. Exactly how 'modular' or 'legacy' that dependency is, and whether it should use autoloaders, is irrelevant to this discussion. Especially when some composer dependencies contain no PHP of any kind, yet are actively used

Johns stuff works, I've ran test cases of deploying sites and keeping them up to date, with no issues. I didn't have to declare that I required the core installer either, it was pulled in automatically, and I had no issues pulling in plugins either, especially with the help of wpackagist.org.

The biggest boilerplate I saw, was trying to pull in the standard twentythirteen theme:

{
    "name": "tomjn/examplesite",
    "repositories": [
        {
            "type": "composer",
            "url": "http://wpackagist.org"
        },
        {
            "type": "package",
            "package": {
                "name": "wordpress/twentythirteen",
                "version": "1.0",
                "type": "wordpress-theme",
                "source": {
                    "url": "http://themes.svn.wordpress.org/",
                    "type": "svn",
                    "reference": "twentythirteen/1.0/"
                }
            }
        }
    ],
    "require": {
        "johnpbloch/wordpress": ">=3.6.1@stable",
        "wpackagist/mp6": ">=2.0",
        "wordpress/twentythirteen": "*",
        "wpackagist/spots": "*"
    },
    "minimum-stability": "dev"
}

Keep in mind I can also do this:

{
    "name": "tomjn/examplesite-with-akismet",
    "repositories": [
        {
            "type": "git",
            "url": "http://somegithubrepository/with/the-above/composer.json"
        }
    ],
    "require": {
        "tomjn/examplesite": "*",
        "wpackagist/akismet": "*"
    },
    "minimum-stability": "dev"
}

comment:59 TJNowell6 months ago

I would also take note of a similar file, doing a similar job that gets used by a tool with the same purpose as Composer albeit written in javascript:

http://develop.svn.wordpress.org/trunk/package.json

By your logic, the introduction of grunt and its dependencies signifies that WordPress javascript has no legacy components, and that future development of WP Core JS will be dictated by the Grunt community...

comment:60 in reply to: ↑ 55 rmccue6 months ago

Replying to bpetty:

Some notes about 23912.6.patch:

I am +1 on this patch.

ext-zlib: I think this should probably just be in "require" (but I left it as 'suggest' in this patch), PclZip will just die() if it's not available (and the ZIP uploaders still don't even check if it's available first before offering an upload form). Also, it's used frequently in core for adding compression support for scripts, styles, RSS feeds, and the HTTP API. Both ZipArchive and PclZip use it during decompression of plugin, theme, language pack, and upgrade ZIPs.

+1 on requiring this.

SimplePie requires ext-xml, and won't work otherwise (correct me if I'm wrong Ryan).

Correct, although it will use ext-dom in other places (not used by core) if available, as well as ext-simplexml. XML is such a pain.


Replying to bpetty:

As Ryan has also expressed, there really is no context under which you would include WordPress as a package dependency through Composer. It's not a library, and none of it can even be loaded by the autoloader, which is how Composer dependencies are designed to be included without relying on known installation paths.

+1. My issue is that adding anything else is trying to make WordPress something it's not, and trying to use Composer to handle everything. A library is decidedly not a project and vice versa, so they naturally have different handling.


Replying to TJNowell:

I would also take note of a similar file, doing a similar job that gets used by a tool with the same purpose as Composer albeit written in javascript:

http://develop.svn.wordpress.org/trunk/package.json

Note that this is part of the build tools, not part of the project itself, so it's not really relevant. Grunt isn't part of WordPress any more than PHPUnit is, they're just both part of our development process.

comment:61 johnpbloch6 months ago

-1 on the latest patch.

Ryan (and bpetty, it seems) and I do not see eye to eye on WordPress' role regarding WordPress as a composer package. He sees WP only as a base package and I see it as a library. I don't see this changing and I don't see any point in wasting any more of everybody's time and energy on this discussion.

comment:62 Rarst6 months ago

-1 on "plain" patch for the record.

comment:63 TJNowell6 months ago

My worry is that this composer.json patch being put forward will make it into core, and then people like myself will have to resort to a raft of horrendous kludges and hacks to undo what's in the core composer.json. This latest patch is a step backwards, and I would prefer to have no composer.json at all if this patch is the alternative

comment:64 rmccue6 months ago

  • Keywords has-patch dev-feedback removed

We had a bit of a discussion on IRC about this, and I think the best thing to do for now is let this lie for a bit. Let's see what happens in the wider ecosystem (Drupal 8, for example) with this sort of stuff before making a decision in any direction.

comment:65 lkraav6 months ago

  • Cc leho@… added

comment:66 tomdxw6 months ago

  • Cc tom@… added

comment:67 tamlyn5 months ago

  • Cc tamlyn@… added

comment:68 follow-up: gboone3 months ago

New to these forums as I am, I want to drop in and restart this conversation. I agree with those saying WordPress Core is most certainly a dependency and any composer.json should declare it as such. I'm not sure how you would create a WordPress-powered site without WordPress.

My worry is that this composer.json patch being put forward will make it into core, and then people like myself will have to resort to a raft of horrendous kludges and hacks to undo what's in the core composer.json.

Agree 100%, composer is a brilliant tool, and this is a huge opportunity for WordPress, we need to do it right.

Let's see what happens in the wider ecosystem…before making a decision.

Also agree that we don't want to move in the opposite direction of other platforms, but WordPress is big enough now, and being used on large enough scales that we could be the project that sets those standards rather than +1s them. Though I don't have a patch, yet, I hope we're still thinking about this.

comment:69 in reply to: ↑ 68 ; follow-up: bpetty3 months ago

Replying to gboone:

New to these forums as I am, I want to drop in and restart this conversation. I agree with those saying WordPress Core is most certainly a dependency and any composer.json should declare it as such. I'm not sure how you would create a WordPress-powered site without WordPress.

No-one is saying you can build WordPress sites without WordPress, we're saying that WordPress is an application, not a library, and as such, it can't be defined as a dependency because if you use Composer to install WordPress, you're supposed to do so using the create-project command, and defining your extra plugin dependencies in your local composer.json file.

This was how MediaWiki defined their package, but since you started this conversation again, I have to point out that MediaWiki has actually since decided (on Nov 1st) that they're only going to package a stripped down and incomplete composer-example.json template designed for end-administrators to copy to their own custom composer.json file and customize with their dependencies.

They even note that "core no longer pretends to be a component that is installable via composer. This never worked, and core will need to be modified in several ways before this can work." This is exactly the same situation WordPress is in.

I still believe we should package an actual composer.json file since just shipping a template doesn't actually help anyone, and you wouldn't even be able to use create-project without it either, but this discovery by the MediaWiki team only helps support my point in regards to incorrectly defining an application as a library that can be used as a dependency.

Also agree that we don't want to move in the opposite direction of other platforms, but WordPress is big enough now, and being used on large enough scales that we could be the project that sets those standards rather than +1s them.

Requiring a custom Composer installer would be moving in the opposite direction of other platforms, no-one else is doing it (with the intention of using it as a permanent, widely distributed, and encouraged solution), and it's entirely non-standard. We wouldn't be defining any "new standard", we'd just be the weird project using Composer in a way that it wasn't designed to be used.

comment:70 in reply to: ↑ 69 ; follow-ups: JohnPBloch3 months ago

Replying to bpetty:

No-one is saying you can build WordPress sites without WordPress, we're saying that WordPress is an application, not a library, and as such, it can't be defined as a dependency because if you use Composer to install WordPress, you're supposed to do so using the create-project command, and defining your extra plugin dependencies in your local composer.json file.

That is how you would install it. I install WordPress in a subdirectory a la Jaquith's WP Skeleton set-up. When I deploy a WordPress site, the site is the main package, with WordPress usually two directories deep (webroot/wp). Using composer create-project makes no sense at all for that workflow. Running WordPress in a subdirectory has been a normal way of setting it up for a long time and has only become more so with the addition of support for multisite in a subdirectory.

I have a personally maintained fork of WordPress with a composer.json file added on packagist. Go to your terminal and type composer create-project johnpbloch/wordpress:~3.8 wp38 and you'll have WordPress core in wp38. The WP-as-a-library way of thinking supports your use-case of WP-as-root-package-only, but your approach precludes our way of using it (including the WP-in-a-subdirectory installation).

They even note that "core no longer pretends to be a component that is installable via composer. This never worked, and core will need to be modified in several ways before this can work." This is exactly the same situation WordPress is in.

Preposterous. This is not at all the situation WordPress is in. I currently deploy several sites with composer and all of them do so

  1. With WordPress as a dependency of the main site package
  2. Without modifying any core file

This is possible because core has made it possible for WordPress to redefine where wp-content is and to look one directory up for wp-config.php.

Requiring a custom Composer installer would be moving in the opposite direction of other platforms, no-one else is doing it (with the intention of using it as a permanent, widely distributed, and encouraged solution), and it's entirely non-standard. We wouldn't be defining any "new standard", we'd just be the weird project using Composer in a way that it wasn't designed to be used.

To be fair, we're also the weird project that still supports back to PHP 5.2 and doesn't care what the PHP community thinks in any other area EVER. So it's kind of silly that we seem to care here suddenly.


Ok. So let's revive the talks about adding a composer.json. I'm fine with that.

It'd be trivial to create a meta package that required both the core package and a core installer, so I am 100% on board with dropping the core installer as a requirement in the core composer.json file. If we drop that requirement, I'd ask that the type be kept as wordpress-core.

Regarding the placement of the composer.json file: I think that it needs to be a sibling to wp-content, wp-includes, wp-admin, etc. Automated repository parsers should be able to find things without being told how and that means using the traditional svn structure found in core.svn.wordpress.org. Which means the composer file needs to be in /trunk and all future /branch/X and /tag/X directories in http://core.svn.wordpress.org.

comment:71 TJNowell3 months ago

+1 to Johns suggestion.

The bpetty proposed composer.json is problematic because it forces all users of composer.json into a certain workflow. It makes it more difficult for people who use SVN externals and Git externals to switch, and it removes the ability for composer to update WP Core itself. This is not the case with the composer.json proposed by Rarst/John and in active use by numerous people already. Such a composer.json would make things more difficult for existing composer users, not less, and it would remove some of the benefits of using a tool such as composer

For people such as bpetty, etc, a composer.json that sits in a git repo that already contains WP Core, that references only plugins and themes would be more appropriate, and their reasoning is based on a workflow that is irrelevant to this issue.

Of note, Drupal 8 has a type of drupal-core, and matches Johns approach. Let this be the end of the "lets see what Drupal etc do" points, they've done it, we've seen it, lets move on.

comment:72 in reply to: ↑ 70 ; follow-up: rmccue3 months ago

Replying to JohnPBloch:

That is how you would install it. I install WordPress in a subdirectory a la Jaquith's WP Skeleton set-up. When I deploy a WordPress site, the site is the main package, with WordPress usually two directories deep (webroot/wp). Using composer create-project makes no sense at all for that workflow. Running WordPress in a subdirectory has been a normal way of setting it up for a long time and has only become more so with the addition of support for multisite in a subdirectory.

I use WP in a subdirectory as well, and I still think create-project is the correct approach. WordPress is not meant to be embeddable in other projects (trust me, I've done it, it's a bad idea), it *is* the project.

WP-in-a-subdirectory is still WP-as-root-package, you're just moving some files around.

Preposterous. This is not at all the situation WordPress is in. I currently deploy several sites with composer and all of them do so

  1. With WordPress as a dependency of the main site package
  2. Without modifying any core file

This is possible because core has made it possible for WordPress to redefine where wp-content is and to look one directory up for wp-config.php.

WP still can't be used inside another application without stuff breaking everywhere. That's the crux of my opposition to this: WordPress is not embeddable, so requiring it via Composer makes no real sense to me.

comment:74 in reply to: ↑ 73 rmccue3 months ago

Replying to wonderboymusic:

WP "can" be used inside another framework: http://scotty-t.com/2014/01/15/rethinking-blogs-at-the-new-york-times/

From the way I read that, the binding between the other framework and WP is over HTTP though, isn't it? Loading WP directly in code is a horrific idea (thanks to globals, and WP being non-reentrant in almost every place possible).

comment:75 rmccue3 months ago

Oh wow, just read that you're using output buffering there instead. That's insanely crazy, but pretty cool too. That said, WP will still cause problems when embedding.

comment:76 nacin3 months ago

  • Component changed from General to Build Tools

comment:77 Rarst3 months ago

Since it hadn't been mentioned while ticket was on ice - I want to note that https://packagist.org/packages/johnpbloch/wordpress package for WordPress core has been available for a while.

comment:78 in reply to: ↑ 72 JohnPBloch3 months ago

Replying to rmccue:

I use WP in a subdirectory as well, and I still think create-project is the correct approach.

It sounds to me like you're just interested in using Composer to download WordPress in this case. Might I suggest that any one of these tools are far more suited to that job?

svn export http://core.svn.wordpress.org/tags/3.8/ wordpress
wget http://wordpress.org/latest.zip; unzip latest.zip
wp core download --path=wordpress

WordPress is not meant to be embeddable in other projects (trust me, I've done it, it's a bad idea), it *is* the project.

WP-in-a-subdirectory is still WP-as-root-package, you're just moving some files around.

Nobody is talking about embedding WordPress in other projects. We're talking about the fact that for any WordPress powered site, the site as a whole is the main project and WordPress is a very important dependency of that site. As are all of the plugins on it, all the themes, etc. All of those themes and plugins also happen to have WordPress as a dependency.

WP still can't be used inside another application without stuff breaking everywhere. That's the crux of my opposition to this: WordPress is not embeddable, so requiring it via Composer makes no real sense to me.

That is the best news I've heard in this thread. Let me be the first to happily tell you: You are fighting a straw man.

The crux of the issue for those of us arguing for WordPress as a dependency is that we want to be able to use Composer to manage the WordPress version for our sites that already use Composer. This is not in any way to say that the site isn't purely a WordPress site.

To get an idea of what this looks like, check out my WordPress package project:

composer create-project johnpbloch/wordpress-project:~0.1

comment:79 TJNowell3 months ago

There's a fundamental misunderstanding here, we're not discussing WordPress as a code library to be integrated into another piece of code.

E.g. if the package example/example.com depends on wordpress/wordpress and wordpress/akismet, and its repository contains a wp-config.php and a htaccess, how is this bad? There's no output buffer library binding trickery being employed, the end result is a standard WordPress subdir install

Using Rarsts/Johns composer.json this is perfectly fine. WordPress is a dependency of the site because it powers the site, not because it's being included in some other application.

The alternative composer.json would have the following impacts:

  • the example.com maintainer would be forced to use a classic install, a subdirectory install via composer is no longer an option
  • example.com can still update their theme and plugin packages, but if say wordpress 3.9 stable was released, they need to checkout, setup, run the updater, commit changes, push back to original source, etc, etc The ability to update and maintain versions of WP Core in a managed way is lost

Neither of these points have been addressed, and are fatal flaws in a composer.json that uses project as a type. I don't like the idea of being forced to treat all my websites as forks of WP Core, they're not, they're aggregations of code I've written ( themes ), code other people have written ( plugins ), and a CMS ( WP Core ), aggregations that are specified using a composer.json.

comment:80 in reply to: ↑ 70 ; follow-up: bpetty3 months ago

Replying to JohnPBloch:

That is how you would install it.

It's how the official installation instructions tell users to install it.

Running WordPress in a subdirectory has been a normal way of setting it up for a long time and has only become more so with the addition of support for multisite in a subdirectory.

Normal, yes. Officially supported, yes. Officially recommended or more common though, no. In fact, I know this isn't how more than 5% of WordPress installations are installed because this isn't how users are instructed to install it, and it's not how any single-click installers provided by any hosting companies install it. It's also not the way WordPress is configured to run by default.

Let's try and keep our primary user base in mind, and our primary user base isn't defining a custom WP_CONTENT_DIR setting with most installations.

It'd be trivial to create a meta package that required both the core package and a core installer, so I am 100% on board with dropping the core installer as a requirement in the core composer.json file.

This is certainly one of the main reasons I'm also against the custom core installer requirement. It's perfectly possible to create 3rd party meta packages that overwrite the default package properties or installation routines for less common configurations. Both yourself and Rarst already have.

If we drop that requirement, I'd ask that the type be kept as wordpress-core.

I feel like I'm running in circles on this one, but maybe I still have to remind everyone that the primary maintainer of the official Composer installer package has recommended against doing this, and not just for WordPress (it's been requested several times before already with no progress either for drupal-core or for fuel-core as well). To quote him again:

core types would exist for only 1 package where other types exist for many packages. I understand core types seem really useful but they seem problematic to me. As soon as 2 packages use the same core type it is going to blow up.

That's not meant to say that people aren't capable of being smart about not writing additional packages using the same type, but more meant to clarify that the type property was not designed to be used for this purpose, and that using it like this imposes limitations for developers building tools designed to work with it's original intended purpose. And again, he recommends the same thing I'm trying to make clear here:

I'm also reluctant because it seems create-project exists to handle this use case.

It's obvious that using this wordpress-core type makes it possible to do things we couldn't before, but this isn't a question of what's possible, it's a question of what's appropriate. It's about using a properly designed architecture both for current use and for keeping future compatibility in mind in case WordPress ever makes the decision to redesign it's architecture in a way that's more "composer compatible" like many other projects have. If we make use of a custom type and/or a dependency on a custom installer, we're burning some of the bridges ahead of us that we might want to use in the future (or at the least, making it much more difficult to cross them).

Right now, WordPress doesn't even support upgrading WordPress from Composer any more than it supports upgrading from VCS. It doesn't perform the same safety and security checks, it doesn't perform any DB migrations (and yes, I understand these will just happen on the next wp-admin request after an admin has been asked to), it doesn't call any plugin or theme hooks designed to be triggered by core (or plugin and theme) updates, and it doesn't flush any caches that should be flushed. It's not calling any core upgrader methods. It's hopefully true that most developers understand this when they manage WP updates from a VCS, and perform those tasks manually, but I want to stress that we should avoid giving users the impression that they can use Composer to manage upgrades for them just as reliably as using the built-in upgrader since offering Composer support opens that user base significantly to non-developers, who aren't using VCS, but are being exposed to those same developer-related limitations and problems.

Regarding the placement of the composer.json file: I think that it needs to be a sibling to wp-content, wp-includes, wp-admin, etc.

Yep. Not arguing that. My patch didn't include it there, but I did say that I think it should be there. I just want to see it done in a way that we avoid maintaining mostly duplicate package files in the develop root as well as the src root (i.e. building an automated grunt task that generates the src version from the main root package file with one simple package name change).

comment:81 follow-up: TJNowell3 months ago

It's obvious that using this wordpress-core type makes it possible to do
things we couldn't before, but this isn't a question of what's possible,
it's a question of what's appropriate.

In which case it's just as appropriate as using git submodules and externals. Also keep in mind that clients can demand these kinds of installs, and there are real advantages to putting WP Core in its own folder.

It's about using a properly
designed architecture both for current use and for keeping future
compatibility in mind in case WordPress ever makes the decision to
redesign it's architecture in a way that's more "composer compatible" like
many other projects have. If we make use of a custom type and/or a
dependency on a custom installer, we're burning some of the bridges ahead
of us that we might want to use in the future (or at the least, making it
much more difficult to cross them).

I fail to see how wordpress-core will be an issue. If in future we change the composer.json to 'moomin-spectacular', composer will re-arrange things as necessary into the vendor folder, at which point all that would be needed is an index.php that loads the autoloader and creates the first object. We may not change WP Core at all and use classmaps etc Who knows.

In practice, having WP Core as a composer dependency has not been an issue. New versions of WordPress have came along and been updated and installed accordingly.

Right now, WordPress doesn't even support upgrading WordPress from
Composer any more than it supports upgrading from VCS.

Actually that's not true. I can say that I need WordPress in my sites composer, specifically stable WordPress, and only the 3.8.x branch. I can specify that I want an exact version, or that I want developer builds, and I can say in my plugins that they need a specific version of WordPress, and composer will handle all of this. Git submodules do not. SVN Externals do not.

It doesn't perform
the same safety and security checks it doesn't perform any DB migrations
(and yes, I understand these will just happen on the next wp-admin request
after an admin has been asked to),

It's not intended for DB migrations. Neither are Git and SVN, or Zip/tarballs on the download page. This is a strawman.

it doesn't call any plugin or theme
hooks designed to be triggered by core (or plugin and theme) updates, and
it doesn't flush any caches that should be flushed.

Neither do svn/git updates, FTP'ing plugins manually, rsync, or extracting zip archives over the top of the filesystem.

The difference here is that composer has a scripts section that can load a WordPress environment and do the necessary work. As a user of Johns composer installer, this has never been an issue, and I would be suspicious of any plugin that breaks as a result of this, however unlikely that is. Even then, a composer.json of that plugin can be set to execute necessary cleanup and upgrade scripts.

It's not calling any
core upgrader methods. It's hopefully true that most developers understand
this when they manage WP updates from a VCS, and perform those tasks
manually, but I want to stress that we should avoid giving users the
impression that they can use Composer to manage upgrades for them just as
reliably as using the built-in upgrader since offering Composer support
opens that user base significantly to non-developers, who aren't using
VCS, but are being exposed to those same developer-related limitations and
problems.

So your argument is that by adding a file to enable the use of a developer-centric command line tool, we're opening ourselves up to a significant userbase of non-developers.

Composer managed WordPress installs are proven to work and deployed on live websites, and it's likely a developer choosing to use composer knows a great deal more about WordPress and PHP environments than a user who clicked on an install button on a shared host, who doesn't know what PHP is, nevermind a PHP CLI tool.

comment:82 JohnPBloch3 months ago

I think I've been letting my ego get into this discussion too much and I need to apologize for that. Things are starting to get kind of nasty and I think we need to rein things in a bit.

I think I'm coming around to the argument that bpetty's patch is the preferable one, for the most part, assuming the composer.json file ended up in trunk, branches/X and tags/X directories of the core.svn repo.

I don't think discussions of workflows and use-cases are really germane to this ticket. A basic clean composer.json file very similar to bpetty's would serve all workflows and use cases (perhaps with other packages required, depending on your needs).

What we should be discussing is:

  1. Whether there even should be a composer.json file in the root of the WordPress package (I think most or all of us agree that there should be?)
  2. What should be in the composer.json file

Regarding the second point, I'm fine with the contens of bpetty's patch with one exception: I think we should omit the type field. It's optional to begin with and there seems to be a significant amount of disagreement about whether we should claim that the WordPress core package is a project or a library (and to be fair, it doesn't really fit either label). We don't have to pick one.

comment:83 in reply to: ↑ 80 ; follow-up: JohnPBloch3 months ago

Replying to bpetty:

Replying to JohnPBloch:

Regarding the placement of the composer.json file: I think that it needs to be a sibling to wp-content, wp-includes, wp-admin, etc.

Yep. Not arguing that. My patch didn't include it there, but I did say that I think it should be there. I just want to see it done in a way that we avoid maintaining mostly duplicate package files in the develop root as well as the src root (i.e. building an automated grunt task that generates the src version from the main root package file with one simple package name change).

Could you elaborate please? I'm not sure what you mean RE: generates the src. version and package name change.

comment:84 TJNowell3 months ago

I'm happy to ommit the type element, a type of 'project' is what I oppose, the rest of the file is fine.

comment:85 JohnPBloch3 months ago

To clarify, since it might cause confusion, I changed my stance because, ultimately, I agree with bpetty that my workflow of a subdirectory installation of WordPress is not the norm. The composer.json file should reflect that. If I have a workflow I like or think is great, it's my responsibility to make it work. Simple as that.

comment:86 Rarst3 months ago

Let me rehash current state of ticket's goals as I see them.

From the position of WordPress developer, considering making use of Composer workflow, I fully expect it to keep up with any default/non-default/esoteric/whatever configuration that core itself can.

From the position of PHP developer, making use of WordPress as Composer package, I fully expect it to work in line with normal Composer mechanics without artificial restrictions and non-standard workarounds.

I see composer.json configuration mandating project-type install as inadequate to both goals.

I see current out-of-core progress on core installer and core package as adequate to both goals.

Given heavy opposition to including installer configuration into core, without offering adequate replacement at this time, I would prefer this ticket go back to sleep until such replacement can be offered or arguments against installer approach reconciled.

comment:87 in reply to: ↑ 81 ; follow-up: bpetty3 months ago

Replying to TJNowell:

There's a fundamental misunderstanding here, we're not discussing WordPress as a code library to be integrated into another piece of code.

Right, that's why we're saying it's not a dependency whose updates can be managed by composer, and shouldn't be treated like one (at least, by default anyway). This is how libraries are managed, not applications.

E.g. if the package example/example.com depends on wordpress/wordpress and wordpress/akismet, and its repository contains a wp-config.php and a htaccess, how is this bad? There's no output buffer library binding trickery being employed, the end result is a standard WordPress subdir install

Using Rarsts/Johns composer.json this is perfectly fine.

And this also still works with my proposed composer.json, it just requires an extra meta-package to implement your customizations to do this non-standard installation.

  • the example.com maintainer would be forced to use a classic install, a subdirectory install via composer is no longer an option

This is still possible.

  • example.com can still update their theme and plugin packages, but if say wordpress 3.9 stable was released, they need to checkout, setup, run the updater, commit changes, push back to original source, etc, etc The ability to update and maintain versions of WP Core in a managed way is lost

You're installed copy of WP will already contain the official core updater as usual, which is great because even if you're using John's custom installer, you're still seeing wp-admin upgrade notifications and functionality that still works as intended (which could even confuse Composer) as long as your files are writable (i.e. all shared hosting for one). And if you're implying that you're managing your own updates via your own commits from the official packages, you're saying you already have your own forked repo (which you were arguing against doing at the end here... so your argument is confusing here), in which case, this is definitely still perfectly possible even without a custom installer.

Replying to TJNowell:

It's obvious that using this wordpress-core type makes it possible to do
things we couldn't before, but this isn't a question of what's possible,
it's a question of what's appropriate.

In which case it's just as appropriate as using git submodules and externals. Also keep in mind that clients can demand these kinds of installs, and there are real advantages to putting WP Core in its own folder.

I'm not arguing the merits of subfolder installations. I'm arguing the merits of abusing Composer package properties for purposes they weren't designed to be used for, and what that means for future changes.

I fail to see how wordpress-core will be an issue. If in future we change the composer.json to 'moomin-spectacular', composer will re-arrange things as necessary into the vendor folder

A default installation directory change on update is by far one of the biggest backwards incompatible changes we could possibly make. Why do you think we went through so much work building an entirely new SVN repository for core rather than just moving all of WP into a src subdirectory on core.svn.wordpress.org?

Right now, WordPress doesn't even support upgrading WordPress from
Composer any more than it supports upgrading from VCS.

Actually that's not true. I can say that I need WordPress in my sites composer, specifically stable WordPress, and only the 3.8.x branch. I can specify that I want an exact version, or that I want developer builds, and I can say in my plugins that they need a specific version of WordPress, and composer will handle all of this. Git submodules do not. SVN Externals do not.

I think you missed my point here, and I'm really not sure where you went with it.

P.S. SVN externals do support that though.

Neither do svn/git updates, FTP'ing plugins manually, rsync, or extracting zip archives over the top of the filesystem.

Exactly, but we still encourage users to upgrade through wp-admin first when possible. When using WP as a composer dependency though, you're telling users not to, even when they could be upgrading through wp-admin (and we're still spitting out upgrade notices to them in the admin panel, just contradicting own recommendations at that point).

The difference here is that composer has a scripts section that can load a WordPress environment and do the necessary work. As a user of Johns composer installer, this has never been an issue

This is exactly what I mean. I'm assuming you would even call yourself a "developer", but you've just made the same mistake I expect regular users will constantly make. I've been through John's custom installer code and patches extensively, and while you're right that it's possible to trigger all sorts of scripts on install/update, not even John's package does any of the above. If you didn't manually flush any object caches, trigger DB migrations, or trigger your installed plugin/theme upgrade hooks manually, your installation was not officially in a properly upgraded state. WP usually finds ways around most of this without causing serious errors or problems (like you said, users do this all the time already with FTP/ZIPs), but these are supposed to happen, and they don't if you upgraded using John's composer package file.

So your argument is that by adding a file to enable the use of a developer-centric command line tool, we're opening ourselves up to a significant userbase of non-developers.

It's just one additional supporting claim, not my primary reason though.

comment:88 in reply to: ↑ 83 bpetty3 months ago

Replying to JohnPBloch:

Replying to bpetty:

Yep. Not arguing that. My patch didn't include it there, but I did say that I think it should be there. I just want to see it done in a way that we avoid maintaining mostly duplicate package files in the develop root as well as the src root (i.e. building an automated grunt task that generates the src version from the main root package file with one simple package name change).

Could you elaborate please? I'm not sure what you mean RE: generates the src. version and package name change.

There's good reasons for having two composer.json files because we have two root folders for WP: (1) the root of develop.svn.wordpress.org/trunk, and (2) the root of core.svn.wordpress.org/trunk (or just develop.svn.wordpress.org/trunk/build if you prefer to think about it that way).

The former is useful if you have good reasons for pulling in pre-compiled JS or (more likely) SASS/CSS, or if you need to pull in the unit testing framework (imagine plugins/themes with their own grunt tasks that use future wp-admin SASS mixins for example, or plugin unit tests run on Travis CI). The latter is obviously the main version most likely to be used though.

Regardless though, the build directory (and thus core.svn) is sort of "special" now anyway, and requires the composer.json file be copied over from src as it is already. Rather than do that though, I'm just thinking a Grunt task that copies it from the root of develop.svn.wp.org/trunk instead of src since we could use it there anyway, and just rename the package so either can be used the same way. Considering our current convention with WP version names with this changed came out with appending -src for develop.svn checkouts, I imagine this might end up with one package called wordpress/wordpress-core for the copy under build while the root develop.svn copy was wordpress/wordpress-core-src for example.

I just don't want to see duplicate composer.json files that are almost exactly the same under the src folder as the parent folder, that would require editing both files manually with any changes. That's all. We can do all this with Grunt, I'm just not bothering with the patch until we have the other issues here resolved.

comment:89 in reply to: ↑ 87 TJNowell3 months ago

Replying to bpetty:

Right, that's why we're saying it's not a dependency whose updates can be managed by composer, and shouldn't be treated like one (at least, by default anyway). This is how libraries are managed, not applications.

RPM, Deb, apt-get, aptitude, synaptic, homebrew, there are quite a few tools, some decades old, that do exactly that. There is even greater precedent for this than for doing it with libraries. These tools help define an environment.

  • example.com can still update their theme and plugin packages, but if say wordpress 3.9 stable was released, they need to checkout, setup, run the updater, commit changes, push back to original source, etc, etc The ability to update and maintain versions of WP Core in a managed way is lost

You're installed copy of WP will already contain the official core updater as usual, which is great because even if you're using John's custom installer, you're still seeing wp-admin upgrade notifications and functionality that still works as intended (which could even confuse Composer) as long as your files are writable (i.e. all shared hosting for one). And if you're implying that you're managing your own updates via your own commits from the official packages, you're saying you already have your own forked repo (which you were arguing against doing at the end here... so your argument is confusing here), in which case, this is definitely still perfectly possible even without a custom installer.

I do not have a forked repo of WP Core and have no desire to. Composer reads the appropriate repositories for information on versions and branches. I also have no desire for a wp-admin or wp-includes that can be written from a web accessible/ran script. This is a security concern, and one that clients and server administrators have and will point it in future if it's allowed.

I can appreciate that this is not normal, but it is done by a nontrivial number of people, and offers significant security benefits.

Replying to TJNowell:

It's obvious that using this wordpress-core type makes it possible to do
things we couldn't before, but this isn't a question of what's possible,
it's a question of what's appropriate.

In which case it's just as appropriate as using git submodules and externals. Also keep in mind that clients can demand these kinds of installs, and there are real advantages to putting WP Core in its own folder.

I'm not arguing the merits of subfolder installations. I'm arguing the merits of abusing Composer package properties for purposes they weren't designed to be used for, and what that means for future changes.

People can already 'abuse' WordPress core using composer without any composer.json at all, be it via VCS externals, or manually defined packages in the remote composer.json. I could declare in my satis repo that WordPress is of type drupal-core if I wish, it will happen, and there is nothing you can do to stop it. Having said that I would be foolish to do so. This is a slippery slope argument and mostly unfounded. The people who will do what you fear are already doing it.

I fail to see how wordpress-core will be an issue. If in future we change the composer.json to 'moomin-spectacular', composer will re-arrange things as necessary into the vendor folder

A default installation directory change on update is by far one of the biggest backwards incompatible changes we could possibly make. Why do you think we went through so much work building an entirely new SVN repository for core rather than just moving all of WP into a src subdirectory on core.svn.wordpress.org?

I have faith the core developers will not suddenly change composer.json and the folder structure on a whim, and just because composer can do it, doesn't mean we should or will.

Right now, WordPress doesn't even support upgrading WordPress from
Composer any more than it supports upgrading from VCS.

Actually that's not true. I can say that I need WordPress in my sites composer, specifically stable WordPress, and only the 3.8.x branch. I can specify that I want an exact version, or that I want developer builds, and I can say in my plugins that they need a specific version of WordPress, and composer will handle all of this. Git submodules do not. SVN Externals do not.

I think you missed my point here, and I'm really not sure where you went with it.

P.S. SVN externals do support that though.

SVN won't tell me if my plugin dependency needs WordPress 3.9 developer but only 3.8 stable is available. SVN won't let me upgrade to any arbitrary version of a plugin between 2.3 and 2.6 without needing the plugin author to maintain a branch. SVN won't tell me when a theme declares that it is incompatible with a plugin. SVN has its limitations, and SVN externals and composer dependency management are not equivalant.

This is exactly what I mean. I'm assuming you would even call yourself a "developer", but you've just made the same mistake I expect regular users will constantly make. I've been through John's custom installer code and patches extensively, and while you're right that it's possible to trigger all sorts of scripts on install/update, not even John's package does any of the above. If you didn't manually flush any object caches, trigger DB migrations, or trigger your installed plugin/theme upgrade hooks manually, your installation was not officially in a properly upgraded state. WP usually finds ways around most of this without causing serious errors or problems (like you said, users do this all the time already with FTP/ZIPs), but these are supposed to happen, and they don't if you upgraded using John's composer package file.

Composer does not activate plugins, it merely installs them. Activation and deactivation hooks etc are still present and will still run. What kind of plugin would run a database migration on installation prior to its activation? Or execute any code prior to activation? I would be very wary of such plugins, using the built in installer or not.

The only case you could put forward here is uninstall/deletion hooks, but even then, Johns core installer wouldn't do this because his core installer is not for installing plugins and themes, it's for installing WP Core.

So your argument is that by adding a file to enable the use of a developer-centric command line tool, we're opening ourselves up to a significant userbase of non-developers.

It's just one additional supporting claim, not my primary reason though.

I don't think you grasped what I was trying to say. Composer is a developer tool, a Command line tool, how does this open us up to a userbase of non developers? I doubt a non-developer would be able to write their composer.json, fire up a terminal, change to the right folder, and plug in the appropriate commands and parameters. It makes no sense.

There's good reasons for having two composer.json files because we have two root folders for WP: (1) the root of develop.svn.wordpress.org/trunk, and (2) the root of core.svn.wordpress.org/trunk (or just develop.svn.wordpress.org/trunk/build if you prefer to think about it that way).

I agree, there should be 2 separate composer.json files because of the differing folder layouts. We should only concern ourselves with the core.svn.wordpress.org composer.json here. develop.svn.wordpress.org has a different purpose, and the suggested and required packages may differ, it should be discussed in a new ticket.

Note: See TracTickets for help on using tickets.