Surveying system to gather user voices

[schlessera]

I'm still not sure whether the "democratize publishing" is part of the .org mission (the wordpressfoundation.org site and several dev posts around here seem to indicate it is), but we also have similar goals like "Design for the Majority" and the 80% rule that are relevant here as well.

Whenever you want to take an informed decision on behalf of someone else, that someone needs to be able to make his/her voice heard.

I think we're just taking the easy way out by saying that WordPress is open source and that everyone can contribute. That has nothing to do with a "democracy", but is a "meritocracy" inherited from the general open source world.

To have any real democratic merits or to be able to truthfully reference the "majority" in arguments, the WordPress project needs to have means to "query" the interested parties so that they can make their voice count.

This is why I would like to propose adding a voting/surveying system to the WordPress project.

I imagine something like a surveying system run on make.wordpress.org that could be tied into the WordPress backend as an opt-in system for end-users who want to participate as well. It should be able to distinguish between the different areas of contribution, so that people can choose one or more areas in which they would like to participate.

Right now, the end-users are disempowered and a select few speak on their behalf. I do not know whether these select few have gotten it right or not, and I am glad I don't have to take such decisions. But I do know that such arguments cannot easily be countered, as there are no facts to check or reasonings to dismantle.

I have not thought about any specific implementation yet, I just want to launch the idea of introducing a system of surveying contributors and end-users.

NOTE: I do not intend to introduce any kind of real political system here. Voting/surveys should not be binding in any way, it should just provide a means to collect data about "what the majority wants" in a targeted and meaningful way. As of now, such data is missing (and I don't consider the server stats to be any meaningful representation), and more elaborate discussions end up being "vetoed" by the "majority" argument without any means to back up or refute such claims.

[schlessera]

Further elaborations on how this ticket came about: ​https://www.alainschlesser.com/on-wordpress-and-democracy/

For those of you who do not want to click through the link, here's the summary:

As there is no "hard" leadership that just takes decisions on more progressive topics, there's often a magical "majority" being referenced as the argument for keeping the status quo. For this to be meaningful in resolving a discussion, there needs to be a tool to fill this "majority" argument with statistically conclusive data, otherwise it is just the opinion of a select few.

[schlessera]

Right now, I think that the such a system should offer the following two mechanisms, right from the WordPress dashboard, and fed into wordpress.org database:

1. A list of planned/discussed features that people can vote on. This allows to prioritize development based on user feedback and lets users have an impact on the roadmap.
2. A kind of "referendum" system to post specific questions or a survey to the users that have opted in, to directly ask feedback for less obvious decisions.

Features for 1. should of course be grouped based on area/component.

For 2. users should be able to opt-in per area, so that they are not included for areas they are not interested in. As an example, when a decision needs to be made as to whether a given feature should be included in Core or not, a "referendum" would be created that is only shown to users that have opted-in to the "Core" contribution area. This should only be done in cases where there's no obvious right/wrong or good/bad. More specifically, it should always be done when something is decided "for the majority of users".

[jdgrimes]

Note that there is ​https://wordpress.org/ideas/, although I'm unsure whether any developers ever look at it (and it isn't particularly obvious to users either, I guess).

[helen]

Based on this ticket and your linked post, it seems that your premise is that you're uncertain if WordPress is a democracy or a meritocracy, and that mentions of "the majority" imply some kind of democratic 1:1 voting system. Calling an open source project a meritocracy is loaded for good reason, but the reality remains that WordPress functions as a meritocracy. This is stated clearly in some places, including ​the security whitepaper.

"Design for the majority" is a philosophy and a goal, not some kind of literal process or a singularly decisive factor. I recognize that "the majority" constantly gets cited as an argument, but so do a lot of things - it does not inherently make that a good argument at all times. And don't forget, that same philosophies page that describes "design for the majority" contains a reminder about the vocal minority.

I don't want data collection efforts to center around what people want - certainly people's wants should be heard, but the data collected should be around what they do and how that informs things WordPress should do to better serve the needs that exposes. It seems that you agree with this at some level, and you know the vote you cite is not representative of "the people", so this ticket is extremely confusing to me.

That particular poll also exposes a frequent problem with votes, in that nuance and reasoning is lost. Do people want Composer in core because they want to be able to use something that's in core, or do they want it because core can actually use it in an effective way? Is this more of a want than, say, a PHP 5.3 baseline, and could it be a million times better enabled if that were addressed first?

What's missing in your specific instance is not a voting mechanism, it's a trusted owner to guide the process of discovering needs and defining outcomes. What we do about usage data more broadly in WordPress is a valuable conversation to have and has been happening in fits and starts, but it's not something for a Trac ticket. My suggestion is to focus on getting the specific project that is #36335 productively moving before worrying about something like votes. It probably needs regular meetings, contribution direction on GitHub or wherever, focused writing prompts as a means to obtain some qualitative data (e.g. "what would having Composer in WordPress core do for you, and can you show a specific example"), and so on.

If you need help wrangling an owner or owners, then please ask for that help - software maintainers aren't just decision makers, they should also be facilitators.

[schlessera]

Replying to jdgrimes:

Note that there is ​https://wordpress.org/ideas/, although I'm unsure whether any developers ever look at it (and it isn't particularly obvious to users either, I guess).

Wow, I have never seen that one before. But yes, that's already pretty much one of the things I described above. So, now, imagine having this be available from within the dashboard for users, and using the collected data to make informed decisions about the roadmap for the next releases...

Replying to helen:

Based on this ticket and your linked post, it seems that your premise is that you're uncertain if WordPress is a democracy or a meritocracy, and that mentions of "the majority" imply some kind of democratic 1:1 voting system. Calling an open source project a meritocracy is loaded for good reason, but the reality remains that WordPress functions as a meritocracy. This is stated clearly in some places, including ​the security whitepaper.

I know it is run as a meritocracy, and I indeed fully support this. It is pretty much one of the foundational principles of the entire open source movement.

I guess I have used terms that are too political to convey my message, and what's more, I'm myself not entirely sure whether what I propose would be a good or a bad idea.

The point is, when large teams collaborate, there are two ways to productively conclude a discussion:

1. Participants agree on a consensus.
2. An authority takes a decision.

The way I feel things are currently run, you either need to be able to achieve 1., or your discussions will turn in circles (or be postponed from release to release), as no-one can or will impose 2.

For 1., the discussion needs to follow clear argumentation, where all the arguments that are introduced by each of the sides can either be asserted or refuted. Only facts and hard data can be asserted or refuted, opinions can't. Arguments are asserted or refuted until only the valid ones are left, and these are then weighed against each other to determine whether it's an overall improvement or not.

This is where the problem lies in my eyes. The needs and wants of the end-users are cited as facts and used as arguments, but most of the time, they are just opinions, as there's no means to verify them. And in the cases where such non-facts are used as arguments, no consensus is possible, so only a final authoritative decisions could properly conclude the discussion. And, as far as I can tell, this is not what happens in most of the cases.

I know that user tests being made for UX changes, for example, and that server data being collected. But why not have a tool that allows us to query the users (that opted-in) directly? Why not let them tell you their biggest pain points? Why not let them help you define the priorities? I don't suggest just using that data and blindly following it. But I think that, in this case, more data is better than less data...

"Design for the majority" is a philosophy and a goal, not some kind of literal process or a singularly decisive factor. I recognize that "the majority" constantly gets cited as an argument, but so do a lot of things - it does not inherently make that a good argument at all times. And don't forget, that same philosophies page that describes "design for the majority" contains a reminder about the vocal minority.

Yes, I understand that this is not a literal process. But it is a philosophical goal that processes should be shaped by.

I don't want data collection efforts to center around what people want - certainly people's wants should be heard, but the data collected should be around what they do and how that informs things WordPress should do to better serve the needs that exposes. It seems that you agree with this at some level, and you know the vote you cite is not representative of "the people", so this ticket is extremely confusing to me.

Yes, as I said, additional data should just inform the decisions, not bind them.

That particular poll also exposes a frequent problem with votes, in that nuance and reasoning is lost. Do people want Composer in core because they want to be able to use something that's in core, or do they want it because core can actually use it in an effective way? Is this more of a want than, say, a PHP 5.3 baseline, and could it be a million times better enabled if that were addressed first?

Statistically speaking, that poll is complete nonsense, and not much more than a nice graphics, I'm fully aware of that. But against a "People don't want an autoloader" or similar argument, it's the best I could come up with. Should I have tried to run a normal survey with much more nuanced questions against the WP developer base as an unknown? How many responses do you think I would have gotten?

What's missing in your specific instance is not a voting mechanism, it's a trusted owner to guide the process of discovering needs and defining outcomes. What we do about usage data more broadly in WordPress is a valuable conversation to have and has been happening in fits and starts, but it's not something for a Trac ticket. My suggestion is to focus on getting the specific project that is #36335 productively moving before worrying about something like votes. It probably needs regular meetings, contribution direction on GitHub or wherever, focused writing prompts as a means to obtain some qualitative data (e.g. "what would having Composer in WordPress core do for you, and can you show a specific example"), and so on.

Yes, I agree that a Trac ticket is not a good fit, but I thought it would be good enough to get the discussion started. If such a discussion is already happening, then I am sorry for duplicating efforts, as I am not aware of it, and would be happy if you can point me to the correct spot and close this ticket.

Also, this was not merely pointing at that one ticket. I think that there's a more general sentiment of "not being heard" (by users/devs/specific communities/...) that would be alleviated by better "hearing tools"... ;)

If you need help wrangling an owner or owners, then please ask for that help - software maintainers aren't just decision makers, they should also be facilitators.

[schlessera]

In the light of more recent talks and posts on the web, I just now realised that I might be rubbing people completely the wrong way...

I want to clarify here that this is not another "Us vs Them" rant. I highly respect what all the core leads/committers/contributors are doing on a daily basis here (and I'm trying my best to be a small part of that), and they are doing a terrific job!

This is more of an "Us vs Us (with unneeded friction)" post, as I feel that we all ultimately want to make WordPress the best platform we can, and I think we're missing one or two important tools to make these efforts as effective as possible. I just hate wasting time on endless discussions that simply cannot be resolved without additional input/data.

I hope this clarifies the intent of my ticket, in case people took it as an offense.

[helen]

Replying to schlessera:

The point is, when large teams collaborate, there are two ways to productively conclude a discussion:

1. Participants agree on a consensus.
2. An authority takes a decision.

Two things: "consensus" means different things to people, and I think even with consensus, there is always a final decision-maker.

For 1., the discussion needs to follow clear argumentation, where all the arguments that are introduced by each of the sides can either be asserted or refuted. Only facts and hard data can be asserted or refuted, opinions can't. Arguments are asserted or refuted until only the valid ones are left, and these are then weighed against each other to determine whether it's an overall improvement or not.

Not all things can result in "facts and hard data" - opinions are not inherently invalid and should not be outright disregarded. Facts and data still need interpretation, which is what I frequently see being written off as "opinion". Besides, why would anecdotal evidence not qualify as some sort of data? It may need to be weighted as less reliable, but it's still information you have at hand.

I frequently find that circular arguments are happening not because of the actual information at hand, but rather the approach itself. Arguing against something is often far less productive than focusing on why something should be done.

But why not have a tool that allows us to query the users (that opted-in) directly? Why not let them tell you their biggest pain points? Why not let them help you define the priorities?

As stated, I don't want to have people tell me their biggest pain points - I want them to show me what they do and then interpret the data collected. It's not about binding decisions, it's about the kind of data that is valuable to collect in the first place. Admin usage data collection is a blessed project - I've even outlined an MVP. What it needs is the same as what I've advised you to find - an active project owner.

Statistically speaking, that poll is complete nonsense, and not much more than a nice graphics, I'm fully aware of that. But against a "People don't want an autoloader" or similar argument, it's the best I could come up with. Should I have tried to run a normal survey with much more nuanced questions against the WP developer base as an unknown? How many responses do you think I would have gotten?

1. "People don't want an autoloader" is so nonsensical as to be a non-argument IMO. I mean, really, "people want an autoloader" is just as true of a statement as "people don't want an autoloader" - it is not meaningful in any real way on its own. This isn't an argument around "wanting" an autoloader, it's about whether it makes sense for WordPress core to bundle and/or use one itself. What developers would gain from a bundled autoloader is important in assessing the value of the feature, but it's not about wanting the actual thing itself.
2. Yes, I think more valuable data in this instance is qualitative data around the why and how of how developers currently and would like to leverage Composer and autoloading in general, and how the two things do or don't fit with WordPress core. This is why you need a project owner and process. You could collect responses in a GitHub issue, or distribute a survey via word of mouth, or whatever - people need reassurance that it is actually being actively collected and interpreted, and that's what an owner and process should help define. Responses will always be limited and biased in some manner, fretting about that will only guarantee that you never have anything to get a response to at all.

I responded to your poll in the extreme small minority of "No, it's too early" because it was the closest response to "WordPress core doesn't seem to be structured in a way that actually makes sense with an autoloader and maybe a 5.3 baseline makes more sense first but I don't have any fixed opinion, so I guess this will let me see the results, in any case". That's another problem with those polls - there's no "other" and you can't view the results without choosing something, so who knows how many people just clicked one to get through (it'll bias toward the top option, which the audience itself is already biased toward). Why cite it at all?

Also, this was not merely pointing at that one ticket. I think that there's a more general sentiment of "not being heard" (by users/devs/specific communities/...) that would be alleviated by better "hearing tools"... ;)

While I heartily agree that WordPress core does not adequately focus on feedback mechanisms, I would be very cautious about the difference between "I'm not being heard" and "I'm not getting my way". If there's a sentiment that "my desire for an autoloader hasn't been heard", I think that is pretty clearly not true - it's the outcome that isn't satisfactory. That's not any less legitimate, but it's a very different thing to focus on improving.

[schlessera]

Replying to helen:

Two things: "consensus" means different things to people, and I think even with consensus, there is always a final decision-maker.

Yes, agree.

Not all things can result in "facts and hard data" - opinions are not inherently invalid and should not be outright disregarded. Facts and data still need interpretation, which is what I frequently see being written off as "opinion". Besides, why would anecdotal evidence not qualify as some sort of data? It may need to be weighted as less reliable, but it's still information you have at hand.

Yes, agree as well. When such an interpretation is written off as opinion, though, there's always rational arguments you can put forward to demonstrate the reasoning.

I frequently find that circular arguments are happening not because of the actual information at hand, but rather the approach itself. Arguing against something is often far less productive than focusing on why something should be done.

I have come to the conclusion time and time again that arguing against something is much easier (and this is probably in part because of my previous government work). I think this is similar to how theories are proven or disproven in science. Depending on your theory, not even the biggest numbers of positive findings can prove a theory to be true in every case, but a single negative finding is enough to disprove it. But this is rather anecdotal here.

But why not have a tool that allows us to query the users (that opted-in) directly? Why not let them tell you their biggest pain points? Why not let them help you define the priorities?

As stated, I don't want to have people tell me their biggest pain points - I want them to show me what they do and then interpret the data collected. It's not about binding decisions, it's about the kind of data that is valuable to collect in the first place. Admin usage data collection is a blessed project - I've even outlined an MVP. What it needs is the same as what I've advised you to find - an active project owner.

I understand what you are saying. However, the users can only show you what they do based on what you previously built and allowed them to do. As a (silly) example, if your UI is just a big red button labeled "PUSH ME!", your tests will reveal something like: "84% of users push the 'PUSH ME' button. That button is hugely important and a full success.".

Yes, this is a very simplified example, but I want to make a point. Watching users interact with your product is already very much biased in that they will only ever be able, for each of your functions, to either use it or not. How would such data be able to show you if something is completely amiss? Users have been trained and are forced to use what you've already decided to build for them before.

1. "People don't want an autoloader" is so nonsensical as to be a non-argument IMO. I mean, really, "people want an autoloader" is just as true of a statement as "people don't want an autoloader" - it is not meaningful in any real way on its own. This isn't an argument around "wanting" an autoloader, it's about whether it makes sense for WordPress core to bundle and/or use one itself. What developers would gain from a bundled autoloader is important in assessing the value of the feature, but it's not about wanting the actual thing itself.

Yes, totally agree. As I had already written in a response on that exact ticket, the end goal is not to have an autoloader be included, the end goal is to have a code base that is optimized to only load code when needed.

I responded to your poll in the extreme small minority of "No, it's too early" because it was the closest response to "WordPress core doesn't seem to be structured in a way that actually makes sense with an autoloader and maybe a 5.3 baseline makes more sense first but I don't have any fixed opinion, so I guess this will let me see the results, in any case". That's another problem with those polls - there's no "other" and you can't view the results without choosing something, so who knows how many people just clicked one to get through (it'll bias toward the top option, which the audience itself is already biased toward). Why cite it at all?

You're absolutely right in that PHP 5.3 should be the absolute first step when talking about any modernization of the code base. On my personal list of crucial changes, this was number 1, with the autoloader only being number 3.

However, after looking at a lot of discussions here, looking at the collected stats, and doing some research of my own, I've come to the conclusion PHP 5.2 compatibility will not be removed any time soon. The way I interpret the collected stats, the 5.2 sites do not contain many serious, active sites that are worth protecting, and their absolute number will not drop any further. The percentage was mostly decreasing because the overall number of installations was climbing, and this will soon hit a natural ceiling. So, my guess would be that we will not drop below an "acceptable" percentage threshold in the coming years, if the current criteria remains as is.

That's the exact reason why I initially started commenting on the autoloader ticket. I wanted to prevent designing an autoloader API for WP & plugins/themes when we don't even have namespaces yet. So, I actually agree that it is too early for an autoloader to be included in form of an API. However, we can't start soon enough refactoring the code base, as that is needed, with an autoloader or not. The autoloader is a nice way of making moving files around less painful.

While I heartily agree that WordPress core does not adequately focus on feedback mechanisms, I would be very cautious about the difference between "I'm not being heard" and "I'm not getting my way". If there's a sentiment that "my desire for an autoloader hasn't been heard", I think that is pretty clearly not true - it's the outcome that isn't satisfactory. That's not any less legitimate, but it's a very different thing to focus on improving.

Hehe, I guess my ticket does really come off as a "Bah, no one wants my autoloader"-whining, right?

But, truth be told, the input I contributed to particular ticket has garnered more attention than I expected, and the fact that there was even a Composer autoloader committed within Core, even if only briefly, is a bigger achievement for me personally than I could have hoped for. At the very least, people are more aware of the issues at play than before.

This ticket here however is not meant to have any impact on the autoloader one, which was merely the impulse that caused me to have a more "philosophical" moment.

The reflection was more along the lines of: "Wow, half of the discussion in that ticket could probably have been avoided if we would have better tools to get representative data!".

I just hate wasting time, and I not only optimize my code, but also my processes, and my life in general. The way that the discussions in some of the tickets evolve just doesn't feel right to me. I feel like dropped back into a three hour government meeting where everyone speaks, no-one listens _{and at the end there's neither agenda nor decisions.}

Maybe this is just me...

Anyway, thanks already for taking the time to discuss this in a productive way and trying to read between the lines, I really appreciate it. If you estimate that this ticket itself is just another waste of time, and that there's already all the tools we need to make meaningful progress, then please just close this ticket. I probably don't know half of the sources of data that are available to the Core team, so this could well be just a meaningless rant on my part.

[SergeyBiryukov]

Related: ​#meta833

[desrosj]

I'm going to close this one out as a maybelater. If this becomes a priority, we can reopen. It should also be accompanied by a Make post or two.