REST API: Increase upper bound allowed on per_page argument

[danielbachhuber]

In contexts where a REST API client needs to fetch all entries for a resource, it would be more practical to fetch entries in sets of 200, 300, or 400, instead of sets of 100. Fetching entries in sets of 100 can cause excessive memory usage because WordPress is loaded over and over again. Increasing the limit will provide a better balance between memory consumption in a single request vs. total memory consumption across all requests.

The original per_page=100 limit was ​somewhat arbitrary; if I recall correctly, we picked 100 as a nice round number that was reasonably certain not to cause performance issues.

Before we pick per_page=200 vs. per_page=300 vs. per_page=400, we should:

1. Profile memory consumption of each.
2. Identify what amount of memory we can reasonably expect on shared hosting these days.

After we've done this, we can pick the best available option.

Notably, we can't go above 500 as we'll hit split_the_query which has negative performance implications.

Previously ​https://github.com/WordPress/gutenberg/issues/6694

[desrosj]

I am working internally with a few team members at Bluehost to get some memory profiling and data based on our shared hosting configurations to help set a better-informed upper bound number.

[dd32]

Before we pick per_page=200 vs. per_page=300 vs. per_page=400, we should:

I think an important part of this is also to look at each end-point individually.

For example, requesting 10,000 terms at once might be viable, but the same for posts not so much.

It's going to be the edge-cases on the 0.001% of sites which will cause problems, those with hard upper 32M of memory and posts which are 1M of text each, etc. It may even be that we might need to set it at max 999 and have the client back-off and try again at 200 if the response isn't valid, etc.

Is there perhaps a smaller set of endpoints which we can focus on? Terms? Posts? Which paginated endpoints affect Gutenberg the most? Which endpoints is Gutenberg NOT handling the pagination for well? etc.

[danielbachhuber]

All good points, thanks @dd32.

Given the ​timeline for releasing 5.0 and that Gutenberg isn't blocked by this particular issue (as it ​iterates the endpoint pagination), let's go ahead and avoid making a rushed decision here. We can review in the future when there's research to evaluate.

[pbearne]

Does this still need to be done?