WordPress.org

Make WordPress Core

Opened 12 months ago

Closed 5 months ago

Last modified 5 months ago

#48821 closed enhancement (fixed)

Add additional array validation keywords to rest_validate_value_from_schema

Reported by: TimothyBlynJacobs Owned by: TimothyBlynJacobs
Milestone: 5.5 Priority: normal
Severity: normal Version: 4.7
Component: REST API Keywords: has-patch has-unit-tests
Focuses: Cc:

Description

JSON Schema describes minItems, maxItems and uniqueItems keywords to assist in validating array types: https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.6.4.1 I think this would be helpful to add support for.

Change History (25)

#1 @TimothyBlynJacobs
9 months ago

  • Milestone changed from Awaiting Review to 5.5

#2 @sorenbronsted
7 months ago

I will work on this

#3 @sorenbronsted
7 months ago

  • Keywords dev-feedback added

In connection with working with these validations, the method rest_validate_value_from_schema is getting rather long. I would like to propose that the validation is refactored into smaller functions, all called from rest_validate_value_from_schema. Any objections?

#4 @TimothyBlynJacobs
7 months ago

rest_validate_value_from_schema is definitely getting rather long, but let's tackle that in a separate ticket.

This ticket was mentioned in PR #253 on WordPress/wordpress-develop by sorenbronsted.


7 months ago

Added min,max and unique to items

Trac ticket: https://core.trac.wordpress.org/ticket/48821

#6 @sorenbronsted
7 months ago

  • Keywords has-patch has-unit-tests added; dev-feedback removed

#7 @prbot
6 months ago

sorenbronsted commented on PR #253:

We should also do research about what the most correct way to check uniqueness would be. SORT_REGULAR does an == comparison which is of particular importance for [arrays and objects](https://www.php.net/manual/en/language.operators.array.php).

For instance with the given schema and values, should those be considered unique or not?

`json
{

"type": "array",
"uniqueItems": true,
"items": {

"type": "array",
"items": {

"type": "string"

}

}

}
`

`json
[

[

"a",
"b",
"c"

],
[

"b",
"a",
"c"

]

]
`

Or what about

`json
{

"type": "array",
"uniqueItems": true,
"items": {

"type": [

"number",
"integer"

]

}

}
`

`json
[

1.0,
1

]
`

We should look at the spec, tests and other validators.

I have looked into justinrainbow approach and he uses var_exports and thereby converts everything to strings, but in my opinion he fails on numbers because [1,1.0] is unique according to justinrainbow but I think not, and the spec for uniqueItems is rather vague on what is unique means.
I will use justinrainbow approach but use the flag SORT_NUMERIC when the array type is declared numeric.
Thank you for your patience and feedback on my contributions :-)

#8 @prbot
6 months ago

TimothyBJacobs commented on PR #253:

Could you also take a look at https://github.com/ajv-validator/ajv/?

Thank you for your patience and feedback on my contributions :-)

No problem at all, thanks for taking on these tickets!

#10 @TimothyBlynJacobs
6 months ago

In 47923:

REST API: Support the (min|max)Items JSON Schema keywords.

A future commit will add support for the uniqueItems keyword.

Props sorenbronsted.
See #48821.

This ticket was mentioned in Slack in #core-restapi by timothybjacobs. View the logs.


5 months ago

#12 @TimothyBlynJacobs
5 months ago

@sorenbronsted are you still taking a look at supporting uniqueItems? I'd like to have a patch for this by next week so we can give it some soak time before 5.5 Beta 1.

#13 @sorenbronsted
5 months ago

@TimothyBlynJacobs yes and I will take a refresh on this.

#14 @prbot
5 months ago

sorenbronsted commented on PR #253:

Could you also take a look at https://github.com/ajv-validator/ajv/?

Thank you for your patience and feedback on my contributions :-)

No problem at all, thanks for taking on these tickets!


The testcases I have written are taken are from [official testsuite](https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/master/tests/draft2019-09/uniqueItems.json), and other implementations uses the same testsuite, so this implementation can the same as others including ajv.

#15 @prbot
5 months ago

TimothyBJacobs commented on PR #253:

Does the official test suite have tests covering the array/object behavior though?

If not, what I think we want is for arrays order should matter. [ 'a', 'b' ] is distinct from [ 'b', 'a' ]. That means that with an uniqueItems keyword the following would pass validation.

`json
[

[ "a", "b" ],
[ "b", "a" ]

]
`

`json
{

"type": "array",
"uniqueItems": true,
"items": {

"type": "array",
"items": {

"type": "string"

}

}

}
`

For objects, the order _should not_ matter. So the following example should fail validation.

`json
[

{

"a": "1",
"b": "2"

},
{

"b": "2",
"a": "1"

}

]
`

`json
{

"type": "array",
"uniqueItems": true,
"items": {

"type": "object",
"properties": {

"a": {

"type": "string",

},
"b": {

"type": "string"

}

}

}

}
`

This would match the idea that JSON arrays are "ordered" but JSON objects are "unordered".

#17 @prbot
5 months ago

sorenbronsted commented on PR #253:

Does the official test suite have tests covering the array/object behavior though?

Yes it does. The above link is the official test suite for unique items.

For objects, the order _should not_ matter. So the following example should fail validation.

`json
[

{

"a": "1",
"b": "2"

},
{

"b": "2",
"a": "1"

}

]
`

`json
{

"type": "array",
"uniqueItems": true,
"items": {

"type": "object",
"properties": {

"a": {

"type": "string",

},
"b": {

"type": "string"

}

}

}

}
`

This would match the idea that JSON arrays are "ordered" but JSON objects are "unordered".

My implementation fails on the object case, so I will fix that. Actually I will implement the official test suite for unique items.

This ticket was mentioned in PR #366 on WordPress/wordpress-develop by sorenbronsted.


5 months ago

@TimothyBlynJacobs I have added the official json schema test suite for uniqueitems

Trac ticket: https://core.trac.wordpress.org/ticket/48821

#19 @prbot
5 months ago

TimothyBJacobs commented on PR #366:

Thanks @sorenbronsted!

Support for min and max items was added in https://core.trac.wordpress.org/changeset/47923.

Let's also skip additionalItems and positional items support please.

#20 @prbot
5 months ago

sorenbronsted commented on PR #366:

Let's also skip additionalItems and positional items support please.

Can I ask why you want to skip theses things, when I have put the time and effort into making all the tests work?

#21 @prbot
5 months ago

TimothyBJacobs commented on PR #366:

I think its a complicated feature to implement and is fairly unintuitive for users to use. I think an object with named properties is almost always a better DUX.

In the latest spec its also been removed from items and moved to a clearer tupleItems.

#22 @prbot
5 months ago

TimothyBJacobs commented on PR #366:

Something like this:

`php
/

  • Checks if an array is made up of unique items. *
  • @since 5.5.0 *
  • @param array $array The array to check.
  • @return bool True if the array contains unique items, false otherwise. */

function rest_validate_array_contains_unique_items( $array ) {

$seen = array();

foreach ( $array as $item ) {

$stabilized = rest_stabilize_value( $item );
$key = maybe_serialize( $stabilized );

if ( ! isset( $seen[ $key ] ) ) {

$seen[ $key ] = 1;

continue;

}

return false;

}

return true;

}

/

  • Stabilizes a value following JSON Schema semantics. *
  • For lists, order is preserved. For objects, properties are reordered alphabetically. *
  • @since 5.5.0 *
  • @param mixed $value The value to stabilize. Must already be sanitized. Objects should have been converted to arrays.
  • @return mixed The stabilized value. */

function rest_stabilize_value( $value ) {

if ( is_scalar( $value )
is_null( $value ) ) {

return $value;

}

if ( is_object( $value ) ) {

_doing_it_wrong( FUNCTION, ( 'Cannot stabilize objects. Convert the object to an array first.' ), '5.5.0' );

return $value;

}

ksort( $value );

foreach ( $value as $k => $v ) {

$value[ $k ] = rest_stabilize_value( $v );

}

return $value;

}
`

#23 @prbot
5 months ago

TimothyBJacobs commented on PR #366:

A possible patch:

`patch
diff --git a/src/wp-includes/rest-api.php b/src/wp-includes/rest-api.php
index b2ea6e9654..8154d03af3 100644
--- a/src/wp-includes/rest-api.php
+++ b/src/wp-includes/rest-api.php
@@ -54,7 +54,7 @@ function register_rest_route( $namespace, $route, $args = array(), $override = f

_doing_it_wrong(

'register_rest_route',
sprintf(

  • /* translators: %s: rest_api_init */

+ /* translators: %s: rest_api_init */

( 'REST API routes must be registered on the %s action.' ),
'<code>rest_api_init</code>'

),

@@ -1438,6 +1438,63 @@ function rest_handle_multi_type_schema( $value, $args, $param = ) {

return $best_type;

}


+/
+ * Checks if an array is made up of unique items.
+ *
+ * @since 5.5.0
+ *
+ * @param array $array The array to check.
+ * @return bool True if the array contains unique items, false otherwise.
+ */
+function rest_validate_array_contains_unique_items( $array ) {
+ $seen = array();
+
+ foreach ( $array as $item ) {
+ $stabilized = rest_stabilize_value( $item );
+ $key = serialize( $stabilized );
+
+ if ( ! isset( $seen[ $key ] ) ) {
+ $seen[ $key ] = 1;
+
+ continue;
+ }
+
+ return false;
+ }
+
+ return true;
+}
+
+/

+ * Stabilizes a value following JSON Schema semantics.
+ *
+ * For lists, order is preserved. For objects, properties are reordered alphabetically.
+ *
+ * @since 5.5.0
+ *
+ * @param mixed $value The value to stabilize. Must already be sanitized. Objects should have been converted to arrays.
+ * @return mixed The stabilized value.
+ */
+function rest_stabilize_value( $value ) {

+ if ( is_scalar( $value )
is_null( $value ) ) {

+ return $value;
+ }
+
+ if ( is_object( $value ) ) {
+ _doing_it_wrong( FUNCTION, ( 'Cannot stabilize objects. Convert the object to an array first.' ), '5.5.0' );
+
+ return $value;
+ }
+
+ ksort( $value );
+
+ foreach ( $value as $k => $v ) {
+ $value[ $k ] = rest_stabilize_value( $v );
+ }
+
+ return $value;
+}
+

/

  • Validate a value based on a schema. *

@@ -1492,10 +1549,12 @@ function rest_validate_value_from_schema( $value, $args, $param = ) {

$value = rest_sanitize_array( $value );


  • foreach ( $value as $index => $v ) {
  • $is_valid = rest_validate_value_from_schema( $v, $argsitems?, $param . '. $index .?' );
  • if ( is_wp_error( $is_valid ) ) {
  • return $is_valid;

+ if ( isset( $argsitems? ) ) {
+ foreach ( $value as $index => $v ) {
+ $is_valid = rest_validate_value_from_schema( $v, $argsitems?, $param . '. $index .?' );
+ if ( is_wp_error( $is_valid ) ) {
+ return $is_valid;
+ }

}

}


@@ -1509,22 +1568,9 @@ function rest_validate_value_from_schema( $value, $args, $param = ) {

return new WP_Error( 'rest_invalid_param', sprintf( ( '%1$s must contain at most %2$s items.' ), $param, number_format_i18n( $argsmaxItems? ) ) );

}


  • if ( isset( $argsuniqueItems? ) && $argsuniqueItems? ) {
  • foreach ( $value as $index => &$object ) {
  • if ( is_array( $object ) ) {
  • ksort( $object );
  • }
  • }
  • $unique = array_map(
  • function ( $e ) {
  • return var_export( $e, true );
  • },
  • $value
  • );
  • if ( count( $value ) !== count( array_unique( $unique ) ) ) {
  • /* translators: 1: Parameter */
  • return new WP_Error( 'rest_invalid_param', sprintf( ( '%1$s has duplicate items.' ), $param ) );
  • }

+ if ( ! empty( $argsuniqueItems? ) && ! rest_validate_array_contains_unique_items( $value ) ) {
+ /* translators: 1: Parameter */
+ return new WP_Error( 'rest_invalid_param', sprintf( ( '%1$s has duplicate items.' ), $param ) );

}

}


@@ -1615,7 +1661,7 @@ function rest_validate_value_from_schema( $value, $args, $param = ) {

return new WP_Error(

'rest_invalid_param',
sprintf(

  • /* translators: 1: Parameter, 2: Number of characters. */

+ /* translators: 1: Parameter, 2: Number of characters. */

_n( '%1$s must be at least %2$s character long.', '%1$s must be at least %2$s characters long.', $argsminLength? ),
$param,
number_format_i18n( $argsminLength? )

@@ -1627,7 +1673,7 @@ function rest_validate_value_from_schema( $value, $args, $param = ) {

return new WP_Error(

'rest_invalid_param',
sprintf(

  • /* translators: 1: Parameter, 2: Number of characters. */

+ /* translators: 1: Parameter, 2: Number of characters. */

_n( '%1$s must be at most %2$s character long.', '%1$s must be at most %2$s characters long.', $argsmaxLength? ),
$param,
number_format_i18n( $argsmaxLength? )

@@ -1647,7 +1693,7 @@ function rest_validate_value_from_schema( $value, $args, $param = ) {

The "format" keyword should only be applied to strings. However, for backward compatibility,
we allow the "format" keyword if the type keyword was not specified, or was set to an invalid value.
if ( isset( $argsformat? )

&& ( ! isset( $argstype? ) + && ( ! isset( $argstype? )
'string' === $argstype? ! in_array( $argstype?, $allowed_types, true ) )
'string' === $argstype? ! in_array( $argstype?, $allowed_types, true ) )

) {

switch ( $argsformat? ) {

case 'hex-color':

@@ -1815,7 +1861,7 @@ function rest_sanitize_value_from_schema( $value, $args, $param = ) {

This behavior matches rest_validate_value_from_schema().
if ( isset( $argsformat? )

&& ( ! isset( $argstype? ) + && ( ! isset( $argstype? )
'string' === $argstype? ! in_array( $argstype?, $allowed_types, true ) )
'string' === $argstype? ! in_array( $argstype?, $allowed_types, true ) )

) {

switch ( $argsformat? ) {

case 'hex-color':

diff --git a/tests/phpunit/tests/rest-api/rest-schema-validation.php b/tests/phpunit/tests/rest-api/rest-schema-validation.php
index fae968b40e..37c03be6d1 100644
--- a/tests/phpunit/tests/rest-api/rest-schema-validation.php
+++ b/tests/phpunit/tests/rest-api/rest-schema-validation.php
@@ -906,11 +906,9 @@ class WP_Test_REST_Schema_Validation extends WP_UnitTestCase {

$this->assertWPError( rest_validate_value_from_schema( 'foobar', $schema ) );

}


  • /
  • * Data providor for uniqueitems tests
  • * @return Generator
  • */
  • public function data_uniqueitems() {

+ public function data_unique_items() {
+ $all_types = array( 'object', 'array', 'null', 'number', 'integer', 'boolean', 'string' );
+

the following test suites is not supported at the moment
$skip = array(

'uniqueItems with an array of items',

@@ -919,6 +917,9 @@ class WP_Test_REST_Schema_Validation extends WP_UnitTestCase {

'uniqueItems=false with an array of items and additionalItems=false',

);
$suites = json_decode( file_get_contents( DIR . '/json_schema_test_suite/uniqueitems.json' ), true );

+
+ $tests = array();
+

foreach ( $suites as $suite ) {

if ( in_array( $suitedescription?, $skip, true ) ) {

continue;

@@ -929,26 +930,36 @@ class WP_Test_REST_Schema_Validation extends WP_UnitTestCase {

}
items is required for our implementation
if ( ! isset( $suiteschema?items? ) ) {

+ $suiteschema?items? = array(
+ 'type' => $all_types,
+ 'items' => array(
+ 'type' => $all_types,
+ ),
+ );

}
foreach ( $suitetests? as $test ) {

  • yield array( $test, $suite );

+ $tests[] = array( $test, $suite );

}

}

+
+ return $tests;

}


/

  • @ticket 48821 *
  • * @dataProvider data_uniqueitems

+ * @dataProvider data_unique_items

*/

  • public function test_uniqueitems( $test, $suite ) {

+ public function test_unique_items( $test, $suite ) {

$test_description = $suitedescription? . ': ' . $testdescription?;
$message = $test_description . ': ' . var_export( $testdata?, true );

+
+ $valid = rest_validate_value_from_schema( $testdata?, $suiteschema? );
+

if ( $testvalid? ) {

  • $this->assertTrue( rest_validate_value_from_schema( $testdata?, $suiteschema? ), $message );

+ $this->assertTrue( $valid, $message );

} else {

  • $this->assertWPError( rest_validate_value_from_schema( $testdata?, $suiteschema? ), $message );

+ $this->assertWPError( $valid, $message );

}

}

}

`

#24 @TimothyBlynJacobs
5 months ago

  • Owner set to TimothyBlynJacobs
  • Resolution set to fixed
  • Status changed from new to closed

In 48357:

REST API: Add support for the uniqueItems keyword.

Props sorenbronsted.
Fixes #48821.

#25 @prbot
5 months ago

TimothyBJacobs commented on PR #366:

Merged.

Note: See TracTickets for help on using tickets.