#5337 closed defect (bug) (invalid)
Wrong timezone with comment_time and the_time since DST ended
| Reported by: |
|
Owned by: | |
|---|---|---|---|
| Milestone: | Priority: | normal | |
| Severity: | normal | Version: | 2.3.1 |
| Component: | General | Keywords: | time date timezone dst utc has-patch has-unit-tests |
| Focuses: | Cc: |
Description
I am designing a theme where I want the current timezone to be displayed after the time in comment and post timestamps. For example, I use <?php comment_time() ?> with my default time format set to H:i T.
I am in the UK, and until the clocks changed on 28 October everything was fine. Posts or comments written during a summer gave the time as e.g. "14:00 BST" while those written in the winter said "13:00 GMT".
However, since BST ended, I now see "UTC" for all datestamps, irrespective of whether they were posted during the summer or winter. So the two examples given above would now show "14:00 UTC" and "13:00 UTC" respectively, despite the fact that the first one is incorrect. I also don't understand why it switched from using GMT to UTC.
I do not know whether this bug is present in other timezones.
Change History (6)
#3
@
16 years ago
- Milestone 2.9 deleted
- Resolution set to invalid
- Status changed from new to closed
comment_time() reports what your clock said when you wrote the comment. Changing the timezone setting in your settings does not change that value for past comments.
Use the data from the GMT column in the comments object if you wish to ignore or update the timezone.
This ticket was mentioned in PR #6387 on WordPress/wordpress-develop by @dmsnell.
15 months ago
#4
- Keywords has-patch has-unit-tests added
Trac ticket: Core-61072
Token Map Trac ticket: Core-60698
From #5337 takes the HTML text decoder.
Replaces WordPress/gutenberg#47040
## Status
The code should be working now with this, and fully spec-compliant.
Tests are covered generally by the html5lib test suite.
### Performance
After some initial testing this appears to be around 20% slower in its current state at decoding text values compared to using html_entity_decode(). I tested against a set of 296,046 web pages at the root domain for a list of the top-ranked domains that I found online.
The decoder itself, when run against a worst-case scenario document with one million randomly-generated hexadecimal and decimal numeric character references is much slower than
html_entity_decode()(which itself is moot given the reliability issues with that function).html_entity_decode()processes that test file at 195 MB/s while the HTML decoder processes it at 6.40 MB/s. With the pre-decoder patch this rises to 9.78 MB/s.
Understanding context is important here because most documents aren't worse-case documents. Typical web performance is reported in this PR based on real web pages on high-ranked domains. The fact that this step is slow does not make page rendering slow because it's such a small part of a real web path. In WordPress this would be even less impactful because we're not parsing all text content of all HTML on the server; only a fraction of it.
The impact is quite marginal, adding around 60 µs per page. For the set of close to 300k pages that took the total runtime from 87s to 105s. I tested with the following main loop, using microtime( true ) before and after the loop to add to the total time in an attempt to eliminate the I/O wait time from the results. This is a worst-case scenario where decode every attribute and every text node. Again, in practice, WordPress would only likely experience a fraction of that 60 µs because it's not decodingevery text node and every attribute of the HTML it ships to a browser.
I attempted to avoid string allocations and this raised a challenge: strpos() doesn't provide a way to stop at a given index. This led me to try replacing it with a simple look to advance character by character until finding a &. This slowed it down to about 25% slower than html_entity_decode() so I removed that and instead relied on using strpos() with the possibility that it scans much further past the end of the value. On the test set of data it was still faster.
while ( $p->next_token() ) {
$token_name = $p->get_token_name();
$token_type = $p->get_token_type();
if ( '#tag' === $token_type && ! $p->is_tag_closer() ) {
foreach ( $p->get_attribute_names_with_prefix( '' ) ?? array() as $name ) {
$chunk = $p->get_attribute( $name );
if ( is_string( $chunk ) ) {
$total_code_points += mb_strlen( $chunk );
}
}
}
$text = $p->get_modifiable_text();
if ( '' !== $text ) {
$total_code_points += mb_strlen( $text );
}
}
For comparison, I built a version that skips the WP_Token_Map and instead relies on a basic associative array whose keys are the character reference names and whose values are the replacements. This was 840% slower than html_decode_entities() and increased the average page processing time by 2.175 ms. The token map is thus approximately 36x faster than the naive implementation.
##### Pre-decoding
In an attempt to rely more on html_entity_decode() I added a pre-decoding step that would handle all well-formed numeric character encodings. The logic here is that if we can use a quick preg_replace_callback() pass to get as much into C-code as we can, by means of html_entity_decode(), then maybe it would be worth it even with the additional pass.
Unfortunately the results were instantly slower, adding another 20% slowdown in my first 100k domains under test. That is, it's over 40% slower than a pure html_entity_decode() whereas the code without the pre-encoding step is only 20% slower.
<details><summary>The Pre-Decoder</summary>
// pre-decode certain known numeric character references.
$text = preg_replace_callback(
'~&#((?P<is_hex>[Xx])0*(?P<hex_digits>[1-9A-Fa-f][0-9A-Fa-f]{0,5})|0*(?P<dec_digits>[1-9][0-9]{0,6}));~',
static function ( $matches ) {
$is_hex = strlen( $matches['is_hex'] ) > 0;
$digits = $matches[ $is_hex ? 'hex_digits' : 'dec_digits' ];
if ( ( $is_hex ? 2 : 3 ) === strlen( $digits ) ) {
$code_point = intval( $digits, $is_hex ? 16 : 10 );
/*
* Noncharacters, 0x0D, and non-ASCII-whitespace control characters.
*
* > A noncharacter is a code point that is in the range U+FDD0 to U+FDEF,
* > inclusive, or U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, U+2FFFE, U+2FFFF,
* > U+3FFFE, U+3FFFF, U+4FFFE, U+4FFFF, U+5FFFE, U+5FFFF, U+6FFFE,
* > U+6FFFF, U+7FFFE, U+7FFFF, U+8FFFE, U+8FFFF, U+9FFFE, U+9FFFF,
* > U+AFFFE, U+AFFFF, U+BFFFE, U+BFFFF, U+CFFFE, U+CFFFF, U+DFFFE,
* > U+DFFFF, U+EFFFE, U+EFFFF, U+FFFFE, U+FFFFF, U+10FFFE, or U+10FFFF.
*
* A C0 control is a code point that is in the range of U+00 to U+1F,
* but ASCII whitespace includes U+09, U+0A, U+0C, and U+0D.
*
* These characters are invalid but still decode as any valid character.
* This comment is here to note and explain why there's no check to
* remove these characters or replace them.
*
* @see https://infra.spec.whatwg.org/#noncharacter
*/
/*
* Code points in the C1 controls area need to be remapped as if they
* were stored in Windows-1252. Note! This transformation only happens
* for numeric character references. The raw code points in the byte
* stream are not translated.
*
* > If the number is one of the numbers in the first column of
* > the following table, then find the row with that number in
* > the first column, and set the character reference code to
* > the number in the second column of that row.
*/
if ( $code_point >= 0x80 && $code_point <= 0x9F ) {
$windows_1252_mapping = array(
'€', // 0x80 -> EURO SIGN (€).
"\xC2\x81", // 0x81 -> (no change).
'‚', // 0x82 -> SINGLE LOW-9 QUOTATION MARK (‚).
'ƒ', // 0x83 -> LATIN SMALL LETTER F WITH HOOK (ƒ).
'„', // 0x84 -> DOUBLE LOW-9 QUOTATION MARK („).
'…', // 0x85 -> HORIZONTAL ELLIPSIS (…).
'†', // 0x86 -> DAGGER (†).
'‡', // 0x87 -> DOUBLE DAGGER (‡).
'ˆ', // 0x88 -> MODIFIER LETTER CIRCUMFLEX ACCENT (ˆ).
'‰', // 0x89 -> PER MILLE SIGN (‰).
'Š', // 0x8A -> LATIN CAPITAL LETTER S WITH CARON (Š).
'‹', // 0x8B -> SINGLE LEFT-POINTING ANGLE QUOTATION MARK (‹).
'Œ', // 0x8C -> LATIN CAPITAL LIGATURE OE (Œ).
"\xC2\x8D", // 0x8D -> (no change).
'Ž', // 0x8E -> LATIN CAPITAL LETTER Z WITH CARON (Ž).
"\xC2\x8F", // 0x8F -> (no change).
"\xC2\x90", // 0x90 -> (no change).
'‘', // 0x91 -> LEFT SINGLE QUOTATION MARK (‘).
'’', // 0x92 -> RIGHT SINGLE QUOTATION MARK (’).
'“', // 0x93 -> LEFT DOUBLE QUOTATION MARK (“).
'”', // 0x94 -> RIGHT DOUBLE QUOTATION MARK (”).
'•', // 0x95 -> BULLET (•).
'–', // 0x96 -> EN DASH (–).
'—', // 0x97 -> EM DASH (—).
'˜', // 0x98 -> SMALL TILDE (˜).
'™', // 0x99 -> TRADE MARK SIGN (™).
'š', // 0x9A -> LATIN SMALL LETTER S WITH CARON (š).
'›', // 0x9B -> SINGLE RIGHT-POINTING ANGLE QUOTATION MARK (›).
'œ', // 0x9C -> LATIN SMALL LIGATURE OE (œ).
"\xC2\x9D", // 0x9D -> (no change).
'ž', // 0x9E -> LATIN SMALL LETTER Z WITH CARON (ž).
'Ÿ', // 0x9F -> LATIN CAPITAL LETTER Y WITH DIAERESIS (Ÿ).
);
return $windows_1252_mapping[ $code_point - 0x80 ];
}
}
return html_entity_decode( $matches[0], ENT_SUBSTITUTE, 'UTF-8' );
},
$text
);
</details>
##### Faster integer decoding.
I attempted to parse the code point inline while scanning the digits in hopes to save some time computing, but this dramatically slowed down the interpret. I think that the per-character parsing is much slower than intval().
##### Faster digit detection.
I attempted to replace strspn( $text, $numeric_digits ) with a custom look examining each character for whether it was in the digit ranges, but this was just as slow as the custom integer decoder.
##### Quick table lookup of group/small token indexing.
On the idea that looking up the group or small word in the lookup strings might be slow, given that it's required to iterate every time, I tried adding a patch to introduce an index table for direct lookup into where words of the given starting letter start, and if they even exist in the table at all.
<details><summary>Table-lookup patch</summary>
-
src/wp-includes/class-wp-token-map.php
diff --git a/src/wp-includes/class-wp-token-map.php b/src/wp-includes/class-wp-token-map.php index c7bb9316ed..60a189b37d 100644
a b class WP_Token_Map { 182 182 */ 183 183 private $groups = ''; 184 184 185 /** 186 * Indicates where the first group key starting with a given 187 * letter is to be found in the groups string. 188 * 189 * - Each position in the string corresponds to the first byte 190 * of a lookup key. E.g. keys starting with `A` will look 191 * at the 65th index (A is U+0041) to see if it's in the set 192 * and where. 193 * 194 * - Each index is stored as a two-byte unsigned integer 195 * indicating where to start looking. This limits the 196 * key size to 256 bytes. 197 * 198 * - A null value indicates that there are no words present 199 * starting with the given byte. 200 * 201 * Example: 202 * 203 * Suppose that there exists a key starting with `B` at 204 * offset 481 (0x01 0xE1) but none starting with `A`. 205 * ┌────────┬───────┬───────┬───────┐ 206 * │ ... │ A │ B │ ... │ 207 * ├────────┼───────┼───────┼───────┤ 208 * │ │ 00 00 │ 01 E1 │ │ 209 * └────────┴───────┴───────┴───────┘ 210 * 211 * @since 6.6.0 212 * 213 * @var string 214 */ 215 private $group_index; 216 185 217 /** 186 218 * Stores an optimized row of small words, where every entry is 187 219 * `$this->key_size + 1` bytes long and zero-extended. … … class WP_Token_Map { 200 232 */ 201 233 private $small_words = ''; 202 234 235 /** 236 * Indicates where the first word starting with a given letter 237 * is to be found in the small words string. 238 * 239 * - Each position in the string corresponds to the first byte 240 * of a lookup word. E.g. words starting with `A` will look 241 * at the 65th index (A is U+0041) to see if it's in the set 242 * and where. 243 * 244 * - Each index is stored as a two-byte unsigned integer 245 * indicating where to start looking. This limits the 246 * key size to 256 bytes. 247 * 248 * - A null value indicates that there are no words present 249 * starting with the given byte. 250 * 251 * Example: 252 * 253 * Suppose that there exists a word starting with `B` at 254 * offset 481 (0x01 0xE1) but none starting with `A`. 255 * ┌────────┬───────┬───────┬───────┐ 256 * │ ... │ A │ B │ ... │ 257 * ├────────┼───────┼───────┼───────┤ 258 * │ │ 00 00 │ 01 E1 │ │ 259 * └────────┴───────┴───────┴───────┘ 260 * 261 * @since 6.6.0 262 * 263 * @var string 264 */ 265 private $small_index; 266 203 267 /** 204 268 * Replacements for the small words, in the same order they appear. 205 269 * … … class WP_Token_Map { 287 351 ); 288 352 } 289 353 354 // Prime the search indices. 355 $map->small_index = str_repeat( "\xFF\xFF", 256 ); 356 $map->group_index = str_repeat( "\xFF\xFF", 256 ); 357 290 358 // Finally construct the optimized lookups. 291 359 360 $last_byte = "\x00"; 292 361 foreach ( $shorts as $word ) { 362 if ( $last_byte !== $word[0] ) { 363 $last_byte = $word[0]; 364 $index_at = 2 * ord( $last_byte ); 365 $offset = pack( 'n', strlen( $map->small_words ) ); 366 $map->small_index[ $index_at ] = $offset[0]; 367 $map->small_index[ $index_at + 1 ] = $offset[1]; 368 } 369 293 370 $map->small_words .= str_pad( $word, $key_length + 1, "\x00", STR_PAD_RIGHT ); 294 371 $map->small_mappings[] = $mappings[ $word ]; 295 372 } … … class WP_Token_Map { 297 374 $group_keys = array_keys( $groups ); 298 375 sort( $group_keys ); 299 376 377 $last_byte = "\x00"; 300 378 foreach ( $group_keys as $group ) { 379 if ( $last_byte !== $group[0] ) { 380 $last_byte = $group[0]; 381 $index_at = 2 * ord( $last_byte ); 382 $offset = pack( 'n', strlen( $map->groups ) ); 383 $map->group_index[ $index_at ] = $offset[0]; 384 $map->group_index[ $index_at + 1 ] = $offset[1]; 385 } 386 301 387 $map->groups .= "{$group}\x00"; 302 388 303 389 $group_string = ''; … … class WP_Token_Map { 327 413 * 328 414 * @param int $key_length Group key length. 329 415 * @param string $groups Group lookup index. 416 * @param string $group_index Locations in the group lookup where each character starts. 330 417 * @param array $large_words Large word groups and packed strings. 331 418 * @param string $small_words Small words packed string. 419 * @param string $small_index Locations in the small word lookup where each character starts. 332 420 * @param array $small_mappings Small word mappings. 333 421 * 334 422 * @return WP_Token_Map Map with precomputed data loaded. 335 423 */ 336 public static function from_precomputed_table( $key_length, $groups, $ large_words, $small_words, $small_mappings ) {424 public static function from_precomputed_table( $key_length, $groups, $group_index, $large_words, $small_words, $small_index, $small_mappings ) { 337 425 $map = new WP_Token_Map(); 338 426 339 427 $map->key_length = $key_length; 340 428 $map->groups = $groups; 429 $map->group_index = $group_index; 341 430 $map->large_words = $large_words; 342 431 $map->small_words = $small_words; 432 $map->small_index = $small_index; 343 433 $map->small_mappings = $small_mappings; 344 434 345 435 return $map; … … class WP_Token_Map { 454 544 if ( $text_length > $this->key_length ) { 455 545 $group_key = substr( $text, $offset, $this->key_length ); 456 546 457 $group_at = $ignore_case ? stripos( $this->groups, $group_key ) : strpos( $this->groups, $group_key ); 547 $group_index = unpack( 'n', $this->group_index, 2 * ord( $text[ $offset ] ) )[1]; 548 if ( 0xFFFF === $group_index && ! $ignore_case ) { 549 // Perhaps a short word then. 550 return strlen( $this->small_words ) > 0 551 ? $this->read_small_token( $text, $offset, $skip_bytes, $case_sensitivity ) 552 : false; 553 } 554 555 $group_at = $ignore_case 556 ? stripos( $this->groups, $group_key ) 557 : strpos( $this->groups, $group_key, $group_index ); 558 458 559 if ( false === $group_at ) { 459 560 // Perhaps a short word then. 460 561 return strlen( $this->small_words ) > 0 … … class WP_Token_Map { 499 600 * @return string|false Mapped value of lookup key if found, otherwise `false`. 500 601 */ 501 602 private function read_small_token( $text, $offset, &$skip_bytes, $case_sensitivity = 'case-sensitive' ) { 502 $ignore_case = 'case-insensitive' === $case_sensitivity; 603 $ignore_case = 'case-insensitive' === $case_sensitivity; 604 605 // Quickly eliminate impossible matches. 606 $small_index = unpack( 'n', $this->small_index, 2 * ord( $text[ $offset ] ) )[1]; 607 if ( 0xFFFF === $small_index && ! $ignore_case ) { 608 return false; 609 } 610 503 611 $small_length = strlen( $this->small_words ); 504 612 $search_text = substr( $text, $offset, $this->key_length ); 505 613 if ( $ignore_case ) { … … class WP_Token_Map { 507 615 } 508 616 $starting_char = $search_text[0]; 509 617 510 $at = 0;618 $at = $ignore_case ? 0 : $small_index; 511 619 while ( $at < $small_length ) { 512 620 if ( 513 621 $starting_char !== $this->small_words[ $at ] && … … class WP_Token_Map { 621 729 $group_line = str_replace( "\x00", "\\x00", $this->groups ); 622 730 $output .= "{$i1}\"{$group_line}\",\n"; 623 731 732 $group_index = ''; 733 $group_index_length = strlen( $this->group_index ); 734 for ( $i = 0; $i < $group_index_length; $i++ ) { 735 $group_index .= '\\x' . str_pad( dechex( ord( $this->group_index[ $i ] ) ), 2, '0', STR_PAD_LEFT ); 736 } 737 $output .= "{$i1}\"{$group_index}\",\n"; 738 624 739 $output .= "{$i1}array(\n"; 625 740 626 741 $prefixes = explode( "\x00", $this->groups ); … … class WP_Token_Map { 685 800 $small_text = str_replace( "\x00", '\x00', implode( '', $small_words ) ); 686 801 $output .= "{$i1}\"{$small_text}\",\n"; 687 802 803 $small_index = ''; 804 $small_index_length = strlen( $this->small_index ); 805 for ( $i = 0; $i < $small_index_length; $i++ ) { 806 $small_index .= '\\x' . str_pad( dechex( ord( $this->small_index[ $i ] ) ), 2, '0', STR_PAD_LEFT ); 807 } 808 $output .= "{$i1}\"{$small_index}\",\n"; 809 688 810 $output .= "{$i1}array(\n"; 689 811 foreach ( $this->small_mappings as $mapping ) { 690 812 $output .= "{$i2}\"{$mapping}\",\n"; -
src/wp-includes/html-api/html5-named-character-references.php
diff --git a/src/wp-includes/html-api/html5-named-character-references.php b/src/wp-includes/html-api/html5-named-character-references.php index 41c467e268..b7827f9b8e 100644
a b global $html5_named_character_references; 31 31 $html5_named_character_references = WP_Token_Map::from_precomputed_table( 32 32 2, 33 33 "AE\x00AM\x00Aa\x00Ab\x00Ac\x00Af\x00Ag\x00Al\x00Am\x00An\x00Ao\x00Ap\x00Ar\x00As\x00At\x00Au\x00Ba\x00Bc\x00Be\x00Bf\x00Bo\x00Br\x00Bs\x00Bu\x00CH\x00CO\x00Ca\x00Cc\x00Cd\x00Ce\x00Cf\x00Ch\x00Ci\x00Cl\x00Co\x00Cr\x00Cs\x00Cu\x00DD\x00DJ\x00DS\x00DZ\x00Da\x00Dc\x00De\x00Df\x00Di\x00Do\x00Ds\x00EN\x00ET\x00Ea\x00Ec\x00Ed\x00Ef\x00Eg\x00El\x00Em\x00Eo\x00Ep\x00Eq\x00Es\x00Et\x00Eu\x00Ex\x00Fc\x00Ff\x00Fi\x00Fo\x00Fs\x00GJ\x00GT\x00Ga\x00Gb\x00Gc\x00Gd\x00Gf\x00Gg\x00Go\x00Gr\x00Gs\x00Gt\x00HA\x00Ha\x00Hc\x00Hf\x00Hi\x00Ho\x00Hs\x00Hu\x00IE\x00IJ\x00IO\x00Ia\x00Ic\x00Id\x00If\x00Ig\x00Im\x00In\x00Io\x00Is\x00It\x00Iu\x00Jc\x00Jf\x00Jo\x00Js\x00Ju\x00KH\x00KJ\x00Ka\x00Kc\x00Kf\x00Ko\x00Ks\x00LJ\x00LT\x00La\x00Lc\x00Le\x00Lf\x00Ll\x00Lm\x00Lo\x00Ls\x00Lt\x00Ma\x00Mc\x00Me\x00Mf\x00Mi\x00Mo\x00Ms\x00Mu\x00NJ\x00Na\x00Nc\x00Ne\x00Nf\x00No\x00Ns\x00Nt\x00Nu\x00OE\x00Oa\x00Oc\x00Od\x00Of\x00Og\x00Om\x00Oo\x00Op\x00Or\x00Os\x00Ot\x00Ou\x00Ov\x00Pa\x00Pc\x00Pf\x00Ph\x00Pi\x00Pl\x00Po\x00Pr\x00Ps\x00QU\x00Qf\x00Qo\x00Qs\x00RB\x00RE\x00Ra\x00Rc\x00Re\x00Rf\x00Rh\x00Ri\x00Ro\x00Rr\x00Rs\x00Ru\x00SH\x00SO\x00Sa\x00Sc\x00Sf\x00Sh\x00Si\x00Sm\x00So\x00Sq\x00Ss\x00St\x00Su\x00TH\x00TR\x00TS\x00Ta\x00Tc\x00Tf\x00Th\x00Ti\x00To\x00Tr\x00Ts\x00Ua\x00Ub\x00Uc\x00Ud\x00Uf\x00Ug\x00Um\x00Un\x00Uo\x00Up\x00Ur\x00Us\x00Ut\x00Uu\x00VD\x00Vb\x00Vc\x00Vd\x00Ve\x00Vf\x00Vo\x00Vs\x00Vv\x00Wc\x00We\x00Wf\x00Wo\x00Ws\x00Xf\x00Xi\x00Xo\x00Xs\x00YA\x00YI\x00YU\x00Ya\x00Yc\x00Yf\x00Yo\x00Ys\x00Yu\x00ZH\x00Za\x00Zc\x00Zd\x00Ze\x00Zf\x00Zo\x00Zs\x00aa\x00ab\x00ac\x00ae\x00af\x00ag\x00al\x00am\x00an\x00ao\x00ap\x00ar\x00as\x00at\x00au\x00aw\x00bN\x00ba\x00bb\x00bc\x00bd\x00be\x00bf\x00bi\x00bk\x00bl\x00bn\x00bo\x00bp\x00br\x00bs\x00bu\x00ca\x00cc\x00cd\x00ce\x00cf\x00ch\x00ci\x00cl\x00co\x00cr\x00cs\x00ct\x00cu\x00cw\x00cy\x00dA\x00dH\x00da\x00db\x00dc\x00dd\x00de\x00df\x00dh\x00di\x00dj\x00dl\x00do\x00dr\x00ds\x00dt\x00du\x00dw\x00dz\x00eD\x00ea\x00ec\x00ed\x00ee\x00ef\x00eg\x00el\x00em\x00en\x00eo\x00ep\x00eq\x00er\x00es\x00et\x00eu\x00ex\x00fa\x00fc\x00fe\x00ff\x00fi\x00fj\x00fl\x00fn\x00fo\x00fp\x00fr\x00fs\x00gE\x00ga\x00gb\x00gc\x00gd\x00ge\x00gf\x00gg\x00gi\x00gj\x00gl\x00gn\x00go\x00gr\x00gs\x00gt\x00gv\x00hA\x00ha\x00hb\x00hc\x00he\x00hf\x00hk\x00ho\x00hs\x00hy\x00ia\x00ic\x00ie\x00if\x00ig\x00ii\x00ij\x00im\x00in\x00io\x00ip\x00iq\x00is\x00it\x00iu\x00jc\x00jf\x00jm\x00jo\x00js\x00ju\x00ka\x00kc\x00kf\x00kg\x00kh\x00kj\x00ko\x00ks\x00lA\x00lB\x00lE\x00lH\x00la\x00lb\x00lc\x00ld\x00le\x00lf\x00lg\x00lh\x00lj\x00ll\x00lm\x00ln\x00lo\x00lp\x00lr\x00ls\x00lt\x00lu\x00lv\x00mD\x00ma\x00mc\x00md\x00me\x00mf\x00mh\x00mi\x00ml\x00mn\x00mo\x00mp\x00ms\x00mu\x00nG\x00nL\x00nR\x00nV\x00na\x00nb\x00nc\x00nd\x00ne\x00nf\x00ng\x00nh\x00ni\x00nj\x00nl\x00nm\x00no\x00np\x00nr\x00ns\x00nt\x00nu\x00nv\x00nw\x00oS\x00oa\x00oc\x00od\x00oe\x00of\x00og\x00oh\x00oi\x00ol\x00om\x00oo\x00op\x00or\x00os\x00ot\x00ou\x00ov\x00pa\x00pc\x00pe\x00pf\x00ph\x00pi\x00pl\x00pm\x00po\x00pr\x00ps\x00pu\x00qf\x00qi\x00qo\x00qp\x00qs\x00qu\x00rA\x00rB\x00rH\x00ra\x00rb\x00rc\x00rd\x00re\x00rf\x00rh\x00ri\x00rl\x00rm\x00rn\x00ro\x00rp\x00rr\x00rs\x00rt\x00ru\x00rx\x00sa\x00sb\x00sc\x00sd\x00se\x00sf\x00sh\x00si\x00sl\x00sm\x00so\x00sp\x00sq\x00sr\x00ss\x00st\x00su\x00sw\x00sz\x00ta\x00tb\x00tc\x00td\x00te\x00tf\x00th\x00ti\x00to\x00tp\x00tr\x00ts\x00tw\x00uA\x00uH\x00ua\x00ub\x00uc\x00ud\x00uf\x00ug\x00uh\x00ul\x00um\x00uo\x00up\x00ur\x00us\x00ut\x00uu\x00uw\x00vA\x00vB\x00vD\x00va\x00vc\x00vd\x00ve\x00vf\x00vl\x00vn\x00vo\x00vp\x00vr\x00vs\x00vz\x00wc\x00we\x00wf\x00wo\x00wp\x00wr\x00ws\x00xc\x00xd\x00xf\x00xh\x00xi\x00xl\x00xm\x00xn\x00xo\x00xr\x00xs\x00xu\x00xv\x00xw\x00ya\x00yc\x00ye\x00yf\x00yi\x00yo\x00ys\x00yu\x00za\x00zc\x00zd\x00ze\x00zf\x00zh\x00zi\x00zo\x00zs\x00zw\x00", 34 "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x30\x00\x48\x00\x72\x00\x93\x00\xc3\x00\xd2\x00\xf6\x01\x0e\x01\x38\x01\x47\x01\x5c\x01\x7d\x01\x95\x01\xb0\x01\xda\x01\xf5\x02\x01\x02\x25\x02\x4c\x02\x6d\x02\x97\x02\xb2\x02\xc1\x02\xcd\x02\xe8\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x03\x00\x03\x30\x03\x60\x03\x8d\x03\xc6\x03\xfc\x04\x20\x04\x53\x04\x71\x04\x9e\x04\xb0\x04\xc8\x05\x0d\x05\x37\x05\x7f\x05\xb5\x05\xd9\x05\xeb\x06\x2a\x06\x63\x06\x8a\x06\xc0\x06\xed\x07\x02\x07\x2c\x07\x44\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", 34 35 array( 35 36 // AElig;[Æ] AElig[Æ]. 36 37 "\x04lig;\x02Æ\x03lig\x02Æ", … … $html5_named_character_references = WP_Token_Map::from_precomputed_table( 1294 1295 "\x03nj;\x03\x02j;\x03", 1295 1296 ), 1296 1297 "GT\x00LT\x00gt\x00lt\x00", 1298 "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\x00\x03\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x06\xff\xff\xff\xff\xff\xff\xff\xff\x00\x09\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff", 1297 1299 array( 1298 1300 ">", 1299 1301 "<",
</details>
This did not introduce a measurable speedup or slowdown on the dataset of 300k HTML pages. While I believe that the table lookup could speed up certain workloads that are heavy with named character references, it does not justify itself on realistic data and so I'm leaving the patch out.
#### Metrics on character references.
From the same set of 296k webpages I counted the frequency of each character reference. This includes the full syntax, so if we were to have come across 9 it would appear in the list. The linked file contains ANSII terminal codes, so view it through cat or less -R.
Based on this data I added a special-case for ", , and & before calling into the WP_Token_Map but it didn't have a measurable impact on performance. I'm led to conclude from this that it's not those common character references slowing things down. Possibly it's the numeric character references.
In another experiment I replaced my custom code_point_to_utf8_bytes() function with a call to mb_chr(), and again the impact wasn't significant. That method performs the same computation within PHP that this application-level does, so this is not surprising.
For clearer performance direction it's probably most helpful to profile a run of decoding and see where the CPU is spending its time. It appears to be fairly quick as it is in this patch.
### Attempted alternatives
- Tried to use an associative array whose keys are the character reference names and whose values are the translated UTF-8 strings. This operated more than 8x slower than the
WP_Token_Mapimplementation. - Based on the frequency of named character references I tried short-circuiting
",&, and , as they might account for up to 70-80% of all named character references in practice. This didn't impact the runtime. Runtime is likely dominated by numeric character reference decoding. - In 0351b78bb9 tried to micro-optimize by eliminating
ifchecks, rearranging code for frequency-analysis of code points, and replaced the Windows-1252 remapping with direct replacement. In a test 10 million randomly generated numeric character references, this performed around 3-5% faster than in the branch, but in real tests I could not measure any impact. The micro-optimizations are likely inert in a real context.
In my benchmark of decoding 10 million randomly-generated numeric character references about half the time is spent exclusively inside read_character_reference() and the other half is spent in code_point_to_utf8_bytes().
- I tried replacing
substr() + intval()with an unrolled table-lookup custom string-to-integer decoder. While that decoder performed significantly better than a native pure-PHP decoder, it was still noticeably slower thanintval().
I'm led to believe that his is nearly optimal for a pure PHP solution.
### Character-set detections.
The following CSV file is the result of surveying the / path of popular domains. It includes detections of whether the given HTML found at that path is valid UTF8, valid Windows-1252, valid ASCII, and whether it's valid in its self-reported character sets.
A 1 indicates that the HTML passes mb_check_encoding() for the encoding of the given column. A 0 indicates that it doesn't. A missing value indicates that the site did not self-report to contain that encoding.
Note that a site might self-report being encoded in multiple simultaneous and mutually-exclusive encodings.
### html5lib tests
| Results | Output |
|---|---|
| Before | Tests: 609, Assertions: 172, Failures: 63, Skipped: 435.
|
| After | Tests: 607, Assertions: 172, Skipped: 435.
|
<details><summary>Tests that are now possible to run that previously weren't.</summary>
</details>
## Differences from html_entity_decode()
<details><summary>PHP misses 720 character references</summary>
&AElig &AMP &AMP; &Aacute &Acirc &Agrave &ApplyFunction; &Aring &Assign; &Atilde &Auml &Backslash; &Barwed; &Bernoullis; &Bumpeq; &COPY &COPY; &Cayleys; &Ccedil &CircleMinus; &ClockwiseContourIntegral; &CloseCurlyDoubleQuote; &CloseCurlyQuote; &Conint; &Copf; &CounterClockwiseContourIntegral; &DD; &Del; &DiacriticalDot; &DiacriticalGrave; &Diamond; &Dot; &DotEqual; &DoubleDownArrow; &DoubleLeftRightArrow; &DoubleLeftTee; &DoubleLongLeftArrow; &DoubleLongLeftRightArrow; &DoubleRightArrow; &DoubleUpDownArrow; &DoubleVerticalBar; &DownArrow; &DownLeftVector; &DownRightVector; &ETH &Eacute &Ecirc &Egrave &Element; &EqualTilde; &Equilibrium; &Escr; &Euml &ExponentialE; &FilledVerySmallSquare; &ForAll; &Fscr; &GT &GT; &GreaterEqual; &GreaterEqualLess; &GreaterFullEqual; &GreaterLess; &GreaterSlantEqual; &Gt; &Hscr; &HumpDownHump; &Iacute &Icirc &Igrave &Im; &Intersection; &InvisibleComma; &Iscr; &Iuml &LT &LT; &Laplacetrf; &LeftAngleBracket; &LeftArrow; &LeftArrowRightArrow; &LeftCeiling; &LeftDownVector; &LeftRightArrow; &LeftTee; &LeftTriangle; &LeftUpVector; &LeftVector; &Leftarrow; &Leftrightarrow; &LessEqualGreater; &LessFullEqual; &LessGreater; &LessSlantEqual; &Lleftarrow; &LongLeftArrow; &Longleftarrow; &Longleftrightarrow; &Longrightarrow; &LowerLeftArrow; &Lscr; &Lt; &Mscr; &NegativeMediumSpace; &NegativeThickSpace; &NegativeThinSpace; &NegativeVeryThinSpace; &NestedGreaterGreater; &NestedLessLess; &NonBreakingSpace; &Nopf; &NotDoubleVerticalBar; &NotElement; &NotEqualTilde; &NotExists; &NotGreater; &NotGreaterEqual; &NotGreaterSlantEqual; &NotGreaterTilde; &NotHumpDownHump; &NotHumpEqual; &NotLeftTriangle; &NotLeftTriangleEqual; &NotLessGreater; &NotLessLess; &NotLessSlantEqual; &NotLessTilde; &NotPrecedes; &NotReverseElement; &NotRightTriangle; &NotSubset; &NotSuperset; &NotTildeEqual; &NotTildeFullEqual; &NotTildeTilde; &NotVerticalBar; &Ntilde &Oacute &Ocirc &Ograve &Oslash &Otilde &Ouml &OverBar; &PartialD; &PlusMinus; &Poincareplane; &Popf; &Precedes; &PrecedesEqual; &PrecedesTilde; &Product; &Proportion; &Proportional; &QUOT &QUOT; &Qopf; &RBarr; &REG &REG; &Rarr; &Re; &ReverseEquilibrium; &RightArrow; &RightArrowLeftArrow; &RightTee; &RightTeeArrow; &RightTriangle; &RightVector; &Rightarrow; &Rrightarrow; &Rscr; &Rsh; &ShortDownArrow; &ShortLeftArrow; &ShortRightArrow; &ShortUpArrow; &SmallCircle; &SquareIntersection; &SquareSubset; &SquareSuperset; &SquareUnion; &Subset; &Succeeds; &SucceedsSlantEqual; &SuchThat; &Sum; &Sup; &Superset; &SupersetEqual; &THORN &TRADE; &Therefore; &Tilde; &TildeEqual; &TildeTilde; &Uacute &Ucirc &Ugrave &UnderBar; &UnderBracket; &Union; &UpArrow; &UpArrowDownArrow; &UpEquilibrium; &UpTee; &Uparrow; &UpperLeftArrow; &Upsi; &Uuml &Vee; &Vert; &VerticalBar; &VerticalLine; &VerticalTilde; &VeryThinSpace; &Wedge; &Yacute &aacute &acirc &acute &acute; &aelig &agrave &alefsym; &amp &ang; &angst; &ap; &approxeq; &aring &asymp; &asympeq; &atilde &auml &backcong; &backsim; &barwedge; &becaus; &because; &bepsi; &bernou; &bigodot; &bigstar; &bigvee; &bigwedge; &blacklozenge; &blacksquare; &bot; &bottom; &boxh; &boxtimes; &bprime; &breve; &brvbar &bsime; &bullet; &bumpe; &bumpeq; &caron; &ccedil &cedil &cedil; &cent &centerdot; &checkmark; &circlearrowleft; &circlearrowright; &circledR; &circledS; &circledast; &circledcirc; &circleddash; &cire; &clubsuit; &colone; &complement; &cong; &conint; &coprod; &copy &cuepr; &cularr; &curlyeqsucc; &curren &curvearrowright; &cuvee; &cuwed; &dArr; &dash; &dblac; &dd; &ddagger; &ddarr; &deg &dharr; &diam; &diams; &die; &digamma; &div; &divide &divideontimes; &dlcorn; &doteq; &dotminus; &dotplus; &dotsquare; &downarrow; &downharpoonleft; &downharpoonright; &dtri; &dtrif; &duarr; &duhar; &eDDot; &eDot; &eacute &ecirc &ecolon; &ee; &efDot; &egrave &emptyset; &emptyv; &epsilon; &epsiv; &eqcirc; &eqsim; &eqslantgtr; &eqslantless; &equiv; &erDot; &eth &euml &exist; &fork; &frac12 &frac12; &frac14 &frac34 &gE; &gel; &geq; &geqslant; &ggg; &gnE; &gnapprox; &gneq; &gsim; &gt &gtdot; &gtrapprox; &gtreqqless; &gtrless; &gtrsim; &gvnE; &hamilt; &hbar; &heartsuit; &hksearow; &hkswarow; &hookleftarrow; &hookrightarrow; &hslash; &iacute &icirc &iexcl &iff; &igrave &ii; &iiint; &image; &imagpart; &imath; &in; &int; &integers; &intercal; &intprod; &iquest &isin; &it; &iuml &kappav; &lArr; &lEg; &lang; &laquo &larrb; &lcub; &ldquo; &ldquor; &le; &leftarrow; &leftarrowtail; &leftleftarrows; &leftrightarrow; &leftrightarrows; &leftrightsquigarrow; &leftthreetimes; &leg; &leqq; &leqslant; &lessapprox; &lesssim; &lfloor; &lg; &lhard; &lharu; &lmoustache; &lnE; &lnapprox; &lneq; &lobrk; &longleftrightarrow; &longmapsto; &longrightarrow; &looparrowleft; &loz; &lrcorner; &lrhar; &lsh; &lsim; &lsqb; &lsquo; &lsquor; &lt &ltdot; &ltrie; &ltrif; &lvnE; &macr &malt; &mapsto; &mapstodown; &mapstoleft; &mapstoup; &measuredangle; &micro &midast; &middot &middot; &minusb; &mldr; &mnplus; &mp; &mstpos; &multimap; &nGtv; &nLeftrightarrow; &nRightarrow; &nap; &natural; &nbsp &ne; &nearr; &nearrow; &nequiv; &nesear; &nexists; &ngE; &nge; &ngeqq; &ngeqslant; &ngt; &nharr; &ni; &niv; &nlArr; &nlarr; &nle; &nleq; &nleqq; &nleqslant; &nless; &nlt; &nmid; &not &notinva; &notni; &npar; &nprcue; &npre; &nprec; &npreceq; &nrightarrow; &nrtri; &nrtrie; &nsc; &nsccue; &nsce; &nshortparallel; &nsim; &nsimeq; &nsmid; &nspar; &nsqsube; &nsqsupe; &nsube; &nsubset; &nsubseteq; &nsubseteqq; &nsucc; &nsucceq; &nsupE; &nsupe; &nsupseteq; &ntgl; &ntilde &ntriangleleft; &ntrianglelefteq; &ntrianglerighteq; &nwarr; &oacute &ocirc &odot; &ograve &ohm; &oint; &oplus; &order; &ordf &ordm &oscr; &oslash &otilde &otimes; &ouml &par; &para &parallel; &phiv; &phmmat; &plankv; &plusb; &plusmn &pm; &pound &pr; &prap; &prcue; &pre; &preccurlyeq; &prnE; &prnap; &prnsim; &propto; &prsim; &qint; &quaternions; &questeq; &quot &rArr; &rBarr; &radic; &rang; &rangle; &raquo &rarr; &rarrb; &rarrlp; &rbarr; &rbrace; &rbrack; &rceil; &rdquor; &real; &realpart; &reals; &reg &rfloor; &rightarrow; &rightarrowtail; &rightharpoondown; &rightharpoonup; &rightrightarrows; &rightsquigarrow; &rightthreetimes; &rlarr; &rlhar; &rmoustache; &robrk; &rsquor; &rtrie; &rtrif; &sc; &scap; &sccue; &sce; &scnap; &scsim; &searr; &searrow; &sect &setminus; &setmn; &sfrown; &shortmid; &shy &sigmaf; &sime; &slarr; &smallsetminus; &smid; &spades; &spar; &sqsube; &sqsubset; &sqsubseteq; &sqsup; &sqsupe; &sqsupseteq; &squ; &square; &squf; &ssmile; &sstarf; &strns; &sube; &subnE; &subne; &subset; &subseteq; &subseteqq; &succeq; &succneqq; &succnsim; &succsim; &sup1 &sup2 &sup3 &supE; &supne; &supset; &supseteq; &supsetneqq; &swarrow; &szlig &tbrk; &tdot; &therefore; &thetav; &thickapprox; &thicksim; &thinsp; &thkap; &thksim; &thorn &tilde; &times &top; &tosa; &triangleleft; &trianglelefteq; &triangleright; &trianglerighteq; &trie; &twixt; &twoheadleftarrow; &uArr; &uacute &ucirc &ugrave &uharr; &ulcorn; &uml &uml; &uparrow; &updownarrow; &upharpoonleft; &upharpoonright; &uplus; &upsilon; &urcorn; &utri; &utrif; &uuarr; &uuml &vArr; &vDash; &varepsilon; &varnothing; &varphi; &varpi; &varpropto; &varr; &varrho; &varsigma; &varsubsetneq; &varsubsetneqq; &varsupsetneq; &vartheta; &vartriangleright; &vee; &verbar; &vltri; &vnsup; &vprop; &vsupnE; &wedge; &weierp; &wreath; &xcap; &xcirc; &xcup; &xdtri; &xharr; &xlarr; &xoplus; &xotime; &xrArr; &xrarr; &xsqcup; &xuplus; &xutri; &yacute &yen &yuml &zeetrf;
</details>
In this list are many named character references without a trailing ;. This is because HTML does not require one in all cases. There's another behavior concerning numeric character references where the trailing ; isn't required at certain boundaries.
Further, whether or not the trailing ; is required is subject to the ambiguous ampersand rule, which guards a legacy behavior for certain query args in URL attributes which weren't properly encoded.
---
Outputs from this PR
The FromPHP column shows how html_entity_decode( $input, ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML5 ) would decode the input.
The Data and Attribute columns show how the HTML API decodes the text in the context of markup (data) and of an attribute value (attribute). These are different in HTML, and unfortunately PHP does not provide a way to differentiate them. The main difference is in the so-called "ambiguous ampersand" rule which allows many "entities" to be written without the terminating semicolon ; (though not all of the named character references may do this). In attributes, however, some of these can look like URL query params. E.g. is ¬=dogs supposed to be ¬=dogs or a query arg named not whose value is dogs? HTML chose to ensure the safety of URLs and forbid decoding character references in these ambiguous cases.
Outputs from a browser
I've compared Firefox and Safari. The middle column shows the data value and the right column has extracted the title attribute of the input and set it as the innerHTML of the TD.
The empty boxes represent unrendered Unicode charactered. While some characters, like the null byte, are replaced with a Replacement Character �, "non-characters" are passed through, even though they are parser errors.
Trac ticket:
@jonsurrell commented on PR #6387:
15 months ago
#5
I've pushed a few changes, I think there was some inconsistency in null/false in the read_character_reference return type. This resulted in calling strlen( null ) showing up in tests as:
1) Tests_HtmlApi_WpHtmlDecoder::test_detects_ascii_case_insensitive_attribute_prefixes with data set "javascript&colon" ('javascript&colon', 'javascript:')
strlen(): Passing null to parameter #1 ($string) of type string is deprecated
/var/www/src/wp-includes/html-api/class-wp-html-decoder.php:63
/var/www/tests/phpunit/tests/html-api/wpHtmlDecoder.php:27
/var/www/vendor/bin/phpunit:122
From the PHP manual http://www.php.net/mktime :
is_dst
The parameter is deprecated in PHP 5.1.0 and "the new timezone handling features should be used instead."
For servers in the UK, it seems PHP is "figuring out" that they are on British time in the summer, but in the winter it gets it slightly wrong and assumes the machine is permanently on UTC (hence UTC not GMT).
Perhaps this should be addressed in Wordpress along with other issues surrounding timezones, for example automatic DST?
Other timezones could be affected if PHP makes the wrong guess as to the timezone.