Arabic stopwords comparison

[alex-ye]

WordPress uses simple string comparison in WP_Query->parse_search_terms() function, which is fine for many languages but Arabic need more than that to provide a really smart search!

There are some chars need to removed before the string comparison, like:

Hamza
​http://en.wikipedia.org/wiki/Hamza

Diacritics
​http://en.wikipedia.org/wiki/Arabic_diacritics

Tāʼ marbūṭah
​http://en.wikipedia.org/wiki/Taw

If this cannot be in core, Could you at least add some filters to do it ?!

[alex-ye]

IF THIS ACCEPTED IN CORE:
We could use some functions from ArPHP project:
​http://sourceforge.net/projects/ar-php/

[azaozz]

If I understand correctly: in some cases in Arabic the search terms have to be additionally processed. Would the filter in 25585.patch do the job?

[azaozz]

Moving to 3.7 for consideration as this is new functionality.

[alex-ye]

Replying to azaozz:

Would the filter in 25585.patch do the job?

Yeah, but..
1- I can't re-use the query stopwords list, I wonder why the function is protected?
2- I would like to cancel the strtolower and old stopwords checks for performance reasons.
3- Why we don't introduce sanitize_search_term function , so this functionality could be used in some other places.

[alex-ye]

I am working now on a plugin that works on @azaozz patch:
​https://github.com/nash-ye/wp-arabic-stopwords

[nacin]

Best I can tell, you cannot add hamza, diacritics, and tāʼ marbūṭah to the stopwords list because the list only operates on entire words, not the stripping of individual characters, correct?

Even without the wp_search_stopwords filter, searching is still better in 3.7 than it was in 3.6. We could also add a filter to the end of the parse_search_terms() method. But at that point, I think the crazy regular expression in parse_search() should possibly be moved into it, and search_terms_count should be set after? Right now, it's possible for parse_search_terms() to return less terms than is specified in search_terms_count. Is that OK?

[alex-ye]

Replying to nacin:

Best I can tell, you cannot add hamza, diacritics, and tāʼ marbūṭah to the stopwords list because the list only operates on entire words, not the stripping of individual characters, correct?

Yes, If we can't remove those chars from the user input before the stopwords comparison, the Arabic translators may need to write every stopword 3-5 times! as you know we can't trust the user input.

We could also add a filter to the end of the parse_search_terms() method. But at that point, I think the crazy regular expression in parse_search() should possibly be moved into it, and search_terms_count should be set after?

1+

[azaozz]

I think the crazy regular expression in parse_search() should possibly be moved into it, and search_terms_count should be set after? Right now, it's possible for parse_search_terms() to return less terms than is specified in search_terms_count.

Yeah, it can be moved to the proposed filter so plugins could change the pattern for specific languages.

The idea is to remove single letter terms from the search. The pattern /^\p{L}$/u is the safest way to match a single letter in any language. It's not particularly fast as it looks through the Unicode character properties. A better (but quite slower) pattern could be /^\p{L}\p{M}*|\p{Z}|\p{P}|\p{C}$/u which also matches separators (any kind of whitespace or invisible separators), punctuation, and invisible control characters and unused code points.

search_terms_count is the count before the terms were cleaned. It's used to determine if the sorting would use AND and OR for the title, or just the sentence match. This part of parse_search_order() has gone through quite a few changes, maybe there is a simpler way to do that now.

[alex-ye]

I will try to work on a new patch.

[alex-ye]

Replying to azaozz:

The idea is to remove single letter terms from the search. The pattern /^\p{L}$/u is the safest way to match a single letter in any language. It's not particularly fast as it looks through the Unicode character properties. A better (but quite slower) pattern could be /^\p{L}\p{M}*|\p{Z}|\p{P}|\p{C}$/u which also matches separators (any kind of whitespace or invisible separators), punctuation, and invisible control characters and unused code points (​more info).

See the example below, I have used the 'str_replace' function and I will try to apply your suggestion about the RegExp:

function ArWP_normalize( $str ) {

	// Normalize the Alef.
	$str = str_replace( array(
		'أ','إ','آ'
	), 'ا', $str );

	// Normalize the Diacritics.
	$str = str_replace( array(
		'َ','ً','ُ','ٌ','ِ','ٍ','ْ','ّ'
	), '', $str );

	// Return the new string.
	return $str;

} // end ArWP_normalize()

Is there a simple function in WordPress core to get the unicode code point?!

[nacin]

This is too late for 3.7, but I'd be happy to revisit it. It would be great to robustly support stopwords in other languages. For now, it's no worse than it was in 3.6.

[netweb]

Related: #26670

[chriscct7]

Closing as maybelater. Complete lack of interest on the feature on the ticket over the last 2 years. Feel free to reopen when more interest re-emerges (particularly if there's a patch)