WordPress.org

Make WordPress Core

Opened 4 years ago

Closed 2 months ago

Last modified 4 days ago

#36669 closed enhancement (fixed)

Update SimplePie to Latest Version and possibly include PHP-MF2

Reported by: dshanske Owned by: desrosj
Milestone: 5.5 Priority: normal
Severity: normal Version: 3.5
Component: External Libraries Keywords: has-patch early has-dev-note
Focuses: SimplePie Cc:

Description

SimplePie Version 1.4 is now available.

https://github.com/simplepie/simplepie/releases

It is the first release in 3 years and has a large number of bug fixes. It also is the first version that includes support for new types of feeds as described by the IndieWebCamp community.

These feeds are made up of h-feed and h-entry microformats, which allows the feed to be parsed from the html rather than a separate xml file.

SimplePie will now detect when it finds such a feed and parse them when it finds the php-mf2 parser available.

Attachments (5)

update-simplepie.patch (86.6 KB) - added by slushman 4 years ago.
Patch updating SimplePie to version 1.4.
SimplePie-1.5.2.diff (120.1 KB) - added by wpshades 21 months ago.
The latest release of SimplePie found at - https://github.com/simplepie/simplepie/releases
36669.diff (121.2 KB) - added by Hareesh Pillai 14 months ago.
Adding refreshed patch
36669.2.diff (170.3 KB) - added by desrosj 3 months ago.
36669.3.diff (166.2 KB) - added by desrosj 2 months ago.

Download all attachments as: .zip

Change History (74)

#1 @jorbin
4 years ago

  • Owner set to rmccue
  • Status changed from new to assigned

@rmccue - thoughts?

This ticket was mentioned in Slack in #core by dshanske. View the logs.


4 years ago

@slushman
4 years ago

Patch updating SimplePie to version 1.4.

#3 @slushman
4 years ago

  • Keywords has-patch added

I've run into an issue with a plugin trying to read an RSS feed from a site with a Lets Encrypt (self-signed) cert. It appears this was fixed in [May 2015](https://github.com/simplepie/simplepie/pull/407) and that fix is part of SimplePie 1.4.
Submitting patch to update SimplePie to version 1.4.

This ticket was mentioned in Slack in #core by dshanske. View the logs.


4 years ago

#5 @jhabdas
3 years ago

SimplePie 1.5 has been released. It still does not support PHP 7.0, which is what I believe WP suggests. It seems apt to deprecate or make private fetch_feed() entirely and let the plugins manage the use cases and forward-facing support role for end users.

Last edited 3 years ago by jhabdas (previous) (diff)

#6 @Presskopp
3 years ago

1.5.1 is latest now

#7 @aquakitty
3 years ago

#42515 was marked as a duplicate.

#8 @etruel
2 years ago

Hi all, someone recommended me to tell the following here. I'll try to say some different behaviours that we have in our plugin WPeMatico by reading different feeds.

I can't give too much technical details, but we're attending so much tickets of lot of users with problems using the Simplepie 1.3.1 included in WordPress that we solved just by using the new version 1.5 or newer.

Almost all the new things we found in the 1.5 version (already to download in simplepie.org) was related with fixed issues and improvements. (https://github.com/simplepie/simplepie/releases)

Using 1.3.1 included in WP, some of the feeds with problem are solved with force_feed and others by changing to a known User-agent. (But in 1.5 works by default)

There are feeds giving different errors in 1.3.1 than 1.5. And some of them work well in 1.5 and not in 1.3 (1.3 tried with and without force_feed)

For example http://thenationonlineng.net/feed/
Works well with 1.5.

1.3.1 = ERROR: The feed http://thenationonlineng.net/feed/ cannot be parsed. Simplepie said: This XML document is invalid, likely due to invalid characters. XML error: Reserved XML Name at line 2, column 38

Many errors that produce notices like examples below are fixed easily by using the new 1.5:

ERROR: Feed https://www.cryptocoinsnews.com/feed could not be parsed. (SimplePie said: A feed could not be found at https://www.cryptocoinsnews.com/feed. A feed with an invalid mime type may fall victim to this error, or SimplePie was unable to auto-discover it.. Use force_feed() if you are certain this URL is a real feed.)

ERROR: Feed https://www.youtube.com/feeds/videos.xml?channel_id=UCCWYNcRdRteOF3wIRVeaJZQ could not be parsed. (SimplePie said: cURL error 60: SSL certificate problem: unable to get local issuer certificate)

Hope this helps.

cheers

etruel

#9 @swissspidy
2 years ago

#44309 was marked as a duplicate.

@wpshades
21 months ago

The latest release of SimplePie found at - https://github.com/simplepie/simplepie/releases

#10 @wpshades
21 months ago

  • Focuses SimplePie added
  • Version changed from 4.6 to 4.9.8

Hi everyone!

I have just submitted a patch that includes the latest release of SimplePie(1.5.2).
If there's any way in which I can help test this out, please let me know.

Thanks!

#11 follow-up: @dmenard
21 months ago

Hi!

I would like to second this request.

With WP 4.9.8 and PHP 7.2, the WordPress RSS widget generates many warnings like this:

Warning: A non-numeric value encountered in wordpress\wp-includes\SimplePie\Parse\Date.php on line 694

10	SimplePie->get_items( $start = ???, $end = ??? )			...\class-simplepie.php:2812
11	SimplePie_Item->get_date( $date_format = 'U' )				...\class-simplepie.php:2925
12	SimplePie_Parse_Date->parse( $date = '2018-10-02T08:38:56Z' )		...\Item.php:773
13	SimplePie_Parse_Date->date_w3cdtf( $date = '2018-10-02T08:38:56Z' )	...\Date.php:603

These are caused by the way SimplePie 1.3.1 parses dates and by the new E_WARNINGs introduced in PHP 7.1.

A simple fix for this has been made in july 2016 and is included since SimplePie 1.4.2.

So upgrading SimplePie to the latest version would solve these warnings.

Thanks!

#12 in reply to: ↑ 11 @miunosoft
21 months ago

Just in case for someone who are facing this issue and need to get away from the warnings without modifying the core, here is a plugin, Fix - SimplePie Errors: https://github.com/michaeluno/_fix-simplepie-errors).

#13 @pento
20 months ago

#45436 was marked as a duplicate.

#14 @Znuff
19 months ago

This bug has been present for years and it affects all production websites on current PHP versions.

There have been patches submitted, people even made plugins.

How come this is taking so long to be adopted in the core?

It's spamming our servers logs so hard that we ended up at 3GB in 3 days. It's incredible.

This ticket was mentioned in Slack in #core by dshanske. View the logs.


19 months ago

#16 @pento
18 months ago

#42684 was marked as a duplicate.

#17 @pento
18 months ago

  • Milestone changed from Awaiting Review to 5.2
  • Version 4.9.8 deleted

PHP 5.2 support was dropped in SimplePie 1.4, 1.5 requires PHP 5.3+.

I'm moving this to the 5.2 milestone, when we'll be bumping WordPress' minimum PHP version to 5.6.

#18 @knutsp
16 months ago

#42515 was marked as a duplicate.

#19 @desrosj
16 months ago

  • Keywords early needs-refresh added
  • Milestone changed from 5.2 to 5.3

5.2 beta is in less than 2 days. The latest patch needs to be refreshed to apply cleanly to trunk, and this needs to be tested. Punting to 5.3.

@rmccue do you have any thoughts about this?

#20 @dshanske
16 months ago

@rmccue was assigned this task 3 years ago and has not commented on this or noted his acceptance of the assignment. Can I suggest that, if he is occupied with other things, this revert to unassigned in the hopes someone else will take this on?

#21 @rmccue
16 months ago

  • Owner rmccue deleted

@Hareesh Pillai
14 months ago

Adding refreshed patch

#22 @Hareesh Pillai
14 months ago

  • Keywords needs-testing added; needs-refresh removed

#23 @elexem
13 months ago

not sure I understand the implications of the latest entries in this thread.
Can we expect the error-causing-behaviour to go away sometime soon? Looking into /wp-includes/SimplePie seems to still be on ver 1.3.1 while https://github.com/simplepie/simplepie/releases/ lists 1.5.2 as released early Aug 2018 - so perhaps I'm too incompetent to grasp the monumental hindrances impeding adapting that.
Either way, it would be nice if that could be mended; the nuisant oversight pelts us with error lines we could clearly do without; 91953 in the last cycle.

Cheers

#24 @dshanske
13 months ago

No one with privileges to commit wants to take ownership of it.

#25 follow-up: @Znuff
13 months ago

How about mark this as a BUG instead of an "enhancement" and mark it as a blocker?

The amount of pussyfooting around this issue is astounding.

At this rate, I'm gonna have to patch that file manually until Wordpress 8.0. And I'm sick of patching it every release.

#26 @elexem
13 months ago

I'll have to agree with @Znuff here.

+1 :/

#27 @dshanske
13 months ago

  • Type changed from enhancement to defect (bug)

Okay. I will classify it as a bug...

This ticket was mentioned in Slack in #core by hareesh-pillai. View the logs.


10 months ago

#29 follow-up: @mikeschroder
10 months ago

@stevenkword as Feeds maintainer, what do you think about upgrading this as part of 5.3?

#30 in reply to: ↑ 29 @stevenkword
10 months ago

Replying to mikeschroder:

@stevenkword as Feeds maintainer, what do you think about upgrading this as part of 5.3?

Thanks for putting this on my radar. The current version in WP is 1.3.1, last updated 6 years ago.

I think a major prerequisite for this would be to have better test coverage across the feeds component. I began introducing tests a few years back, but they are still not complete.

I think this is a great idea to inherit the bugfixes, but I'd like to ensure we have 1:1 tests against every function touched in the SimplePie changelogs between 1.3.1 and 1.5. I'd always want this big of a change to land in the release early to allow for a long bake time.

Last edited 10 months ago by stevenkword (previous) (diff)

#31 @dshanske
10 months ago

SimplePie has its own test coverage. Do we need to redo them or simply incorporate them?

#32 @davidbaumwald
10 months ago

  • Keywords early removed
  • Milestone changed from 5.3 to Future Release

This ticket still needs a decision, and with 5.3 Beta 1 only a few days away, this is being moved to Future Release.

#33 @jrf
10 months ago

Related #47746 - the current version used of SimplePie contains a PHP 7.4 compatibility issue.

This issue has been patched upstream, but is not contained in a tagged release yet. Tagging a new release has been requested: https://github.com/simplepie/simplepie/issues/620

#34 follow-up: @jrf
10 months ago

SimplePie has released a new version including the changes needed for PHP 7.4 support: https://github.com/simplepie/simplepie/releases/tag/1.5.3

#35 in reply to: ↑ 34 @stevenkword
10 months ago

  • Keywords early added
  • Milestone changed from Future Release to 5.4

Replying to jrf:

SimplePie has released a new version including the changes needed for PHP 7.4 support: https://github.com/simplepie/simplepie/releases/tag/1.5.3

Due to the late introduction into this cycle, I'm very hesitant for a push. The PHP 7.4 support doesn't concern me necessarily since there is a patch available in #47746, but I would like this sort of request to hit a release early. Let's aim for 5.4 with a tag of 'early' so that we can properly kick the tires. I'm not nearly as worried about breaking Core functionality as I am dependent plugins.

We should do some discovery against the largest plugin consumers of the library and confirm compatibility first.

#36 @desrosj
10 months ago

#44087 was marked as a duplicate.

#37 @Hareesh Pillai
9 months ago

#29238 was marked as a duplicate.

This ticket was mentioned in Slack in #core by picard102. View the logs.


9 months ago

#39 @nasiralamreeki
8 months ago

WordPress really needs to take more seriously updating external libraries. Years between updating libraries from upstream is unacceptable. There’s tremendous benefits to getting these libraries updated faster.

We can’t wait for theme and plugin developers forever to test. Sometimes we have to push the web forward by pushing ahead with changes even if they break themes and plugins as this in turn will push developers to update their code.

We pushed Gutenberg ahead even though most major themes and plugins weren’t ready for it.

#40 follow-up: @Ipstenu
8 months ago

@stevenkword Here's a list of everyone using SimplePie: https://wpdirectory.net/search/01DT3A1NPYN8FRVGACQ4FTPDHE

If you need a more precise search, let me know and I'll get it for you. But this is a constant bugbear with random plugins and we could use this updated.

#41 in reply to: ↑ 40 ; follow-up: @stevenkword
8 months ago

Replying to Ipstenu:

@stevenkword Here's a list of everyone using SimplePie: https://wpdirectory.net/search/01DT3A1NPYN8FRVGACQ4FTPDHE

If you need a more precise search, let me know and I'll get it for you. But this is a constant bugbear with random plugins and we could use this updated.

Thank you for the ping, @Ipstenu! The list is more that sufficient for now. I'm all in favor of implementing the latest stable release of Simple Pie based on trac feedback and having looked over the changelogs. I'd like to reach out to a sample set of plugins from the list to make sure we don't have any blind spots. It seems like a no-brainer from here, but I just want to double check with a few different types of use case plugins (aggregators, podcasts, xml sync, etc) to ensure we've been thorough and allow the opportunity for testing. Now is a great time since we're early into the release cycle.

#42 in reply to: ↑ 41 ; follow-up: @stevenkword
8 months ago

Replying to stevenkword:

Replying to Ipstenu:

@stevenkword Here's a list of everyone using SimplePie: https://wpdirectory.net/search/01DT3A1NPYN8FRVGACQ4FTPDHE

If you need a more precise search, let me know and I'll get it for you. But this is a constant bugbear with random plugins and we could use this updated.

Thank you for the ping, @Ipstenu! The list is more that sufficient for now. I'm all in favor of implementing the latest stable release of Simple Pie based on trac feedback and having looked over the changelogs. I'd like to reach out to a sample set of plugins from the list to make sure we don't have any blind spots. It seems like a no-brainer from here, but I just want to double check with a few different types of use case plugins (aggregators, podcasts, xml sync, etc) to ensure we've been thorough and allow the opportunity for testing. Now is a great time since we're early into the release cycle.

Is anyone able to assist with doing some compatibility checks against plugins from this list with the latest patch applied to trunk?

#43 in reply to: ↑ 42 @etruel
8 months ago

Replying to stevenkword:

Is anyone able to assist with doing some compatibility checks against plugins from this list with the latest patch applied to trunk?

I could make some tests. How can I help ?
I have WordPress beta tester plugin 2.1.0 with WordPress 5.3.1-alpha-46728

Update:: I have WordPress beta tester plugin 2.1.0 with WordPress 5.4-alpha-46750 from trunk
Simplepie 1.3.1

Last edited 8 months ago by etruel (previous) (diff)

#44 @diddledan
6 months ago

Updating simplepie to latest will also close #48615.

Last edited 6 months ago by diddledan (previous) (diff)

This ticket was mentioned in Slack in #core by david.baumwald. View the logs.


5 months ago

#46 @SergeyBiryukov
5 months ago

  • Owner set to SergeyBiryukov
  • Status changed from assigned to reviewing

#47 @dd32
5 months ago

#49451 was marked as a duplicate.

#48 @audrasjb
5 months ago

  • Milestone changed from 5.4 to Future Release

Hi,

With 5.4 Beta 3 approaching and the Beta period reserved for bugs introduced during the cycle, this is being moved to Future Release. If any maintainer or committer feels this should be included or wishes to assume ownership during a specific cycle, feel free to update the milestone accordingly.

#49 follow-ups: @Znuff
5 months ago

Is this a joke? Are you for real? AGAIN?!

#50 in reply to: ↑ 49 @nasiralamreeki
5 months ago

Replying to Znuff:

Is this a joke? Are you for real? AGAIN?!

The kicking these updates including jquery which is updated down the road for years is the disappointing reality of WordPress as a project.

#51 in reply to: ↑ 49 @leemon
3 months ago

Replying to Znuff:

Is this a joke? Are you for real? AGAIN?!

If the bug is not Gutenberg-related, the probability of being fixed is near zero now

#52 in reply to: ↑ 25 @antoine2711
3 months ago

Replying to Znuff:

How about mark this as a BUG instead of an "enhancement" and mark it as a blocker?

The amount of pussyfooting around this issue is astounding.

At this rate, I'm gonna have to patch that file manually until WordPress 8.0. And I'm sick of patching it every release.

+1

This ticket was mentioned in PR #220 on WordPress/wordpress-develop by desrosj.


3 months ago

This PR updates SimplePie to version 1.5.4 for testing.

Trac ticket: https://core.trac.wordpress.org/ticket/36669.

This ticket was mentioned in PR #220 on WordPress/wordpress-develop by desrosj.


3 months ago

This PR updates SimplePie to version 1.5.4 for testing.

Trac ticket: https://core.trac.wordpress.org/ticket/36669.

@desrosj
3 months ago

#55 @desrosj
3 months ago

I'd like to kindly remind everyone that time is a limited resource. The WordPress project is maintained almost entirely by volunteers (with the exception of a very, very small handful of people). While it is frustrating to see an issue repeatedly miss each new release, it is not because the ticket is being avoided. Leaving negative comments does nothing to progress the ticket closer to a resolution, and probably has the opposite affect. Newer contributors will be hesitant to jump in to help.

I've created a PR with the latest version of SimplePie (and included the refreshed patch here), and it looks like all of the tests that are currently included pass.

@stevenkword I dug in and looked at some of the top plugins using SimplePie. I re-ran the scan linked above by @Ipstenu and there were a few additional plugins. I looked at the top 5 plugins ordered by number of active installs. This is what I found:

  • UpdraftPlus: There are 2 occurrences of SimplePie in their code base. One never runs because it is intentionally disabled, and the other just pulls their news blog's RSS feed in and includes it in emails when backups run. I tested this out and looks like it works without issue.
  • Duplicator: Looks like this plugin has a list of all core WordPress files and classes and does not actually utilize the class.
  • WP Statistics: SimplePie is only included in their code base to identify what is crawling/accessing a site.
  • Google Tag Manager: SimplePie is only included in their code base to identify bots accessing the site.
  • Caldera Forms: simplepie is mentioned once, but looks like it is code from Wonolog, a package that is included with Composer.

I also looked at the top 5 plugins ordered by the most occurrences of "simplepie" with at least 1,000 active installs:

  • RSSImport (10,000 active): This plugin worked when I used the shortcode and the default URL [RSSImport], but broke when I attempted to pass a different feed URL to the shortcode with the feedurl argument. I am not sure if this is a block editor, plugin issue, or an issue from the upgrade.
  • WP Social Blogroll (1,000 active): This plugin doesn't even activate on 5.4. Based on the support forums, it stopped working at WordPress version 4.7. It also has not been updated in 7 years.
  • FeedWordPress (30,000 active): The plugin still works and successfully pulls in posts from a feed, but there are a few warnings thrown when SimplePie is upgraded.
Declaration of FeedWordPie_Item::get_id($hash = false) should be compatible with SimplePie_Item::get_id($hash = false, $fn = 'md5') in wp-content/plugins/feedwordpress/feedwordpie_item.class.php on line 4
Declaration of FeedWordPress_Parser::parse(&$data, $encoding) should be compatible with SimplePie_Parser::parse(&$data, $encoding, $url = '') in wp-content/plugins/feedwordpress/feedwordpress_parser.class.php on line 28
  • WPide (50,000 active) and AceIDE (20,000 active): These had a lot of shared code. They only had "simplepie" in strings within JS files.
  • Libsyn Publisher Hub (4,000 active): I was unable to test this because it appears you need an active Libsyn account, which is paid. But, I looked at the occurrences of "simplepie" in the code, and it seems to be very standard use of SimplePie.

This research considered and the 5.5 release cycle being very, very early, I think we could make this change and publish a blog post on Make to test thoroughly to help identify problems. We should still work to increase the overall test coverage.

Another thing I wanted to mention was that I was unable to find a full changelog for SimplePie on the GitHub repository. I could be missing it, though. When we make this change, we should have a full changelog somewhere that we can link to or include in a dev note or documentation.

#56 @dshanske
3 months ago

Current maintainer of SimplePie is fairly responsive and would be interested in any help he could give, as far as he implied to me.

#57 @desrosj
3 months ago

Opened an issue upstream requesting a changelog be added to the official repository.

#58 follow-up: @desrosj
2 months ago

  • Owner changed from SergeyBiryukov to desrosj

Closed out #33003 as reported-upstream as it appears to have been fixed and will be solved once this update is committed.

I also reported a separate PR to help progress PHPCompatibility testing in #49922. I plan on requesting a new release from the maintainer so those fixes can be included. But if that doesn't happen in time, the changes in that PR should be included in this update.

Spent some time today building a proper changelog upstream: https://github.com/simplepie/simplepie/pull/646

There are several other SimplePie tickets (#37763, #21872, #42254, #29204, #43357 at first glance - I am sure there are more). The last box that should be checked before updating the library is to figure out if:

  • Will be solved by this update.
  • Will be made worse by this update.
  • Will become irrelevant after this update.

This will hopefully identify any additional compatibility issues or blockers not being identified by the current less than adequate test coverage in Core.

#59 in reply to: ↑ 58 @antoine2711
2 months ago

Hi, I just want to take the time to thank you for you work on this issue. It's good to see it going forward. It's very much appreciated. Best Regards, Antoine

Replying to desrosj:

Closed out #33003 as reported-upstream as it appears to have been fixed and will be solved once this update is committed.

I also reported a separate PR to help progress PHPCompatibility testing in #49922. I plan on requesting a new release from the maintainer so those fixes can be included. But if that doesn't happen in time, the changes in that PR should be included in this update.

Spent some time today building a proper changelog upstream: https://github.com/simplepie/simplepie/pull/646

There are several other SimplePie tickets (#37763, #21872, #42254, #29204, #43357 at first glance - I am sure there are more). The last box that should be checked before updating the library is to figure out if:

  • Will be solved by this update.
  • Will be made worse by this update.
  • Will become irrelevant after this update.

This will hopefully identify any additional compatibility issues or blockers not being identified by the current less than adequate test coverage in Core.

This ticket was mentioned in Slack in #core by desrosj. View the logs.


2 months ago

#61 @johnbillion
2 months ago

I spent a couple of hours looking through SimplePie changes since 1.3.3 to see if I could identify anything that gives me cause for concern, and the tl;dr is no I didn't. The majority of the changes are bug fixes and several iterations on PHP compatibility.

I think at this point our best approach is to get this into 5.5 early (ie. ASAP), followed up by communication with the authors of the plugins mentioned above who are using SimplePie directly and ask them to do some testing and check for deprecated notices or warnings. We could extend that to plugins that use fetch_feed() but that will probably have diminishing returns.

1.5.5 was just tagged, I believe @desrosj is going to submit a patch to include the latest changes.

@desrosj
2 months ago

#62 @desrosj
2 months ago

  • Keywords commit added; needs-testing removed
  • Milestone changed from Future Release to 5.5
  • Type changed from defect (bug) to enhancement
  • Version set to 3.5

36669.3.diff updates to 1.5.5, which only includes a PHP compatibility fix (see #49922), and some doc fixes.

Also, marking as 3.5 since that was the last release this was updated.

Build is passing on my attached PR. Let's get this in so it can be more widely tested.

#63 @desrosj
2 months ago

Also, forgot to mention that @johnbillion and I discussed including the new Redis and Memcached cache classes and we decided it was best to include them.

  • SimplePie won't know they are not present.
  • We don't currently exclude anything else in the library from inclusion.
  • Makes it easier to continue maintaining the library.

#64 @desrosj
2 months ago

  • Resolution set to fixed
  • Status changed from reviewing to closed

In 47733:

External Libraries: Update the SimplePie library to the latest version (1.5.5).

This brings SimplePie in sync with the most up to date version, 1.5.5.

This update brings many bug fixes, small enhancements, and PHP compatibility fixes for newer versions of PHP.

For a full list of changes, see https://github.com/simplepie/simplepie/blob/master/CHANGELOG.md#155-may-1-2020.

Props dshanske, slushman, etruel, wpshades, dmenard, desrosj, hareesh-pillai, stevenkword, jrf, Ipstenu, johnbillion.
Fixes #36669.

This ticket was mentioned in Slack in #core by ipstenu. View the logs.


2 months ago

#67 @ocean90
8 weeks ago

#42515 was marked as a duplicate.

#68 @shojibur
4 weeks ago

#50315 was marked as a duplicate.

#69 @desrosj
4 days ago

  • Keywords has-dev-note added; commit removed

This was noted in the following dev note: https://make.wordpress.org/core/2020/07/01/external-library-updates-in-wordpress-5-5-call-for-testing/.

There was also a post published on the Make Plugins blog in early May: https://make.wordpress.org/plugins/2020/05/01/simplepie-updated-for-wordpress-5-5/

The plugin team also should have sent an email to all plugins with active installs that appear to be using the library based on some rough static searches on wpdirectory.net.

Note: See TracTickets for help on using tickets.