Ticket #4421 (closed defect (bug): fixed)

Opened 5 years ago

Last modified 4 years ago

MT and WP Importers memory usage 2.3 fixes

Reported by: ryan Owned by: ryan
Priority: normal Milestone: 2.3
Component: Administration Version: 2.2
Severity: normal Keywords: import importer memory performance
Cc:

Description

Most of the importers slurp entire files into memory and store the results of parsing into memory, making for a big footprint. They also use lots of unnecessary regular expressions. Importers should parse files one line at a time and write posts to the DB as they are processed to help keep us within memory limits.

Attachments

mt.diff Download (16.4 KB) - added by ryan 5 years ago.
Reworked MT importer
import-memory-reduction-r6606.patch Download (2.1 KB) - added by tellyworth 5 years ago.

Change History

ryan — 5 years ago

Reworked MT importer

comment:1   ryan — 5 years ago

(In [5684]) MT importer rework. see #4421

tellyworth — 5 years ago

comment:2   tellyworth — 5 years ago

import-memory-reduction-r6606.patch reduces memory usage in the WP importer by avoiding the use of file(). It still doesn't process entries one by one, but it's a start.

comment:3   ryan — 5 years ago

(In [5802]) Don't slurp in entire file. Props tellyworth. see #4421

comment:4   foolswisdom — 4 years ago

  • Status changed from new to closed
  • Resolution set to fixed
  • Summary changed from Importers are memory hungry to MT and WP Importers memory performance 2.3 fixes

comment:5   foolswisdom — 4 years ago

  • Summary changed from MT and WP Importers memory performance 2.3 fixes to MT and WP Importers memory usage 2.3 fixes
Note: See TracTickets for help on using tickets.