Adding "fields" to arguments array for get_posts()/query_posts()/WP_Query()

[mikeschinkel]

Hi all,

I find myself more and more often needing to get a list of post IDs so I can call another WordPress database API function to include or exclude those posts. Calling get_posts() on a potentially large number of records and passing all the fields (especially post_content) between MySQL and WordPress is hugely inefficient when I only need the one ID.

My two (2) options are:

1.) Code is directly in SQL. This is easy but I know it's definitely not a best practice and I would like to use the WordPress API wherever possible.

2.) To use a post_fields hook. Problem is that those are global and I have to start wrapping logic around my code to ensure I don't accidentally break some plugin or some other part of WordPress (this approach is much like trying to secure a server by starting with all the attack vectors open and then trying to close them all.)

So I'd like to propose we simply add "fields" as a recognized argument for get_posts(), i.e.

$posts = get_posts(array(
   'fields' => 'ID,post_title',
   'post_type' => 'movie',
   'post_status' => 'publish',
   'order' => 'ASC',
   'posts_per_page' => -1
));

I know I could make the same argument for joins, where, orderbys et. al. but I'd argue this is enough of a special case it could really use some early attention. For a query on the home page of a high traffic site that changes often enough to not be able to be cached this tiny change could make a major difference in performance.

If you'd like to see a use case here is one:

• ​Removing Duplicate Custom Taxonomy Terms from within a Dropdown Select?

Unfortunately I still struggle with creating patches even though I have done it a few times in the past but each time I seem to have to start relearning from scratch. I seem to have a mental block for some reason on patches (​''can someone help me with this?'') so I haven't gone ahead and written a patch but will tackle it if I get the task blessed.

-Mike

[mikeschinkel]

Update on this: The example use case turned out not to be exactly as I expect; that use case need taxonomy term IDs based on a post. Similar use-case though.

[scribu]

I think this is a good idea, similar to how we have a 'fields' argument in get_terms().

[filosofo]

I agree that this would be helpful. I am continually iterating through an array of post objects just to populate another array with a single object property. This would eliminate that extra runtime work and reduce db work.

[nacin]

Since you are on OS X you may want to look at Mark's post: ​http://markjaquith.wordpress.com/2005/11/02/my-wordpress-toolbox/.

[mikeschinkel]

Replying to nacin:

Since you are on OS X you may want to look at Mark's post: ​http://markjaquith.wordpress.com/2005/11/02/my-wordpress-toolbox/.

Thanks. I hadn't seen that in a while (not since I was on Windows) so it's helpful, but doesn't address my key issue. Rather than crud up this thread with discussions of patching I'll email you; reply to let me know if email is okay.

[mikeschinkel]

Replying to scribu:

I think this is a good idea, similar to how we have a 'fields' argument in get_terms().

Replying to filosofo:

I agree that this would be helpful. I am continually iterating through an array of post objects just to populate another array with a single object property. This would eliminate that extra runtime work and reduce db work.

Awesome, thanks.

Note: get_terms() delineates a named set of fields: all, ids, id=>parent, names and count. Would we do the same or just let it be free-form?

I like the consistency of the former but the flexibility of the latter. There have been times when I needed fields from get_terms() that just were not available via $args and having it freeform would solve that issue. Also, if we have to delineate them we'll have to go into decision mode to see what will be our options.

All in all I think I vote for simple field list.

Or we could reserve fields for a name sets and instead add a field_list in 3.1 now which would be freeform. Later we could add fields if needed and it would override field_list (or vice-versa?) That would give both flexibility and consistency.

[scribu]

I think we can have both:

get_terms() style:

'fields' => 'ids'

individual fields:

'fields' => array('ID', 'post_title')

As a follow up, we could add the second style to get_terms() too.

[mikeschinkel]

Replying to scribu:

I think we can have both:

get_terms() style:

'fields' => 'ids'

individual fields:

'fields' => array('ID', 'post_title')

As a follow up, we could add the second style to get_terms() too.

Cool!

How would that work with the URL style of querying? i.e. something like?

fields=ids&post_type=movie&post_status=publish&order=ASC&posts_per_page=-1

And?

fields=(ID,post_title)&post_type=movie&post_status=publish&order=ASC&posts_per_page=-1

Of course the latter would be ugly is urlencoded, but...

And what would be the options made available? The only two that come to mind are id+title, id+slug and * the latter of which is of course the default, but I'm sure there are other use-case needs that don't occur to me at the moment.

[scribu]

How would that work with the URL style of querying?

I think this would work:

fields[]=ID&fields[]=post_title

but I wouldn't worry too much about it. If you need specific fields, you're likely to pass an array or args anyway.

And what would be the options made available?

'all', 'ids', 'id=>parent'

[scribu]

Reason why we need this: #15033 #15052

[scribu]

[14777.diff] accepts 'ids' and 'id=>parent' as arguments for 'fields'.

It returns immediately after the query is executed, to prevent errors from plugins filtering 'the_posts'.

[scribu]

Related: #14405

[ryan]

My first concern is cache pollution, but it looks it this is handled by returning early. Seems good.

[scribu]

(In [15982]) Introduce 'fields' query var to WP_Query. See #14777

[scribu]

Next enhancement would be for 'fields' to accept an array.

[scribu]

Closing as fixed for 3.1. Open a new ticket for further enhancements.

[SergeyBiryukov]

Follow-up: #19866