Removing all hooks too aggressive if a priority number is given

[GeertDD]

When you call remove_all_filters() or remove_all_actions() with a priority number (second parameter), all hooks get removed if none is found with the given priority.

Example:

<?php
global $wp_filter;

add_action('my_action', 'callback_foo');
add_action('my_action', 'callback_bar', 20);
var_dump($wp_filter['my_action']);

// At this point only callback_foo is left. Fine.
remove_all_actions('my_action', 20);
var_dump($wp_filter['my_action']);

// Exactly the same function call as above,
// but now all hooks bound to my_action are removed!
remove_all_actions('my_action', 20);
var_dump($wp_filter['my_action']);

Calling remove_all_filters() or remove_all_actions() with a priority should never remove hooks with another priority.