function do_robots() should use admin_url()

[Hrohh]

Dear all,

we have /wp-admin/ hardcoded in do_robots() function. How can i use filter admin_url('admin-ajax.php') ?

<?php
        $site_url = parse_url( site_url() );
        $path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output  .= "Disallow: $path/wp-admin/\n";
        $output  .= "Allow: $path/wp-admin/admin-ajax.php\n";

Thank you.

Trac ticket: https://core.trac.wordpress.org/ticket/63467

### Description

This PR introduces the use of the admin_url() function to dynamically retrieve the admin URL, replacing previously hardcoded values.

### Screenshots

Installation	Before	After
Single Site
Multisite

Updates do_robots() to use filtered admin URL paths instead of hardcoded /wp-admin/ values.

This changes the default robots.txt output to derive the Disallow and Allow paths from admin_url() and admin_url( 'admin-ajax.php' ), while continuing to output only the path portion. This allows installs that filter admin URLs to generate correct robots.txt directives.

A PHPUnit regression test is included to verify that filtered admin paths are reflected in the do_robots() output.

Trac ticket: https://core.trac.wordpress.org/ticket/63467

## Use of AI Tools
AI assistance: Yes
Tool(s): GitHub Copilot
Model(s): GPT-5.4
Used for: Drafting the initial patch and regression test. The final implementation, test coverage, and PR description were reviewed and edited by me.

[masteradhoc]

Hey all

I've just refreshed the PR with test. Thanks to @yogeshbhutkar for the initial PR!
@mukesh27 As you've already checked the initial PR, can i ask you to check the refreshed PR as well?

I think we can Milestone this as we'll for 7.1 now.

Thanks

Thank for for the PR. Closing in favor of ​https://github.com/WordPress/wordpress-develop/pull/11998

[westonruter]

In 62633:

Robots: Use admin_url() for the robots.txt admin paths.

Derive the Disallow and Allow directives in do_robots() from admin_url() and admin_url( 'admin-ajax.php' ) rather than from a hardcoded /wp-admin/ path built off site_url(). Only the URL path portion is emitted, as before, so installs that relocate or filter their admin URL now produce a correct default robots.txt.

Also guard the Content-Type header with a headers_sent() check to avoid a warning when the headers have already been sent.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/11998.
Follow-up to r34985.

Props masteradhoc, yogeshbhutkar, hrohh, westonruter, mukeshpanchal27, 1ucay.
Fixes #63467.