Experiment: Discovery docs diff with in generator improvements

[ohmayr]

This PR demonstrates the discovery docs changes with the following changes in rst.py file in the generator:

# Cache for the few complex items we actually send to pandoc
_RAW_RST_CACHE: Dict[str, str] = {}

def _tuned_fast_convert(text: str) -> Optional[str]:
    """
    Converts Markdown to RST using pure Python.
    Only falls back to Pandoc for Tables and Images.
    """
    # --- 1. FALLBACKS ---
    # Tables (pipe surrounded by spaces) or Images (![).
    # We allow "][" (Reference Links) to be handled by Python now.
    if (re.search(r" \| ", text) or re.search(r"\|\n", text)) or "![" in text:
        return None

    # --- 2. CONVERSION ---

    # A. CODE BLOCKS: `code` -> ``code``
    # CRITICAL: Run this FIRST. This ensures we handle existing backticks
    # before we create NEW backticks for links.
    # (?<!:) ensures we don't break Sphinx roles like :class:`MyClass`
    converted = re.sub(r"(?<!:|`)`([^`]+)`(?!`)", r"``\1``", text)
    
    # B. REFERENCE LINKS: [Text][Ref] -> `Text <Ref>`__
    # We fix the broken documentation by converting these to valid RST links.
    # Since step A is done, these new backticks will NOT be doubled.
    converted = re.sub(r"\[([^\]]+)\]\[([^\]]+)\]", r"`\1 <\2>`__", converted)

    # C. STANDARD LINKS: [Text](URL) -> `Text <URL>`__
    converted = re.sub(r"\[([^\]]+)\]\(([^)]+)\)", r"`\1 <\2>`__", converted)

    # D. BOLD/ITALICS:
    converted = re.sub(r"(?<!_)\b_([^_]+)_\b(?!_)", r"*\1*", converted)

    # E. HEADINGS: # Heading -> Heading\n=======
    converted = re.sub(r"^# (.*)$", r"\1\n" + "=" * 10, converted, flags=re.MULTILINE)
    converted = re.sub(r"^## (.*)$", r"\1\n" + "-" * 10, converted, flags=re.MULTILINE)

    # F. LISTS: Markdown (- item) needs a preceding newline for RST.
    converted = re.sub(r"(\n[^-*].*)\n\s*([-*] )", r"\1\n\n\2", converted)

    return converted

def rst(
    text: str,
    width: int = 72,
    indent: int = 0,
    nl: Optional[bool] = None,
    source_format: str = "commonmark",
):
    # 1. Super Fast Path: No special chars? Just wrap.
    if not re.search(r"[|*`_[\]#]", text):
        answer = wrap(text, indent=indent, offset=indent + 3, width=width - indent)
        return _finalize(answer, nl, indent)

    # 2. Check Cache
    if text in _RAW_RST_CACHE:
        raw_rst = _RAW_RST_CACHE[text]
    else:
        # 3. Try Tuned Python Convert (Fastest)
        fast_result = _tuned_fast_convert(text)
        
        if fast_result is not None:
            raw_rst = fast_result.strip()
        else:
            # 4. Fallback to Pandoc (Only for Tables/Images)
            raw_rst = pypandoc.convert_text(
                text, "rst", format=source_format, extra_args=["--columns=1000"]
            ).strip()
            
        _RAW_RST_CACHE[text] = raw_rst

    # 5. Python Formatting
    if "::" in raw_rst or ".. code" in raw_rst:
        answer = raw_rst.replace("\n", f"\n{' ' * indent}")
    else:
        answer = wrap(raw_rst, indent=indent, offset=indent, width=width - indent)

    return _finalize(answer, nl, indent)


def _finalize(answer, nl, indent):
    """Helper to handle trailing newlines and quotes."""
    if nl or ("\n" in answer and nl is None):
        answer += "\n" + " " * indent
    if answer.endswith('"'):
        answer += "."
    return answer

Fixes #<issue_number_goes_here> 🦕

[gemini-code-assist]

Summary of Changes

Hello @ohmayr, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces significant internal enhancements to the client library generator, focusing on performance, build efficiency, and code quality tooling. It integrates a new performance logging system to provide insights into execution times, transitions the code formatting workflow to the high-performance ruff tool, and optimizes the Docker build process for faster iterations. These changes are accompanied by updates to client library generation configurations, including the addition of new dialogflow-cx and discoveryengine libraries.

Highlights

• Performance Logging Infrastructure: Implemented a new performance logging system within the generator's CLI tool (cli.py), featuring a @track_time decorator to measure and log the execution duration of key functions, controlled by an ENABLE_PERF_LOGS environment variable.
• Code Formatting Tool Transition: Migrated the code formatting and import sorting process from black and isort to ruff, a faster and more integrated tool, enhancing the efficiency of the code generation pipeline.
• Dockerfile Optimization: Reorganized dependency installation commands in the Dockerfile to improve Docker layer caching, reducing build times for the generator.
• Client Library Generation Updates: Updated versions and last generated commits for numerous client libraries in .librarian/state.yaml, and added new generation request configurations for google-cloud-dialogflow-cx and google-cloud-discoveryengine.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces significant performance improvements to the generator, primarily by replacing black and isort with the much faster ruff for formatting and import sorting. The Dockerfile has also been optimized for better layer caching, which will speed up local development builds. Additionally, a helpful performance profiling decorator (@track_time) has been added to measure the execution time of key functions. The changes are well-structured and focused on performance. I have a couple of minor suggestions to improve the robustness of the new ruff integration and to clean up the code.

I am having trouble creating individual review comments. Click here to see my feedback.

.generator/cli.py (391)

This log line appears to be commented out. To keep the code clean, it should either be re-enabled if logging is desired here, or removed if it's no longer needed.

.generator/cli.py (392-418)

The subprocess.run calls for ruff use check=False and redirect both stdout and stderr to subprocess.DEVNULL. This will silently ignore any errors that ruff encounters (e.g., parsing errors in the generated Python files), which could mask underlying problems. While the FileNotFoundError is handled, other execution errors will not be reported.

I suggest capturing stderr and logging it as a warning if the subprocess returns a non-zero exit code. This will make debugging much easier if ruff has issues with the generated code.

        # STEP A: Fix Imports (like isort)
        check_run = subprocess.run(
            [
                "ruff", "check",
                "--select", "I",               # Only run Import sorting rules
                "--fix",                       # Auto-fix them
                "--line-length=88",            # Match Black default
                "--known-first-party=google",  # Prevent 'google' moving to 3rd party block
                output
            ],
            check=False,
            capture_output=True,
            text=True
        )
        if check_run.returncode != 0:
            logger.warning(f"Ruff check failed:\n{check_run.stderr}")

        # STEP B: Format Code (like black)
        format_run = subprocess.run(
            [
                "ruff", "format",
                "--line-length=88",            # Match Black default
                output
            ],
            check=False,
            capture_output=True,
            text=True
        )
        if format_run.returncode != 0:
            logger.warning(f"Ruff format failed:\n{format_run.stderr}")