ci: reproduce default-Windows wheel install to guard MAX_PATH (#29597)

* ci: reproduce default-Windows wheel install to guard MAX_PATH

The existing using_litellm_on_windows job installs the project with
`uv sync`, an editable source install that never copies package files
into a deep site-packages path, so it cannot see the 260-char MAX_PATH
overflow that breaks `pip install litellm` on default Windows. The
content-filter benchmark fixtures have hit that limit three times
(#21941, #22039, #29536), each caught only after release.

This adds a guard to the same job that builds the wheel and installs it
the way an end user would: into a venv whose site-packages prefix is
padded to a realistic worst-case Windows length (~100 chars), then
asserts the install completes and litellm imports. Any packaged path
long enough to bust MAX_PATH at that prefix is reported up front, so the
check is deterministic regardless of the runner's long-path setting,
while the real install also covers failure modes a length heuristic
cannot (half-unpacked packages, reserved names, case collisions).

This commit is the guard only; on the current tree it correctly fails
because nine fixtures still exceed the limit. The rename that brings
them back under it follows on this branch.

* fix(packaging): shorten content-filter benchmark fixtures under MAX_PATH

The 10 content-filter benchmark result fixtures used the legacy
block_{topic}_-_contentfilter_({yaml}).json naming, up to 176 chars
inside the wheel, which busts the Windows 260-char MAX_PATH limit once
extracted under a realistic site-packages prefix and aborts
`pip install litellm` on default Windows.

Rename them to the short {topic}_cf.json scheme that
_save_confusion_results already emits today (it splits the label on the
em-dash and writes f"{topic}_cf"), matching the insults_cf.json and
investment_cf.json files fixed earlier. Re-running the eval suite now
regenerates these same short names rather than recreating the long ones.

This drops the longest packaged path from 176 to 128, so the guard added
in the previous commit goes from red to green with a 32-char margin.

* test(windows): tidy MAX_PATH guard per review

Close the wheel zip via a context manager rather than leaning on
refcount collection, and select the wheel under dist/ by newest mtime so
a stale artifact from an earlier build cannot be tested instead of the
one just produced. Also pin down the venv-depth formula with a short
note: the +2 is the separator joining the venv root to "Lib" plus the
trailing separator before the entry, which lands the simulated
site-packages prefix at exactly 100 chars.

.circleci/config.yml

@@ -182,7 +182,14 @@ jobs:
       - run:
           name: Run Windows-specific test
           command: |
-            uv run --no-sync python -m pytest tests/windows_tests/test_litellm_on_windows.py -v
+            uv run --no-sync python -m pytest tests/windows_tests/ -v
+      - run:
+          name: Guard against MAX_PATH-busting packaged wheel paths
+          environment:
+            UV_HTTP_TIMEOUT: "300"
+          command: |
+            uv build --wheel --out-dir dist
+            uv run --no-sync python tests/windows_tests/check_windows_wheel_install.py
 
   local_testing_part1:
     docker:

tests/windows_tests/check_windows_wheel_install.py

@@ -0,0 +1,76 @@
+"""Reproduce a default-Windows ``pip install litellm`` to catch the 260-char
+MAX_PATH regression that content-filter benchmark fixtures keep reintroducing
+(#21941, #22039, #29536). Run after ``uv build --wheel --out-dir dist``.
+"""
+
+import glob
+import os
+import subprocess
+import sys
+import zipfile
+
+MAX_PATH = 260
+# Worst-case Windows site-packages prefix: long profile name + roaming AppData venv.
+WORST_CASE_PREFIX = 100
+
+
+def overlong_install_paths(wheel, prefix_len=WORST_CASE_PREFIX, max_path=MAX_PATH):
+    with zipfile.ZipFile(wheel) as zf:
+        names = zf.namelist()
+    return sorted(
+        (n for n in names if prefix_len + len(n) > max_path), key=len, reverse=True
+    )
+
+
+def _deep_venv_dir(target_prefix=WORST_CASE_PREFIX):
+    drive = os.path.splitdrive(os.getcwd())[0] or "C:"
+    root = drive + os.sep + "lmwin" + os.sep
+    # +2: the sep joining the venv root to "Lib", plus the trailing sep before the entry
+    suffix = len(os.path.join("Lib", "site-packages")) + 2
+    return root + "x" * (target_prefix - suffix - len(root))
+
+
+def _run(cmd):
+    print("+ " + subprocess.list2cmdline(cmd), flush=True)
+    return subprocess.call(cmd)
+
+
+def main():
+    wheels = glob.glob(os.path.join("dist", "*.whl"))
+    if not wheels:
+        print("::error::no wheel in dist/; run `uv build --wheel --out-dir dist` first")
+        return 1
+    wheel = max(wheels, key=os.path.getmtime)
+
+    offenders = overlong_install_paths(wheel)
+    if offenders:
+        print(
+            f"::error::{len(offenders)} packaged path(s) bust the Windows MAX_PATH limit "
+            f"at a {WORST_CASE_PREFIX}-char install prefix:"
+        )
+        for n in offenders[:15]:
+            print(f"  on-disk {WORST_CASE_PREFIX + len(n):4}  {n}")
+        return 1
+
+    venv = _deep_venv_dir()
+    os.makedirs(os.path.dirname(venv), exist_ok=True)
+    if _run(["uv", "venv", venv]) != 0:
+        return 1
+    python = os.path.join(venv, "Scripts", "python.exe")
+    if _run(["uv", "pip", "install", "--python", python, wheel]) != 0:
+        print(
+            f"::error::installing {os.path.basename(wheel)} into a deep prefix failed"
+        )
+        return 1
+    if _run([python, "-c", "import litellm; import litellm.types.utils"]) != 0:
+        print("::error::litellm did not import after install (half-unpacked package)")
+        return 1
+
+    print(
+        f"ok: {os.path.basename(wheel)} installs into a worst-case prefix and imports"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/windows_tests/test_check_windows_wheel_install.py

@@ -0,0 +1,36 @@
+import zipfile
+
+from check_windows_wheel_install import (
+    MAX_PATH,
+    WORST_CASE_PREFIX,
+    overlong_install_paths,
+)
+
+
+def _wheel(tmp_path, *entry_names):
+    path = tmp_path / "pkg.whl"
+    with zipfile.ZipFile(path, "w") as zf:
+        for name in entry_names:
+            zf.writestr(name, "{}")
+    return str(path)
+
+
+def test_flags_entry_one_char_over_budget(tmp_path):
+    busts = "a" * (MAX_PATH - WORST_CASE_PREFIX + 1)
+    assert overlong_install_paths(_wheel(tmp_path, busts)) == [busts]
+
+
+def test_allows_entry_exactly_at_budget(tmp_path):
+    at_limit = "a" * (MAX_PATH - WORST_CASE_PREFIX)
+    assert (
+        overlong_install_paths(_wheel(tmp_path, at_limit, "litellm/__init__.py")) == []
+    )
+
+
+def test_orders_offenders_longest_first(tmp_path):
+    longer = "a" * (MAX_PATH - WORST_CASE_PREFIX + 5)
+    shorter = "b" * (MAX_PATH - WORST_CASE_PREFIX + 1)
+    assert overlong_install_paths(_wheel(tmp_path, shorter, longer)) == [
+        longer,
+        shorter,
+    ]