Skip to content

doc db to include types.* #6391

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
youknowone merged 3 commits into from
Dec 10, 2025
Merged

doc db to include types.* #6391

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3d0b809
Add types.* for doc/generate.py
youknowone Dec 10, 2025
2a16d63
udpate-doc-db workflow
youknowone Dec 10, 2025
b2367ac
Update doc DB for CPython 3.13.9
github-actions[bot] Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 27 additions & 14 deletions .github/workflows/update-doc-db.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Update doc DB

permissions:
contents: read
contents: write

on:
workflow_dispatch:
Expand All @@ -10,11 +10,14 @@ on:
description: Target python version to generate doc db for
type: string
default: "3.13.9"
ref:
description: Branch to commit to (leave empty for current branch)
type: string
default: ""

defaults:
run:
shell: bash
working-directory: ./crates/rustpython-doc

jobs:
generate:
Expand All @@ -30,19 +33,19 @@ jobs:
with:
persist-credentials: false
sparse-checkout: |
crates/rustpython-doc
crates/doc

- uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0
with:
python-version: ${{ inputs.python-version }}

- name: Generate docs
run: python ./generate.py
run: python crates/doc/generate.py

- uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
with:
name: doc-db-${{ inputs.python-version }}-${{ matrix.os }}
path: "crates/rustpython-doc/generated/*.json"
path: "crates/doc/generated/*.json"
if-no-files-found: error
retention-days: 7
overwrite: true
Expand All @@ -53,26 +56,25 @@ jobs:
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
persist-credentials: false
sparse-checkout: |
crates/rustpython-doc
ref: ${{ inputs.ref || github.ref }}
token: ${{ secrets.AUTO_COMMIT_PAT }}

- name: Download generated doc DBs
uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
with:
pattern: "doc-db-${{ inputs.python-version }}-**"
path: crates/rustpython-doc/generated/
path: crates/doc/generated/
merge-multiple: true

- name: Transform JSON
run: |
# Merge all artifacts
jq -s "add" --sort-keys generated/*.json > generated/merged.json
jq -s "add" --sort-keys crates/doc/generated/*.json > crates/doc/generated/merged.json

# Format merged json for the phf macro
jq -r 'to_entries[] | " \(.key | @json) => \(.value | @json),"' generated/merged.json > generated/raw_entries.txt
jq -r 'to_entries[] | " \(.key | @json) => \(.value | @json),"' crates/doc/generated/merged.json > crates/doc/generated/raw_entries.txt

OUTPUT_FILE='src/data.inc.rs'
OUTPUT_FILE='crates/doc/src/data.inc.rs'

echo -n '' > $OUTPUT_FILE

Expand All @@ -83,13 +85,24 @@ jobs:
echo '' >> $OUTPUT_FILE

echo "pub static DB: phf::Map<&'static str, &'static str> = phf::phf_map! {" >> $OUTPUT_FILE
cat generated/raw_entries.txt >> $OUTPUT_FILE
cat crates/doc/generated/raw_entries.txt >> $OUTPUT_FILE
echo '};' >> $OUTPUT_FILE

- uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
with:
name: doc-db-${{ inputs.python-version }}
path: "crates/rustpython-doc/src/data.inc.rs"
path: "crates/doc/src/data.inc.rs"
if-no-files-found: error
retention-days: 7
overwrite: true

- name: Commit and push (non-main branches only)
if: github.ref != 'refs/heads/main' && inputs.ref != 'main'
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
if [ -n "$(git status --porcelain)" ]; then
git add crates/doc/src/data.inc.rs
git commit -m "Update doc DB for CPython ${{ inputs.python-version }}"
git push
fi
Comment on lines +99 to +108
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

find .github -name "update-doc-db.yml" -type f

Repository: RustPython/RustPython

Length of output: 100

🏁 Script executed:

cat -n .github/workflows/update-doc-db.yml | head -120

Repository: RustPython/RustPython

Length of output: 4365

🏁 Script executed:

ls -la .github/workflows/ | grep -i update

Repository: RustPython/RustPython

Length of output: 132

🏁 Script executed:

find . -name "*.yml" -o -name "*.yaml" | grep -i update

Repository: RustPython/RustPython

Length of output: 102

🏁 Script executed:

git ls-files .github/workflows/ | head -20

Repository: RustPython/RustPython

Length of output: 264

🏁 Script executed:

ls -la .github/ 2>/dev/null || echo "Checking git tracked files"; git ls-files .github/ | head -20

Repository: RustPython/RustPython

Length of output: 887

🏁 Script executed:

git ls-files | grep -E "\.github.*\.(yml|yaml)$" | sort

Repository: RustPython/RustPython

Length of output: 287

🏁 Script executed:

rg -l "Commit and push (non-main branches only)" --type yaml

Repository: RustPython/RustPython

Length of output: 47

🏁 Script executed:

rg "AUTO_COMMIT_PAT" --type yaml

Repository: RustPython/RustPython

Length of output: 47

🏁 Script executed:

rg "Update doc DB for CPython" --type yaml

Repository: RustPython/RustPython

Length of output: 47

🌐 Web query:

GitHub Actions sparse checkout git operations authentication

💡 Result:

Summary + quick examples

  1. Use actions/checkout sparse options to do sparse-checkout in a workflow:
  • inputs: sparse-checkout (newline-separated patterns), sparse-checkout-cone-mode (true/false), fetch-depth, filter (partial clone).
    Example:
    • uses: actions/checkout@v6
      with:
      fetch-depth: 0
      sparse-checkout: |
      .github
      src
      sparse-checkout-cone-mode: true
      (cite: actions/checkout). [1]
  1. Authenticated git operations in the job
  • By default actions/checkout persists an auth token into local git config so subsequent git commands (fetch, push) run authenticated. That token is the job-scoped GITHUB_TOKEN. Use persist-credentials: true to keep it (default). To push from a workflow you also need appropriate token permissions (contents: write) — set repository permissions in the workflow or use a PAT (secret) if you need broader scopes or create new workflow runs. GITHUB_TOKEN expires at job end and actions-triggered events from it don’t re-run workflows. [2]

Example push with GITHUB_TOKEN:
permissions:
contents: write
jobs:
push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
persist-credentials: true
fetch-depth: 0
sparse-checkout: |
README.md
- run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
# make changes...
git add .
git commit -m "Update"
git push origin HEAD:main

If you need to use a PAT (for cross-repo write, finer scopes, or to trigger new workflows), store it as a secret and set actions/checkout.token or configure git remote HTTPS with that PAT. [1][2]

  1. Caveats & troubleshooting
  • Sparse/partial-clone behavior depends on Git and runner versions; some git versions and combinations of partial-clone/filter vs sparse-checkout have known issues (see issues where sparse may fall back to full checkout). If you see all files fetched, try changing cone-mode, fetch-depth, or use full fetch-depth: 0 and explicit git sparse-checkout commands on the runner. Test runner git version and update if needed. [1][3]

References
[1] actions/checkout README (sparse-checkout inputs & usage).
[2] GitHub Docs — GITHUB_TOKEN and using it for authentication in workflows.
[3] actions/checkout GitHub issues (sparse-checkout / git-version caveats).

🏁 Script executed:

git ls-files .github/ 2>/dev/null | head -50

Repository: RustPython/RustPython

Length of output: 509

🏁 Script executed:

test -d .github && find .github -type f | head -30 || echo ".github directory not found in current repo"

Repository: RustPython/RustPython

Length of output: 509

Ensure the AUTO_COMMIT_PAT secret is configured in repository settings with write access.

The merge job requires the AUTO_COMMIT_PAT secret to authenticate git push operations. Configure this secret in your repository settings with appropriate permissions to push commits to non-main branches. The workflow correctly uses the full repository checkout (without sparse checkout) for the merge and commit step, so git operations will work as expected.

🤖 Prompt for AI Agents 
.github/workflows/update-doc-db.yml lines 99-108: the workflow assumes a
push-able token but doesn't ensure one is provided; configure the
AUTO_COMMIT_PAT secret in the repository settings with repo write permissions
and update the workflow to use it when pushing (for example export a GIT_TOKEN
env from secrets.AUTO_COMMIT_PAT and run git remote set-url origin
https://x-access-token:${GIT_TOKEN}@github.com/${{ github.repository }}.git
before git push); also ensure the workflow has repository write permissions
enabled so the push succeeds.

11 changes: 10 additions & 1 deletion crates/doc/generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,15 @@ def find_doc_entries() -> "Iterable[DocEntry]":
type(str().__iter__()),
type(tuple().__iter__()),
]

# Add types from the types module (e.g., ModuleType, FunctionType, etc.)
for name in dir(types):
if name.startswith("_"):
continue
obj = getattr(types, name)
if isinstance(obj, type):
builtin_types.append(obj)

for typ in builtin_types:
parts = ("builtins", typ.__name__)
yield DocEntry(parts, pydoc._getowndoc(typ))
Expand All @@ -204,7 +213,7 @@ def main():
docs = {
entry.key: entry.doc
for entry in find_doc_entries()
if entry.raw_doc is not None
if entry.raw_doc is not None and isinstance(entry.raw_doc, str)
}
dumped = json.dumps(docs, sort_keys=True, indent=4)
OUTPUT_FILE.write_text(dumped)
Expand Down
Loading
Loading