feat: Implement host-level telemetry batching to reduce rate limiting #718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

samikshya-db merged 19 commits into main from fix-telemetry-host-level-batching

Dec 3, 2025

Contributor

samikshya-db commented Dec 1, 2025 •

edited

Loading

There were 2 gaps in telemetry python before:

Per host batching was not present, which caused our pre-release steps to fail [Link]
log levels could be misleading : all telemetry log levels should be as low as possible.

Key changes:

Add _TelemetryClientHolder with reference counting for shared clients
Change TelemetryClientFactory to key clients by host_url instead of session_id
Add getHostUrlSafely() helper for defensive null handling
Update all callers (client.py, exc.py, latency_logger.py) to pass host_url

Before: 100 connections to same host = 100 separate TelemetryClients
After: 100 connections to same host = 1 shared TelemetryClient (refcount=100)

This fixes rate limiting issues seen in e2e tests where 300+ parallel connections were overwhelming the telemetry endpoint with 429 errors.

What type of PR is this?

Refactor
Feature
Bug Fix
Other

Description

How is this tested?

Unit tests
E2E Tests
Manually
N/A

Related Tickets & Documents


          feat: Implement host-level telemetry batching to reduce rate limiting

3bfbac1

Changes telemetry client architecture from per-session to per-host batching,
matching the JDBC driver implementation. This reduces the number of HTTP
requests to the telemetry endpoint and prevents rate limiting in test
environments.

Key changes:
- Add _TelemetryClientHolder with reference counting for shared clients
- Change TelemetryClientFactory to key clients by host_url instead of session_id
- Add getHostUrlSafely() helper for defensive null handling
- Update all callers (client.py, exc.py, latency_logger.py) to pass host_url

Before: 100 connections to same host = 100 separate TelemetryClients
After:  100 connections to same host = 1 shared TelemetryClient (refcount=100)

This fixes rate limiting issues seen in e2e tests where 300+ parallel
connections were overwhelming the telemetry endpoint with 429 errors.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:26

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:26

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          chore: Change all telemetry logging to DEBUG level

5e29089

Reduces log noise by changing all telemetry-related log statements
(info, warning, error) to debug level. Telemetry operations are
background tasks and should not clutter logs with operational messages.

Changes:
- Circuit breaker state changes: info/warning -> debug
- Telemetry send failures: error -> debug
- All telemetry operations now consistently use debug level

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:27

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:27

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          chore: Fix remaining telemetry warning log to debug

26b67c7

Changes remaining logger.warning in telemetry_push_client.py to debug level
for consistency with other telemetry logging.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:27

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:27

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          fix: Update tests to use host_url instead of session_id_hex

da13b9e

- Update circuit breaker test to check logger.debug instead of logger.info
- Replace all session_id_hex test parameters with host_url
- Apply Black formatting to exc.py and telemetry_client.py

This fixes test failures caused by the signature change from session_id_hex
to host_url in the Error class and TelemetryClientFactory.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:41

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:41

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          fix: Revert session_id_hex in tests for functions that still use it

ffc7b27

Only Error classes changed from session_id_hex to host_url.
Other classes (TelemetryClient, ResultSetDownloadHandler, etc.) still use session_id_hex.

Reverted:
- test_telemetry.py: TelemetryClient and initialize_telemetry_client
- test_downloader.py: ResultSetDownloadHandler
- test_download_manager.py: ResultFileDownloadManager

Kept as host_url:
- test_client.py: Error class instantiation

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:48

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 19:48

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          fix: Update all Error raises and test calls to use host_url

b11a461

Changes:
1. client.py: Changed all error raises from session_id_hex to host_url
   - Connection class: session_id_hex=self.get_session_id_hex() -> host_url=self.session.host
   - Cursor class: session_id_hex=self.connection.get_session_id_hex() -> host_url=self.connection.session.host

2. test_telemetry.py: Updated get_telemetry_client() and close() calls
   - get_telemetry_client(session_id) -> get_telemetry_client(host_url)
   - close(session_id) -> close(host_url=host_url)

3. test_telemetry_push_client.py: Changed logger.warning to logger.debug
   - Updated test assertion to match debug logging level

These changes complete the migration from session-level to host-level
telemetry client management.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 20:04

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 20:04

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          fix: Update thrift_backend.py to use host_url instead of session_id_hex

60e50de

Changes:
1. Added self._host attribute to store server_hostname
2. Updated all error raises to use host_url=self._host
3. Changed method signatures from session_id_hex to host_url:
   - _check_response_for_error
   - _hive_schema_to_arrow_schema
   - _col_to_description
   - _hive_schema_to_description
   - _check_direct_results_for_error
4. Updated all method calls to pass self._host instead of self._session_id_hex

This completes the migration from session-level to host-level error reporting.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 20:20

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 1, 2025 20:20

— with

GitHub Actions Failure

github-actions bot commented Dec 1, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Fix Black formatting by adjusting fmt directive placement

01ea1e1

Moved the `# fmt: on` directive to the except block level instead
of inside the if statement to resolve Black parsing confusion.

samikshya-db had a problem deploying to azure-prod

December 1, 2025 20:35

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:21

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Fix Black formatting in telemetry_client.py

8cb66ec

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:26

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:26

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Use 'test-host' instead of 'test' for mock host in telemetry tests

69789ee

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:39

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:39

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Replace test-session-id with test-host in test_client.py

b0aa889

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:43

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:43

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Fix telemetry client lookup to use test-host in tests

c8cfc23

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:47

— with

GitHub Actions Failure

samikshya-db had a problem deploying to azure-prod

December 2, 2025 18:47

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).


          Make session_id_hex keyword-only parameter in Error.__init__

962def5

samikshya-db temporarily deployed to azure-prod

December 2, 2025 19:45

— with

GitHub Actions Inactive

samikshya-db had a problem deploying to azure-prod

December 2, 2025 19:45

— with

GitHub Actions Failure

github-actions bot commented Dec 2, 2025

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

nikhilsuri-db reviewed

View reviewed changes

Contributor

nikhilsuri-db left a comment

LGTM

please manually run Daily Telemetry E2E Tests
Can you also fix [daily-telemetry-e2e.yml ] to run daily not evey sunday [ seems gap there ]

samikshya-db temporarily deployed to azure-prod

December 3, 2025 05:46

— with

GitHub Actions Inactive

Contributor Author

samikshya-db commented Dec 3, 2025

Thanks for your review @nikhilsuri-db

Ran it now, https://github.com/databricks/databricks-sql-python/actions/runs/19883810489 : the run passes.
I'll fix it in a later PR if that is ok?

samikshya-db had a problem deploying to azure-prod

December 3, 2025 06:55

— with

GitHub Actions Failure

samikshya-db marked this pull request as ready for review

December 3, 2025 07:03

nikhilsuri-db approved these changes

View reviewed changes

samikshya-db merged commit ebe4b07 into main

29 of 32 checks passed

samikshya-db deleted the fix-telemetry-host-level-batching branch

December 3, 2025 15:09

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet