-
-
Notifications
You must be signed in to change notification settings - Fork 32.5k
gh-64192: Make imap()
/imap_unordered()
in multiprocessing.pool
actually lazy
#136871
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
obaltian
wants to merge
21
commits into
python:main
Choose a base branch
from
obaltian:feature/add-buffersize-to-multiprocessing
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+334
−76
Open
Changes from all commits
Commits
Show all changes
21 commits
Select commit
Hold shift + click to select a range
d8e8a02
draft: impl lazy input consumption in mp.Pool.imap(_unordered)
obaltian f6f423c
Use semaphore to synchronize threads
obaltian 937862d
Update buffersize behavior to match concurrent.futures.Executor behavior
obaltian b6f6caa
Release all `buffersize_lock` obj from the parent thread when terminate
obaltian 3bafd5d
Add 2 basic `ThreadPool.imap()` tests w/ and w/o buffersize
obaltian e43232b
Fix accidental swap in imports
obaltian dd416e0
clear Pool._taskqueue_buffersize_semaphores safely
obaltian 99f5a8c
Slightly optimize Pool._taskqueue_buffersize_semaphores terminate
obaltian 2a53398
Rename `Pool.imap()` buffersize-related tests
obaltian f8878eb
Fix typo in `IMapIterator.__init__()`
obaltian 2ca51e3
Add tests for buffersize combinations with other kwargs
obaltian bf27d5d
Remove if-branch in `_terminate_pool`
obaltian 508c765
Add more edge-case tests for `imap` and `imap_unodered`
obaltian dff1167
Split inf iterable test for `imap` and `imap_unordered`
obaltian 94cc0b9
Add doc for `buffersize` argument of `imap` and `imap_unordered`
obaltian 816fb6c
add *versionadded* for `imap_unordered`
obaltian 88cc10a
Remove ambiguity in `buffersize` description.
obaltian 05e3b24
Set *versionadded* as next in docs
obaltian f1c25b3
Add whatsnew entry
obaltian 199eb42
Fix aggreed comments on code formatting/minor refactoring
obaltian d1dee31
Remove `imap` and `imap_unordered` body code duplication
obaltian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't it this method the same as
_imap
? Is there a reason why we duplicated them since onlyIMapIterator
vsIMapUnorderedIterator
is changed. I don't know if it's a good target for refactoring where we would pass the wrapper:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are almost the same, indeed. Sure, I will make implementation shared, thanks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually don't know if it's something that was thought to be duplicated. But it appears that it is. If @gpshead thinks it's better to duplicate the code for future improvements, then we can keep it as is (because if in the future only one of the two functions has issues or a different behavior, it's better to have two separate functions)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another argument to reuse code is the fact that
_map_async
is also shared here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have added the shared method in a separate commit, can revert it if it's unwanted.