-
Notifications
You must be signed in to change notification settings - Fork 560
[v2] Randomize test order for unit tests #3434
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Global tests objects are harmful, in so far as any attempt to modify them can result in failures further down the food chain. Don't use them and instead generate new objects on the fly. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit 610f545)
The fake service clients rely on the fake server being initialised, even if we don't actually use it. Add the missing calls, renaming some tests in the process. PS: You can reproduce this without the '-shuffle on' by simply deleting any of the test cases preceding those that we are modifying here and running the tests. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit 47b361b)
This is a mildly backwards-incompatible change, but since most people only use this API on the level of pagination.Pager, it ought not be a big deal in practice. Closes gophercloud#3222. (cherry picked from commit 97c0a32)
Ahead of some large-scale rework here. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit 1476dae)
All aliases were discovered with: ag '[^\t]"github.com/gophercloud/gophercloud/v2/testhelper/client"' They were then replaced with judicious use of sed and ag. Starting with the most common alias, 'fake', first replace the alias: sed -i 's/\<fake\>/client/' $(ag 'fake "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l) This was repeated for the other aliases, 'fakeclient' and 'thclient'. We then dropped the now unnecessary aliasing: sed -i 's/\<client\> "/"/' $(ag 'client "github.com/gophercloud/gophercloud/v2/testhelper/client"' -l) Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit beeed5b)
Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit 2f52485)
The meat of this changes is in 'testhelper/http_responses.go' and 'testhelper/client/fake.go'. The rest of the change is bulk generated updates to handle those changes: Create a "FakeServer" instance instead of configuring the global fake: sed -i 's/th.SetupHTTP/fakeServer := th.SetupHTTP/g' $(ag 'th.SetupHTTP' -l) Tear down this instance on test completion: sed -i 's/defer th.TeardownHTTP/defer fakeServer.Teardown/g' $(ag 'defer th.TeardownHTTP' -l) Update calls to the global 'Mux' instance with calls to the FakeServer version: sed -i 's/th.Mux.HandleFunc/fakeServer.Mux.HandleFunc/g' $(ag 'th.Mux.HandleFunc' -l) Update calls to ServiceClient to include a 'FakeServer' parameter: sed -i 's/client.ServiceClient()/client.ServiceClient(fakeServer)/' $(ag -w client.ServiceClient -l) Plus alises for the same: sed -i 's/fake.ServiceClient()/fake.ServiceClient(fakeServer)/' $(ag 'fake.ServiceClient' -l) Update definitions and calls to various MockFoo and HandleFoo helpers - plus lowercase, private variants of same - to include a 'FakeServer' parameter: sed -i 's/\(func Mock\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func Mock\w+\(t \*testing.T' -l) sed -i 's/\(\tMock\w\+(t\)/\1, fakeServer/' $(ag '\tMock\w+\(t' -l) sed -i 's/\(func mock\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func mock\w+\(t \*testing.T' -l) sed -i 's/\(\tmock\w\+(t\)/\1, fakeServer/' $(ag '\tmock\w+\(t' -l) sed -i 's/\(func Handle\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func Handle\w+\(t \*testing.T' -l) sed -i 's/\(\tHandle\w\+(t\)/\1, fakeServer/' $(ag '\tHandle\w+\(t' -l) sed -i 's/\(func handle\w\+(t \*testing.T\)/\1, fakeServer th.FakeServer/' $(ag 'func handle\w+\(t \*testing.T' -l) sed -i 's/\(\thandle\w\+(t\)/\1, fakeServer/' $(ag '\thandle\w+\(t' -l) Finally, there were some manual fixes to handle stuff that the regexes missed. Conflicts: openstack/baremetal/v1/portgroups/testing/fixtures.go openstack/baremetal/v1/portgroups/testing/requests_test.go openstack/compute/v2/servers/testing/requests_test.go openstack/db/v1/quotas/testing/fixtures_test.go openstack/db/v1/quotas/testing/requests_test.go openstack/dns/v2/zones/testing/requests_test.go openstack/loadbalancer/v2/flavors/testing/fixtures.go openstack/loadbalancer/v2/flavors/testing/requests_test.go openstack/networking/v2/extensions/security/rules/testing/requests_test.go Changes: openstack/blockstorage/v3/manageablevolumes/testing/fixtures_test.go openstack/blockstorage/v3/manageablevolumes/testing/requests_test.go openstack/blockstorage/v3/volumes/testing/fixtures_test.go openstack/blockstorage/v3/volumes/testing/requests_test.go openstack/networking/v2/extensions/security/addressgroups/testing/requests_test.go NOTE(stephenfin): Conflicts are generally due to tests or test files that don't exist on the v2 branch. Changes are due to tests that have been removed or moved on the main branch. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit eed41cd)
I don't know why this isn't the default behavior, but relying on test order is a bad idea (TM) that we should do our best to avoid. Signed-off-by: Stephen Finucane <stephenfin@redhat.com> (cherry picked from commit 9251e99)
idk if we want to include fcf44f4 here. It's technically a breaking change but it's also a bugfix. Input welcome. |
@stephenfin I don't see anything breaking there. |
From the
|
I've never seen that someone used direct access to |
In that case, I think we're safe to merge this. |
This is manual backport of #3108, with trivial conflicts on the penultimate patch addressed.
This is not critical, but backporting it will ease future backports from the
main
branch.