Skip to content

MHD & RMHD #809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 42 commits into from
Apr 5, 2025
Merged

MHD & RMHD #809

merged 42 commits into from
Apr 5, 2025

Conversation

ChrisZYJ
Copy link
Contributor

Description

  • MHD
  • HLLD Riemann Solver for MHD
  • Powell's 8-wave method for MHD
  • Relativistic MHD (RMHD)

Type of change

  • New feature (non-breaking change which adds functionality)

Scope

  • This PR comprises a set of related changes with a common goal

How Has This Been Tested?

  • HLL vs HLLD Dai and Woodward test (examples/1D_dai_woodward and examples/1D_dai_woodward_hlld)
    Miyoshi, T. & Kusano, K. (2005). A multi-state HLL approximate Riemann solver for ideal magnetohydrodynamics.

HLLvsHLLD_Dai_Woodward_test_Miyoshi_paper

  • Orszag-Tang vortex (examples/2D_orszag_tang)
    Orszag, S. A. & Tang, C.-M. (1979). Small-scale structure of two-dimensional magnetohydrodynamic turbulence.
orszag_tang_1k_t.0.66.mp4
  • RMHD Brio-Wu (examples/1D_brio_wu_rmhd)
    Mignone, A. & Bodo, G. (2006). An HLLC Riemann solver for relativistic flows – II. Magnetohydrodynamics.

RMHD_brio_wu_1 6k

  • RMHD Shock Cloud (examples/2D_shock_cloud_rmhd)
    Mignone, A. & Bodo, G. (2006). An HLLC Riemann solver for relativistic flows – II. Magnetohydrodynamics.
shock_cloud_rmhd.mp4

Performance

  • NCSA Delta A40

compare_MHD_speed

Checklist

  • I have added comments for the new code
  • I added Doxygen docstrings to the new code
  • I have made corresponding changes to the documentation (docs/)
  • I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
  • I have added example cases in examples/ that demonstrate my new feature performing as expected.
    They run to completion and demonstrate "interesting physics"
  • I ran ./mfc.sh format before committing my code
  • New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
  • This PR does not introduce any repeated code (it follows the DRY principle)
  • I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

  • Checked that the code compiles using NVHPC compilers
  • Checked that the code compiles using CRAY compilers
  • Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
  • Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
  • Enclosed the new feature via nvtx ranges so that they can be identified in profiles
  • Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
  • Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
  • Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

@ChrisZYJ ChrisZYJ requested review from a team and sbryngelson as code owners March 27, 2025 05:30
sbryngelson
sbryngelson reviewed Mar 30, 2025
View reviewed changes
sbryngelson
sbryngelson reviewed Mar 30, 2025
View reviewed changes
sbryngelson
sbryngelson reviewed Mar 30, 2025
View reviewed changes
sbryngelson
sbryngelson requested changes Mar 30, 2025
View reviewed changes
Copy link
Member

@sbryngelson sbryngelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i left some requests for changes. overall very very good! as usual. exciting new stuff. can you update the README for the new capability and, if you have it, maybe we can even put an image in the readme as well.

@sbryngelson
Copy link
Member

@ChrisZYJ, I realized I've saddled you with a somewhat thankless task -- I hope it isn't so tough. I appreciate it very much, regardless. You can also look at the benchmarking logs to see how any of your code changes affected the --case-optimization feature.

@sbryngelson
Copy link
Member

FYI the Phoenix runners are down right now -- hopefully back up soon.

@ChrisZYJ
Copy link
Contributor Author

@sbryngelson Thanks for the info! I think all tests should pass. I've updated my new features with the requested changes while keeping the code simple and readable:

  • Removed unicode characters
  • Grouped %L and %R (also cm(i)%L) using types
  • Replaced Bxb, Bxe with B_idx%beg, B_idx%end

Apparently, removing Bxb, Bxe doesn't break AMD compilers. Points 2 and 3 could probably be applied throughout the code, but due to the the extent of changes, maybe it's best to do this in a separate PR so if anything goes wrong, we can pinpoint the error more easily.

@sbryngelson
Copy link
Member

@sbryngelson Thanks for the info! I think all tests should pass. I've updated my new features with the requested changes while keeping the code simple and readable:

  • Removed unicode characters
  • Grouped %L and %R (also cm(i)%L) using types
  • Replaced Bxb, Bxe with B_idx%beg, B_idx%end

Apparently, removing Bxb, Bxe doesn't break AMD compilers. Points 2 and 3 could probably be applied throughout the code, but due to the the extent of changes, maybe it's best to do this in a separate PR so if anything goes wrong, we can pinpoint the error more easily.

@ChrisZYJ I agree, we should separate concerns.

This was referenced Apr 3, 2025
Merged
Merged
Merged
@sbryngelson
Copy link
Member

@ChrisZYJ This looks good but I want to wait until Phoenix is back online for those runners (a day or two, probably). Also I think you may have a case optimization problem? I can't remember why the Phoenix benchmark CI failed but if you download the logs you can check.

@ChrisZYJ
Copy link
Contributor Author

ChrisZYJ commented Apr 4, 2025 •
edited
Loading

@sbryngelson You are right! Sorry that I kept thinking it was the runner's problem - really didn't expect it to work only on GPU without case-opt. In case anyone runs into this in the future: var(i)%field is not allowed for !$acc private(...) (only fails on GPU when case-optimization is enabled), but var%field(i) is fine.

sbryngelson
sbryngelson reviewed Apr 4, 2025
View reviewed changes
sbryngelson
sbryngelson reviewed Apr 4, 2025
View reviewed changes
sbryngelson
sbryngelson reviewed Apr 4, 2025
View reviewed changes
sbryngelson
sbryngelson reviewed Apr 4, 2025
View reviewed changes
@sbryngelson sbryngelson merged commit 4298db0 into MFlowCode:master Apr 5, 2025
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wilfonba wilfonba wilfonba left review comments

@sbryngelson sbryngelson sbryngelson approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ChrisZYJ @sbryngelson @wilfonba