12/09-12/13

Timeline

Total runtime on low-resolution grid was 16h across 8 nodes. Total runtime data available here.

Currently limited by queue time (usually 12+ hours), and I cannot get the ssh key working correctly.

Current method to do high-res runs is to use the development queue while waiting by chaining jobs, e.g.,

• job1 runs model1 for 2 orbits then dumps and requests job2,
• job2 runs model1 for 2 additional orbits then dumps and requests job3,
• job3 runs model1 for the final 2 orbits, deletes dump files, moves all output to model1 directory, and then requests job4
• job4 runs model2 for 2 orbits then dumps and requests job5...

Files are then mass-transferred

Detailed timeline coming soon

• Goal 1: Get everything working by 12/15 (includes job / transfer scripts)
• Goal 2: Finish Inner grid by 12/22 (will likely increase processor count to 2048 on 32 nodes)
• Goal 3: Finish Outer grid by 12/27

NOTE: Stampede says units remaining is: TG-AST090085 29485 2019-12-31

Low Resolution (Inner Grid)

High Resolution (Inner Grid)

HR grid is running at 384x128x384 on 16x32x2 processors (16 nodes).

HR scales almost exactly as 2^4, requiring 128 hrs across 16 nodes (2048 hrs total).

High Resolution (Outer Grid)

This is suddenly running into an infinite dt on the first time-step. Investigating (should not take long to fix)

Supernova

Overleaf