...
The gmon_shallow.* files can be combined to a summary file with: gmon -s shallow gmon_shallow* . You can also rename a gmon_shallow.NNNNN file to gmon.out to compare the gprof data for one rank with the gmon.sum summary (try to avoid the first PID which is rank zero because it's not as interesting as the others ) .
gcov is not supported at -O3 opt. level
Interesting areas that seem to recur in the timing profile are:
...