Difference between revisions of "(K)TAU"

From ZeptoOS
Jump to navigationJump to search
 
(87 intermediate revisions by one other user not shown)
Line 1: Line 1:
−
[[ZeptoOS_Documentation|Top]]
+
[[ZOID]] | [[ZeptoOS_Documentation|Top]] | [[Other Packages]]
 
----
 
----
  
−
=THIS SECTION UNDER CONSTRUCTION, PLEASE RETURN LATER=
+
Both [http://tau.uoregon.edu TAU] (Tuning and Analysis Utilities) and its Linux kernel counterpart, [http://www.cs.uoregon.edu/research/ktau/home.php KTAU] have been ported to ZeptoOS. In addition the build system of the ZeptoOS kernel has been modified to simplify the integration of KTAU patches. The following sections describe how to acquire, configure, build and use TAU and KTAU under ZeptoOS, with Surveyor at ANL as the test platform.
  
−
Both TAU (Tuning and Analysis Utilities) and its Linux kernel counterpart, KTAU have been ported to ZeptoOS. In addition the build system of the ZeptoOS kernel has been modified to simplify the integration of KTAU patches. The following sections describe how to acquire, configure, build and use TAU and KTAU under ZeptoOS, with Surveyor at ANL as the test platform. The KTAU section appears first as the TAU section has some dependencies on that information.
+
 
 +
==TAU==
 +
 
 +
===Acquiring TAU===
 +
You can download the current (as of 2009-04-28) supported version [http://www.cs.uoregon.edu/research/ktau/tarballs/tau-2.18.1p2-zepto.tar.gz 2.18.1p2-zepto] of [http://tau.uoregon.edu TAU]. All future versions (greater than 2.18.1) should support ZeptoOS and will be available from the usual [http://www.cs.uoregon.edu/research/tau/downloads.php TAU downloads page].
 +
<BR>
 +
<BR>
 +
===Configuration under ZeptoOS on BG/P===
 +
Before proceeding we assume that ZeptoOS has been downloaded, configured and installed. For the purpose of this how-to, the ZeptoOS install location is assumed to be "/home/anataraj/ZeptoReleases/install-BGP". You may have a different location or may opt to use the default ZeptoOS install location on your machine. Installing your own ZeptoOS can done by issuing 'python install.py /path/to/install' at the root-directory of the ZeptoOS release (but please refer to the detailed ZeptoOS documentation).
 +
 
 +
TAU needs to be told where the ZeptoOS install location is using the '-zeptodir' configure option.
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2> pwd
 +
/home/anataraj/tau-2.18.1p2
 +
 
 +
[email protected]:~/tau-2.18.1p2> ./configure -fullhelp | grep Zepto
 +
-zeptodir=<dir> ............................. ZeptoOS install location.
 +
</pre>
 +
 
 +
A simple example configuration is shown below that includes the -mpi option. It is important to note that the TAU configuration under ZeptoOS on BGP is different from the standard TAU configuration on BGP when using the default IBM OS suite.
 +
 
 +
In this case the architecture needs to be specified as 'ppc' in addition to the ZeptoOS install location.
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2> ./configure -arch=ppc \
 +
                                                    -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
 +
                                                    -mpi
 +
</pre>
 +
<BR>
 +
<BR>
 +
===Configuring with PDT under ZeptoOS on BGP===
 +
The [http://www.cs.uoregon.edu/research/pdt/home.php Program Database Toolkit(PDT)] provides TAU with automatic source instrumentation capability. This allows routines, particularly in large projects, to be instrumented without tedious manual intervention.
 +
 
 +
First download and untar PDT. Then configure and build it:
 +
<pre>
 +
[email protected]:~/pdtoolkit-3.14> pwd
 +
/home/anataraj/pdtoolkit-3.14
 +
 
 +
[email protected]:~/pdtoolkit-3.14> ./configure
 +
 
 +
[email protected]:~/pdtoolkit-3.14> make
 +
 
 +
[email protected]:~/pdtoolkit-3.14> make install
 +
</pre>
 +
 
 +
Next configure TAU with the above PDT:
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2> ./configure  -arch=ppc \
 +
                                                      -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
 +
                                                      -mpi \
 +
                                                      -pdt=/home/anataraj/pdtoolkit-3.14 \
 +
                                                      -pdtarchdir=ppc64 -pdt_c++=xlC
 +
</pre>
 +
Please pay close attention to the '-pdtarchdir=' and the '-pdt_c++=' options as they are important to it working on the ZeptoOS/BGP platform.
 +
 +
Please refer to the [http://tau.uoregon.edu TAU] documentation for various other TAU configuration options that may be used.
 +
<BR>
 +
<BR>
 +
===Building TAU===
 +
Lastly, build TAU in-place by issuing a 'make install'.
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2> make install
 +
</pre>
 +
This should provide a TAU built for ZeptoOS with MPI profiling and automatic instrumentation capabilities.
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
  
 
==KTAU==
 
==KTAU==
  
 
===Acquiring KTAU===
 
===Acquiring KTAU===
−
You can download a gzipped tarball of KTAU (version: 1.7.15-bgpzepto) from http://www.cs.uoregon.edu/research/ktau/downloads.php .
+
You can download the latest version (1.7.15-bgpzepto-p1) of KTAU available as a gzipped tarball from the [http://www.cs.uoregon.edu/research/ktau/downloads.php KTAU downloads page].
−
 
+
<BR>
 +
<BR>
 
===Integrating KTAU into ZeptoOS===
 
===Integrating KTAU into ZeptoOS===
 
For the below examples showing the configuration and build steps, it is assumed that the KTAU tarball has been expanded into the location /home/anataraj/ktau-1.7.15-bgpzepto and that the ZeptoOS tarball has been expanded into /home/anataraj/BGP.
 
For the below examples showing the configuration and build steps, it is assumed that the KTAU tarball has been expanded into the location /home/anataraj/ktau-1.7.15-bgpzepto and that the ZeptoOS tarball has been expanded into /home/anataraj/BGP.
−
 
+
<BR>
−
====Configure ZeptoOS to point to KTAU path and patch====
+
====Configure ZeptoOS to point to KTAU patch and path====
 
Begin by configuring the ZeptoOS by pointing it to the correct KTAU path and the name of the patch to be used.
 
Begin by configuring the ZeptoOS by pointing it to the correct KTAU path and the name of the patch to be used.
 
<pre>
 
<pre>
 
[email protected]:~/BGP> ./configure --edit
 
[email protected]:~/BGP> ./configure --edit
 
</pre>
 
</pre>
 +
[[Image: KTAU_configure-edit.jpg|200px]]
  
−
Once this is done, the changes will be saved to the Make.rules file as can be checked.
+
Once this is done, the changes will be saved to the Make.rules file.
 
<pre>
 
<pre>
 
[email protected]:~/BGP> cat Make.rules
 
[email protected]:~/BGP> cat Make.rules
Line 42: Line 114:
 
KTAU_PATCH="patch-2.6.19.2-ktau-1.7.15-bgpzepto"
 
KTAU_PATCH="patch-2.6.19.2-ktau-1.7.15-bgpzepto"
 
</pre>
 
</pre>
−
 
+
<BR>
 
====Patching ZeptoOS with KTAU====
 
====Patching ZeptoOS with KTAU====
−
Step down into the "kernel" directory of ZeptoOS next. Issuing a 'make help' will show all the targets, including the KTAU specific ones.
+
Step down into the "kernel" directory of ZeptoOS next. Issuing a 'make help' will show all the targets, including the KTAU specific ones (i.e. the last two targets in the list).
 
<pre>
 
<pre>
 
[email protected]:~/BGP> cd kernel/
 
[email protected]:~/BGP> cd kernel/
Line 76: Line 148:
 
</pre>
 
</pre>
  
−
Issuing 'make bgp-cn-linux-add-ktau' target patches the ZeptoOS compute node Linux kernel with KTAU the configured patch and then links to headers and sources from the KTAU path.
+
Issuing 'make bgp-cn-linux-add-ktau' patches the ZeptoOS compute node Linux kernel with the configured KTAU patch and then links to headers and sources from the KTAU path.
  
 
<pre>
 
<pre>
Line 130: Line 202:
 
  After that, please follow the standard ZeptoOS build steps (e.g. "make bgp-cn-linux-build").
 
  After that, please follow the standard ZeptoOS build steps (e.g. "make bgp-cn-linux-build").
 
</pre>
 
</pre>
−
 
+
<BR>
 
====Configuring KTAU parameters of the patched ZeptoOS CN Linux====
 
====Configuring KTAU parameters of the patched ZeptoOS CN Linux====
 
Next the KTAU parameters of the kernel configuration need to be set. This can be done by issuing a 'make bgp-cn-linux-menuconfig'.
 
Next the KTAU parameters of the kernel configuration need to be set. This can be done by issuing a 'make bgp-cn-linux-menuconfig'.
Line 136: Line 208:
 
[email protected]:~/BGP/kernel> make bgp-cn-linux-menuconfig  
 
[email protected]:~/BGP/kernel> make bgp-cn-linux-menuconfig  
 
</pre>
 
</pre>
−
 
+
[[Image:KTAU_menuconfig.jpg|200px]]
−
 
+
<BR>
 +
<BR>
 
===Building ZeptoOS with KTAU===
 
===Building ZeptoOS with KTAU===
−
 
 
After configuration is performed and saved, the kernel needs to be built.
 
After configuration is performed and saved, the kernel needs to be built.
 
<pre>
 
<pre>
Line 175: Line 247:
  
 
You should now have a KTAU-patched and configured ZeptoOS Compute Node Linux kernel ready for use.
 
You should now have a KTAU-patched and configured ZeptoOS Compute Node Linux kernel ready for use.
 +
<BR>
 +
<BR>
 +
===Building KTAU User-level Utilities===
 +
Next please build the user-level utils of KTAU, as follows:
 +
<pre>
 +
[email protected]:~/ktau-1.7.15-bgpzepto> pwd
 +
/home/anataraj/ktau-1.7.15-bgpzepto
 +
 +
[email protected]:~/ktau-1.7.15-bgpzepto> cd user-src/
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src> cd src/
  
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> make \
 +
                          ARCH=ppc \
 +
                          BGP_KERNEL_INC=/home/anataraj/ZeptoReleases/BGP/kernel/work/linux-2.6.19.2-BGP-V1R3/include/ \
 +
                          clean all
 +
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> cd runktau/
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src/runktau> ls -alhtr timeKtau
 +
-rwxr-xr-x 1 anataraj users 6.0M 2009-04-28 13:09 timeKtau
 +
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> cd ..
 +
[email protected]:~/ktau-1.7.15-bgpzepto/user-src> ls lib/
 +
libktau.a
 +
</pre>
 +
<BR>
 +
<BR>
 
===Removing KTAU from ZeptoOS===
 
===Removing KTAU from ZeptoOS===
 
If you wish to revert back to the original version of ZeptoOS without KTAU integration, please perform the below steps.
 
If you wish to revert back to the original version of ZeptoOS without KTAU integration, please perform the below steps.
Line 212: Line 309:
 
</pre>
 
</pre>
 
Please remember to reconfigure and rebuild ZeptoOS after undoing the KTAU integration.
 
Please remember to reconfigure and rebuild ZeptoOS after undoing the KTAU integration.
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
 +
<BR>
  
 +
==Using TAU with KTAU : Integrated Profiles==
 +
TAU can be used in a tightly coupled fashion with KTAU to provide integrated profiles that show both user and kernel performance data under certain configurations. The following subsection describes such a configuration.
 +
<BR>
 +
<BR>
 +
===Configuring under ZeptoOS===
 +
Use all the previously described configuration options, but in addition add the -MULTIPLECOUNTERS and -ktau* options to the configure line. As follows:
 +
<pre>
 +
./configure  -arch=ppc -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
 +
            -mpi \
 +
            -pdt=/home/anataraj/pdtoolkit-3.14 -pdtarchdir=ppc64 -pdt_c++=xlC \
 +
            -MULTIPLECOUNTERS \
 +
            -ktau -ktau_shctr \
 +
            -ktauinc=/home/anataraj/BGP/kernel/work/linux-2.6.19.2-BGP-V1R3/include \
 +
            -ktauincuser=/home/anataraj/ktau-1.7.15-bgpzepto/user-src/include \
 +
            -ktaulib=/home/anataraj/ktau-1.7.15-bgpzepto/user-src/lib/ \
 +
            -ktausym=/proc/kallsyms
 +
</pre>
 +
The ktau* options tell TAU where to find: the patched kernel includes, the user-level KTAU utils includes, the user-level KTAU libs and the kernel symbol table (at application runtime) respectively.
  
 +
Then build as usual:
 +
<pre>
 +
make clean install
 +
</pre>
  
 +
This will provide a TAU that is capable of profiling MPI applications on ZeptoOS/BGP while also providing kernel-level performance information. It is assumed, of course, that ZeptoOS has also been configured and built with KTAU.
 +
<BR>
 +
<BR>
 +
===An Example===
 +
TAU has several example codes listed in the "tau-<version>/examples" directory. Depending on the configuration of TAU, not all of those examples can be run.
 +
<BR>
 +
====Building the 'ring' Example Code====
 +
If PDT and MPI have been included in the configuration, then the sample codes in "tau-<version>/examples/pdt_mpi/" directory can be run as follows:
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2> cd examples/pdt_mpi/c
  
 +
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls
 +
Makefile  ring.c
  
−
==TAU==
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> make
  
−
===Acquiring TAU===
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls
−
You can get [http://tau.uoegon.edu TAU] from the [http://www.cs.uoregon.edu/research/tau/downloads.php downloads page].
+
Makefile  ring  ring.c  ring.inst.c  ring.o
−
 
+
</pre>
−
===Configuration under ZeptoOS on BG/P===
+
<BR>
−
Before proceeding we assume that ZeptoOS has been downloaded, configured and installed. For the purpose of this how-to, the ZeptoOS install location is assumed to be "/home/anataraj/ZeptoReleases/install-BGP". You may have a different location or may opt to use the default ZeptoOS install location on your machine. Installing your own ZeptoOS can done by issuing 'python install.py /path/to/install' at the root-directory of the ZeptoOS release (but please refer to the detailed ZetpoOS documentation).
+
====Enabling ZCB-flag in 'ring'====
−
 
+
Once 'ring' is built it needs to be turned into a zepto-compute-binary (zcb) by turning a flag on in the elf-header of the binary. This is done using the 'zelftool' which is part of the ZeptoOS installation.
−
TAU needs to be told where the ZeptoOS install location is using the '-zeptodir' configure option.
+
<pre>
 +
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool ./ring
 +
zcb is off
 +
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool -e ./ring
 +
anataraj@login2.surveyor:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool ./ring
 +
zcb is on
 +
</pre>
 +
<BR>
 +
====Running 'ring' under a ZeptoOS kernel profile====
 +
Now this binary can be run on ZeptoOS/BGP by specifying the correct kernel-profile to cqsub. In the below example, the '-k anataraj' profile points to a ZeptoOS kernel patched with KTAU.
 +
<pre>
 +
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> cqsub -p ZeptoOS -k anataraj -t 10 -n 64 ./ring
 +
</pre>
 +
<BR>
 +
<BR>
 +
===Resulting Profile Data===
 +
After the run, user and kernel profiles will be created for each MPI rank. These (user-readable ascii) profiles can be read through the [http://www.cs.uoregon.edu/research/tau/docs/paraprof/index.html Paraprof Analysis and Visualization] tool in TAU.
 +
<BR>
 +
====User-level Profile====
 +
The user profiles are named "profile.<rank>.0.0". In this example run there are 64 ranks.
 
<pre>
 
<pre>
−
[email protected]:~/tau-2.18.1p2> pwd
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls profile.*
−
/home/anataraj/tau-2.18.1p2
+
profile.0.0.0  profile.16.0.0  profile.23.0.0  profile.30.0.0  profile.38.0.0  profile.45.0.0  profile.52.0.0  profile.6.0.0
−
 
+
profile.1.0.0  profile.17.0.0  profile.24.0.0  profile.31.0.0  profile.39.0.0  profile.46.0.0  profile.53.0.0  profile.60.0.0
−
anataraj@login2.surveyor:~/tau-2.18.1p2> ./configure -fullhelp | grep Zepto
+
profile.10.0.0  profile.18.0.0  profile.25.0.0  profile.32.0.0  profile.4.0.0  profile.47.0.0  profile.54.0.0  profile.61.0.0
−
-zeptodir=<dir> ............................. ZeptoOS install location.
+
profile.11.0.0  profile.19.0.0  profile.26.0.0  profile.33.0.0  profile.40.0.0  profile.48.0.0  profile.55.0.0  profile.62.0.0
 +
profile.12.0.0  profile.2.0.0  profile.27.0.0  profile.34.0.0  profile.41.0.0  profile.49.0.0  profile.56.0.0  profile.63.0.0
 +
profile.13.0.0  profile.20.0.0  profile.28.0.0  profile.35.0.0  profile.42.0.0  profile.5.0.0  profile.57.0.0  profile.7.0.0
 +
profile.14.0.0  profile.21.0.0  profile.29.0.0  profile.36.0.0  profile.43.0.0  profile.50.0.0  profile.58.0.0  profile.8.0.0
 +
profile.15.0.0  profile.22.0.0  profile.3.0.0  profile.37.0.0  profile.44.0.0  profile.51.0.0  profile.59.0.0  profile.9.0.0
 
</pre>
 
</pre>
−
 
+
[[Image:KTAU_ring_allranks_profile.png|200px|Paraprof screen-shot of user-level performance data from all 'ring' ranks.]]
−
A simple example configuration is shown below that includes the -mpi option. It is important to note that the TAU configuration under ZeptoOS on BGP is different from the standard TAU configuration on BGP when using the default IBM OS suite.  
+
<BR>
−
 
+
====Kernel-level Profile====
−
In this case the architecture needs to be specified as 'ppc' in addition to the ZeptoOS install location.
+
The kernel profiles for all processes corresponding to each node are under the respective "Kprofile.<rank>.0.perprocess" directories and there are 64 of those as well in this example. The files in these directories are numbered by their process-ids.
 
<pre>
 
<pre>
−
[email protected]:~/tau-2.18.1p2> ./configure -arch=ppc -zeptodir=/home/anataraj/ZeptoReleases/install-BGP -mpi
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls Kprofile.0.0.perprocess/
−
-------------------- TAU configure script ---------------
+
profile.1.0.0    profile.12.0.0  profile.17.0.0  profile.21.0.0  profile.26.0.0  profile.30.0.0  profile.44.0.0  profile.55.0.0  profile.9.0.0
−
  The TAU source code has just been configured to use the
+
profile.10.0.0  profile.13.0.0  profile.18.0.0  profile.22.0.0  profile.27.0.0  profile.31.0.0  profile.48.0.0  profile.6.0.0
−
  tau root directory /home/anataraj/tau-2.18.1p2.
+
profile.11.0.0  profile.14.0.0  profile.19.0.0  profile.23.0.0  profile.28.0.0  profile.32.0.0  profile.5.0.0  profile.62.0.0
−
  If you move the Tau distribution, you must either
+
profile.115.0.0  profile.15.0.0  profile.2.0.0  profile.24.0.0  profile.29.0.0  profile.34.0.0  profile.53.0.0  profile.7.0.0
−
    * set an environment variable TAUROOT containing the new
+
profile.116.0.1  profile.16.0.0  profile.20.0.0  profile.25.0.0  profile.3.0.0  profile.4.0.0  profile.54.0.0  profile.8.0.0
−
      location before running any TAU tools
 
−
  or
 
−
    * run configure again and recompile
 
−
-----------------------------------------------------------------
 
−
WARNING!! Auto-detect:ppc64 overridden with ppc
 
−
-PROFILE and/or -TRACE should be specified. Choosing -PROFILE as the default
 
−
Default C++ compiler will be  g++ version 4.1.2 20070115 (prerelease) (SUSE Linux)
 
−
Thread-safe MPI interface is defined...  
 
−
MPI-2 Interface is defined...
 
−
MPI-2 Grequest Interface is defined...
 
−
MPI-2 MPI_Datarep_conversion_function Interface is defined...
 
−
MPI-2 Comm_create_errhandler Interface is defined...
 
−
MPI-2 MPI*attr_functions are defined...
 
−
MPI-2 MPI_File functions are defined...
 
−
MPI-2 MPI_Type_dup and MPI_Exscan are defined...
 
−
MPI-2 MPI_Add_error* functions are defined...
 
−
MPI_Status f2c/c2f found...
 
−
NOTE: MPI library does not have a threaded _r suffix
 
−
Default C compiler will be /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc
 
−
Checking if open takes O_LARGEFILE... yes
 
−
Checking for weak symbols... yes
 
−
C++ compiler supports Std Runtime Type Information (default)
 
−
Testing for Standard C++ Library support for string class
 
−
Found support for Standard C++ Library string
 
−
NOTE: Compiling Profiling Library and Applications with -DTAU_STDCXXLIB
 
−
***********************************************************************
 
−
No special modifications found for architecture "ppc"
 
−
Checking for bfd.h... no
 
−
Checking for shared library compatible libbfd... no
 
−
TAU: installing TAU's ParaProf profile browser
 
−
jdk version 1.4 or better found
 
−
TAU: installed tau_compiler.sh in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed tau_load.sh in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed Java trace reader/writer api in /home/anataraj/tau-2.18.1p2/ppc/lib
 
−
TAU: installed tau2slog2, slog2print and jumpshot in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed required slog2sdk jars in /home/anataraj/tau-2.18.1p2/ppc/lib
 
−
TAU: installed tauinc.pl in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed TAU's PerfDMF database utilities in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed ParaProf, PerfDMF, PerfExplorer and supporting jars in:
 
−
/home/anataraj/tau-2.18.1p2/ppc/lib
 
−
Platform does not support JOGL, no 3d displays will be available in ParaProf
 
−
TAU: installed tau_analyze in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
TAU: installed Makefile.inc in /home/anataraj/tau-2.18.1p2/tools/src/TauIL
 
−
TAU: installed TauIL and supporting jars in /home/anataraj/tau-2.18.1p2/ppc/lib
 
−
TAU: installed tau_throttle script
 
−
TAU: installing tools in /home/anataraj/tau-2.18.1p2/ppc/bin
 
−
 
 
−
NOTE: *** Using ZeptoOS BGP gfortran. ***
 
−
NOTE: Enabled Profiling. Compiling with -DPROFILING_ON
 
−
NOTE: Using the TAU MPI Profiling Interface
 
−
NOTE: Using the TAU MPI_Init_thread Profiling Interface Wrapper
 
−
NOTE: Using TAU's MPI-2 extensions
 
−
NOTE: Using TAU's MPI-2 Grequest extensions
 
−
NOTE: Using TAU's MPI-2 Datarep_conversion extensions
 
−
NOTE: Using TAU's MPI-2 Error handler extensions
 
−
NOTE: Using TAU's MPI-2 Attr extensions
 
−
NOTE: Using TAU's MPI-2 File extensions
 
−
NOTE: Using TAU's MPI-2 Type and Exscan extensions
 
−
NOTE: Using TAU's MPI-2 Add_error extensions
 
−
NOTE: Using default cc compiler.
 
−
NOTE: Using the O_LARGEFILE flag to open
 
−
NOTE: Using Weak Symbol support
 
−
NOTE: Using Standard C++ Library
 
−
NOTE: Disabling building of shared objects
 
−
Script to modify Makefiles created.
 
−
Applying script to all Makefiles, please wait, this may take a while...
 
−
.........................................................................................................................................................
 
−
 
 
−
Configuring TAU Build scripts...
 
−
Modifying Makefiles in the examples subdirectory...
 
−
...........
 
−
 
 
−
Configuration complete!
 
 
</pre>
 
</pre>
 +
[[Image:KTAU_ring-rank0-kprofile.png|300px|Paraprof screen-shot of kernel-level performance data from 'ring' rank 0.]]
 +
<BR>
 +
====Integrated Profile====
 +
One method to obtain an integrated profile is to specify which kernel activities need to be closely tracked within the context of user-level application routines. The integration of TAU and KTAU allows specification of those activities through TAU's multiple-counter interface.
  
−
Next build TAU in-place by issuing a 'make install'.
+
For instance, if close tracking of the "timer_interrupt" and "schedule" kernel operations are required then the following cqsub command may be issued:
 
<pre>
 
<pre>
−
[email protected]:~/tau-2.18.1p2> make install
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> cqsub -p ZeptoOS -k anataraj -t 10 -n 64 \
 +
                -e COUNTER1=GET_TIME_OF_DAY:COUNTER2=KTAU_INCL_timer_interrupt:COUNTER3=KTAU_schedule \
 +
                ./ring
 
</pre>
 
</pre>
−
This should provide a TAU built for ZeptoOS with MPI profiling capability. Please refer to the [http://tau.uoregon.edu TAU] documentation to various TAU configuration options that may be used.
+
The above command uses environment variables to tell TAU to track the two specific kernel operations in addition to standard elapsed time (gettimeofday). Any instrumented kernel symbol can be used with 'KTAU_' prefixed.
  
−
===Example Run===
+
The resulting profile data is placed in "MULTI__*" directories, one for each specified counter.
−
 
+
<pre>
−
==Using TAU with KTAU : Integrated Profiles==
+
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls | grep MULTI__
−
===Configuring under ZeptoOS===
+
MULTI__GET_TIME_OF_DAY
−
===Example Run===
+
MULTI__INCL_timer_interrupt
 +
MULTI__schedule
  
 +
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls MULTI__INCL_timer_interrupt/
 +
profile.0.0.0  profile.16.0.0  profile.23.0.0  profile.30.0.0  profile.38.0.0  profile.45.0.0  profile.52.0.0  profile.6.0.0
 +
profile.1.0.0  profile.17.0.0  profile.24.0.0  profile.31.0.0  profile.39.0.0  profile.46.0.0  profile.53.0.0  profile.60.0.0
 +
profile.10.0.0  profile.18.0.0  profile.25.0.0  profile.32.0.0  profile.4.0.0  profile.47.0.0  profile.54.0.0  profile.61.0.0
 +
profile.11.0.0  profile.19.0.0  profile.26.0.0  profile.33.0.0  profile.40.0.0  profile.48.0.0  profile.55.0.0  profile.62.0.0
 +
profile.12.0.0  profile.2.0.0  profile.27.0.0  profile.34.0.0  profile.41.0.0  profile.49.0.0  profile.56.0.0  profile.63.0.0
 +
profile.13.0.0  profile.20.0.0  profile.28.0.0  profile.35.0.0  profile.42.0.0  profile.5.0.0  profile.57.0.0  profile.7.0.0
 +
profile.14.0.0  profile.21.0.0  profile.29.0.0  profile.36.0.0  profile.43.0.0  profile.50.0.0  profile.58.0.0  profile.8.0.0
 +
profile.15.0.0  profile.22.0.0  profile.3.0.0  profile.37.0.0  profile.44.0.0  profile.51.0.0  profile.59.0.0  profile.9.0.0
 +
</pre>
 +
[[Image:KTAU_ring_allranks_kshctr_timerinterrupt.png|300px|Paraprof screen-shot of timer_interrupt activity observed from within the user-level routines of the 64 ranks of the 'ring' application.]][[Image:KTAU_ring_rank0_shctr_timer_interrupt.png|400px|Paraprof screen-shot of timer_interrupt activity observed from within the user-level routines of Rank-0 of the 'ring' application.]]
  
 
----
 
----
−
[[ZeptoOS_Documentation|Top]]
+
[[ZOID]] | [[ZeptoOS_Documentation|Top]] | [[Other Packages]]

Latest revision as of 15:06, 5 May 2009

ZOID | Top | Other Packages

Both TAU (Tuning and Analysis Utilities) and its Linux kernel counterpart, KTAU have been ported to ZeptoOS. In addition the build system of the ZeptoOS kernel has been modified to simplify the integration of KTAU patches. The following sections describe how to acquire, configure, build and use TAU and KTAU under ZeptoOS, with Surveyor at ANL as the test platform.


TAU

Acquiring TAU

You can download the current (as of 2009-04-28) supported version 2.18.1p2-zepto of TAU. All future versions (greater than 2.18.1) should support ZeptoOS and will be available from the usual TAU downloads page.

Configuration under ZeptoOS on BG/P

Before proceeding we assume that ZeptoOS has been downloaded, configured and installed. For the purpose of this how-to, the ZeptoOS install location is assumed to be "/home/anataraj/ZeptoReleases/install-BGP". You may have a different location or may opt to use the default ZeptoOS install location on your machine. Installing your own ZeptoOS can done by issuing 'python install.py /path/to/install' at the root-directory of the ZeptoOS release (but please refer to the detailed ZeptoOS documentation).

TAU needs to be told where the ZeptoOS install location is using the '-zeptodir' configure option. 

[email protected]:~/tau-2.18.1p2> pwd
/home/anataraj/tau-2.18.1p2

[email protected]:~/tau-2.18.1p2> ./configure -fullhelp | grep Zepto
-zeptodir=<dir> ............................. ZeptoOS install location.

A simple example configuration is shown below that includes the -mpi option. It is important to note that the TAU configuration under ZeptoOS on BGP is different from the standard TAU configuration on BGP when using the default IBM OS suite.

In this case the architecture needs to be specified as 'ppc' in addition to the ZeptoOS install location. 

[email protected]:~/tau-2.18.1p2> ./configure -arch=ppc \
                                                     -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
                                                     -mpi



Configuring with PDT under ZeptoOS on BGP

The Program Database Toolkit(PDT) provides TAU with automatic source instrumentation capability. This allows routines, particularly in large projects, to be instrumented without tedious manual intervention.

First download and untar PDT. Then configure and build it: 

[email protected]:~/pdtoolkit-3.14> pwd
/home/anataraj/pdtoolkit-3.14

[email protected]:~/pdtoolkit-3.14> ./configure

[email protected]:~/pdtoolkit-3.14> make

[email protected]:~/pdtoolkit-3.14> make install

Next configure TAU with the above PDT: 

[email protected]:~/tau-2.18.1p2> ./configure  -arch=ppc \
                                                      -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
                                                      -mpi \
                                                      -pdt=/home/anataraj/pdtoolkit-3.14 \
                                                      -pdtarchdir=ppc64 -pdt_c++=xlC

Please pay close attention to the '-pdtarchdir=' and the '-pdt_c++=' options as they are important to it working on the ZeptoOS/BGP platform.

Please refer to the TAU documentation for various other TAU configuration options that may be used.

Building TAU

Lastly, build TAU in-place by issuing a 'make install'. 

[email protected]:~/tau-2.18.1p2> make install

This should provide a TAU built for ZeptoOS with MPI profiling and automatic instrumentation capabilities.







KTAU

Acquiring KTAU

You can download the latest version (1.7.15-bgpzepto-p1) of KTAU available as a gzipped tarball from the KTAU downloads page.

Integrating KTAU into ZeptoOS

For the below examples showing the configuration and build steps, it is assumed that the KTAU tarball has been expanded into the location /home/anataraj/ktau-1.7.15-bgpzepto and that the ZeptoOS tarball has been expanded into /home/anataraj/BGP.

Configure ZeptoOS to point to KTAU patch and path

Begin by configuring the ZeptoOS by pointing it to the correct KTAU path and the name of the patch to be used. 

[email protected]:~/BGP> ./configure --edit

KTAU configure-edit.jpg

Once this is done, the changes will be saved to the Make.rules file. 

[email protected]:~/BGP> cat Make.rules
#
# Automatically generated make config: don't edit
#

#
# BG/P DIST_DIR
#
DRV_DIR="/bgsys/drivers/ppcfloor/"
BGP_CROSS="$(DRV_DIR)/gnu-linux/bin/powerpc-bgp-linux-"
BGCNS_H_PATH="$(DRV_DIR)/mcp-2.6.16.46/include/"
BGCNS_H="$(BGCNS_H_PATH)/bgcns.h"
OS_DIR="/bgsys/linux/1.3.020081029"

#
# KTAU
#
KTAU_DIR="/home/anataraj/ktau-1.7.15-bgpzepto"
KTAU_PATCH="patch-2.6.19.2-ktau-1.7.15-bgpzepto"


Patching ZeptoOS with KTAU

Step down into the "kernel" directory of ZeptoOS next. Issuing a 'make help' will show all the targets, including the KTAU specific ones (i.e. the last two targets in the list). 

[email protected]:~/BGP> cd kernel/

[email protected]:~/BGP/kernel> make help

[menu]

bgp-ion-linux                 : Simply use recent built or prebuilt ION kernel (no kernel rebuild)
bgp-ion-linux-prebuilt        : Use prebuilt ION kernel
bgp-ion-linux-menuconfig      : Invoke ION kernel menuconfig
bgp-ion-linux-build           : Rebuild ION kernel and copy
bgp-ion-linux-try-patching    : Try to apply new patch to ION kernel
bgp-ion-linux-update-prebuilt : Copy ION kernel from tmp dir to prebuilt dir (no rebuild)

bgp-cn-linux                  : Rebuild CN ramdisk and merge with recent built or prebuilt CN kernel objs
bgp-cn-linux-prebuilt         : Use prebuilt CN kernel
bgp-cn-linux-menuconfig       : Invoke CN kernel menuconfig
bgp-cn-linux-build            : Rebuild CN kernel objs and CN ramdisk, then merge them into CN Kernel
bgp-cn-linux-try-patching     : Try to apply new patch to CN kernel
bgp-cn-linux-update-prebuilt  : Copy CN kernel objs from tmp dir to prebuilt dir (no build)
bgp-cn-linux-add-ktau         : Patch CN kernel with KTAU and link in KTAU headers & sources
bgp-cn-linux-del-ktau         : Remove KTAU patch and links from CN kernel

NOTE:
- The following targets copy prebuilt or newly built kernel image to the top dir
  bgp-ion-linux, bgp-ion-linux-build, bgp-cn-linux, bgp-cn-linux-build and bgp-cn-linux-ramdisk

- 'make GIT=1 ...' clone repo from public git repo via http and use them as target source tree
- By default, kernel tarballs are extracted and use them as target source tree

Issuing 'make bgp-cn-linux-add-ktau' patches the ZeptoOS compute node Linux kernel with the configured KTAU patch and then links to headers and sources from the KTAU path. 

[email protected]:~/BGP/kernel> make bgp-cn-linux-add-ktau 
[ -d work ] || mkdir work
tar xfj tarball/linux-2.6.19.2-BGP-V1R3.tar.bz2 -C work
if [ -f tarball/linux-2.6.19.2-BGP-V1R3.patch ] ; then \
	( cd work/linux-2.6.19.2-BGP-V1R3 ; patch -p1 < ../../tarball/linux-2.6.19.2-BGP-V1R3.patch ) ; \
	cp tarball/linux-2.6.19.2-BGP-V1R3.patch work/linux-2.6.19.2-BGP-V1R3.patch.applied ; \
fi
patching file Makefile
patching file arch/ppc/Kconfig
patching file arch/ppc/kernel/head_44x.S
patching file arch/ppc/kernel/vmlinux.lds.S
patching file arch/ppc/mm/zepto_flatmem.c
patching file arch/ppc/syslib/bgdd/bluegene_console.c
patching file arch/ppc/syslib/bgdd/bluegene_dma.c
patching file bgp-cn-2.6.19.2-dot-config
patching file bgp-ion-flatmem-2.6.19.2-dot-config
patching file fs/Kconfig
patching file include/asm-ppc/page.h
patching file include/asm-ppc/pgtable.h
patching file include/linux/mmzone.h
patching file kernel/exit.c
touch work/.cn_kernel_prepared
Linking to KTAU headers...
Linking to KTAU sources...
Linking to KTAU make...
Applying KTAU patch...
patching file arch/powerpc/kernel/irq.c
patching file arch/powerpc/oprofile/Kconfig
patching file arch/ppc/kernel/entry.S
Hunk #4 succeeded at 306 (offset -6 lines).
patching file arch/ppc/kernel/smp.c
patching file arch/ppc/kernel/time.c
patching file arch/ppc/kernel/traps.c
Hunk #1 succeeded at 69 (offset -7 lines).
patching file arch/ppc/syslib/bgdd/bluegene_network.c
Hunk #2 succeeded at 496 (offset -34 lines).
Hunk #3 succeeded at 513 (offset -34 lines).
patching file include/linux/sched.h
patching file init/main.c
patching file kernel/fork.c
patching file kernel/irq/handle.c
patching file kernel/Makefile
patching file kernel/sched.c
patching file kernel/softirq.c
patching file kernel/timer.c
patching file kernel/workqueue.c
patching file Makefile
NOTE: Integration of KTAU into ZeptoOS is complete.
 Next, please configure KTAU kernel options using "make bgp-cn-linux-menuconfig".
 After that, please follow the standard ZeptoOS build steps (e.g. "make bgp-cn-linux-build").


Configuring KTAU parameters of the patched ZeptoOS CN Linux

Next the KTAU parameters of the kernel configuration need to be set. This can be done by issuing a 'make bgp-cn-linux-menuconfig'. 

[email protected]:~/BGP/kernel> make bgp-cn-linux-menuconfig

KTAU menuconfig.jpg

Building ZeptoOS with KTAU

After configuration is performed and saved, the kernel needs to be built. 

[email protected]:~/BGP/kernel> make bgp-cn-linux-build

Do not forget to update the prebuilt binaries with the kernel image you just built. 

[email protected]:~/BGP/kernel> make bgp-cn-linux-update-prebuilt 
Copying kernel objs from tmp/cnkernelobjs to prebuilt/2.6.19/objs/ ...
done
CN kernel prebuilt objs and kernel image has been updated

Lastly, go back up to the root-level ZeptoOS directory and perform a 'make' to set / update / build any other targets. 

[email protected]:~/BGP/kernel> cd ..

[email protected]:~/BGP> pwd
/home/anataraj/BGP

[email protected]:~/BGP> make 
Create root password for I/O Node
   Leave the password field empty if you want to disable root login
   New password:
   Retype new password:
<snipped out>
Zepto Kernel and ramdisk images are ready:
BGP-CN-zImage-with-initrd.elf
BGP-ION-ramdisk.elf
BGP-ION-ramdisk-for-CNL.elf
BGP-ION-zImage.elf

You should now have a KTAU-patched and configured ZeptoOS Compute Node Linux kernel ready for use.

Building KTAU User-level Utilities

Next please build the user-level utils of KTAU, as follows: 

[email protected]:~/ktau-1.7.15-bgpzepto> pwd
/home/anataraj/ktau-1.7.15-bgpzepto

[email protected]:~/ktau-1.7.15-bgpzepto> cd user-src/
[email protected]:~/ktau-1.7.15-bgpzepto/user-src> cd src/

[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> make \
                          ARCH=ppc \
                          BGP_KERNEL_INC=/home/anataraj/ZeptoReleases/BGP/kernel/work/linux-2.6.19.2-BGP-V1R3/include/ \
                          clean all

[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> cd runktau/
[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src/runktau> ls -alhtr timeKtau
-rwxr-xr-x 1 anataraj users 6.0M 2009-04-28 13:09 timeKtau

[email protected]:~/ktau-1.7.15-bgpzepto/user-src/src> cd ..
[email protected]:~/ktau-1.7.15-bgpzepto/user-src> ls lib/
libktau.a



Removing KTAU from ZeptoOS

If you wish to revert back to the original version of ZeptoOS without KTAU integration, please perform the below steps. 

[email protected]:~/BGP> pwd
/home/anataraj/BGP

[email protected]:~/BGP> cd kernel/

[email protected]:~/BGP/kernel> make bgp-cn-linux-del-ktau
Reversing the KTAU patch (using a local copy of the applied patch)...
patching file arch/powerpc/kernel/irq.c
patching file arch/powerpc/oprofile/Kconfig
patching file arch/ppc/kernel/entry.S
Hunk #4 succeeded at 287 (offset -6 lines).
patching file arch/ppc/kernel/smp.c
patching file arch/ppc/kernel/time.c
patching file arch/ppc/kernel/traps.c
Hunk #1 succeeded at 69 (offset -7 lines).
patching file arch/ppc/syslib/bgdd/bluegene_network.c
Hunk #2 succeeded at 493 (offset -34 lines).
Hunk #3 succeeded at 505 (offset -34 lines).
patching file include/linux/sched.h
patching file init/main.c
patching file kernel/fork.c
patching file kernel/irq/handle.c
patching file kernel/Makefile
patching file kernel/sched.c
patching file kernel/softirq.c
patching file kernel/timer.c
patching file kernel/workqueue.c
patching file Makefile
NOTE: KTAU integration has been undone (patch reversed and all links removed).
 Next, please reconfigure and build the ZeptoOS CN Linux following standard steps.
 You can "make bgp-cn-linux-menuconfig" and "make bgp-cn-linux-build".

Please remember to reconfigure and rebuild ZeptoOS after undoing the KTAU integration.







Using TAU with KTAU : Integrated Profiles

TAU can be used in a tightly coupled fashion with KTAU to provide integrated profiles that show both user and kernel performance data under certain configurations. The following subsection describes such a configuration.

Configuring under ZeptoOS

Use all the previously described configuration options, but in addition add the -MULTIPLECOUNTERS and -ktau* options to the configure line. As follows: 

./configure  -arch=ppc -zeptodir=/home/anataraj/ZeptoReleases/install-BGP \
             -mpi \
             -pdt=/home/anataraj/pdtoolkit-3.14 -pdtarchdir=ppc64 -pdt_c++=xlC \
             -MULTIPLECOUNTERS \
             -ktau -ktau_shctr \
             -ktauinc=/home/anataraj/BGP/kernel/work/linux-2.6.19.2-BGP-V1R3/include \
             -ktauincuser=/home/anataraj/ktau-1.7.15-bgpzepto/user-src/include \
             -ktaulib=/home/anataraj/ktau-1.7.15-bgpzepto/user-src/lib/ \
             -ktausym=/proc/kallsyms

The ktau* options tell TAU where to find: the patched kernel includes, the user-level KTAU utils includes, the user-level KTAU libs and the kernel symbol table (at application runtime) respectively.

Then build as usual: 

make clean install

This will provide a TAU that is capable of profiling MPI applications on ZeptoOS/BGP while also providing kernel-level performance information. It is assumed, of course, that ZeptoOS has also been configured and built with KTAU.

An Example

TAU has several example codes listed in the "tau-<version>/examples" directory. Depending on the configuration of TAU, not all of those examples can be run.

Building the 'ring' Example Code

If PDT and MPI have been included in the configuration, then the sample codes in "tau-<version>/examples/pdt_mpi/" directory can be run as follows: 

[email protected]:~/tau-2.18.1p2> cd examples/pdt_mpi/c

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls
Makefile  ring.c

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> make

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls
Makefile  ring  ring.c  ring.inst.c  ring.o


Enabling ZCB-flag in 'ring'

Once 'ring' is built it needs to be turned into a zepto-compute-binary (zcb) by turning a flag on in the elf-header of the binary. This is done using the 'zelftool' which is part of the ZeptoOS installation. 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool ./ring
zcb is off
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool -e ./ring
[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> /home/anataraj/ZeptoReleases/install-BGP/bin/zelftool ./ring
zcb is on


Running 'ring' under a ZeptoOS kernel profile

Now this binary can be run on ZeptoOS/BGP by specifying the correct kernel-profile to cqsub. In the below example, the '-k anataraj' profile points to a ZeptoOS kernel patched with KTAU. 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> cqsub -p ZeptoOS -k anataraj -t 10 -n 64 ./ring



Resulting Profile Data

After the run, user and kernel profiles will be created for each MPI rank. These (user-readable ascii) profiles can be read through the Paraprof Analysis and Visualization tool in TAU.

User-level Profile

The user profiles are named "profile.<rank>.0.0". In this example run there are 64 ranks. 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls profile.*
profile.0.0.0   profile.16.0.0  profile.23.0.0  profile.30.0.0  profile.38.0.0  profile.45.0.0  profile.52.0.0  profile.6.0.0
profile.1.0.0   profile.17.0.0  profile.24.0.0  profile.31.0.0  profile.39.0.0  profile.46.0.0  profile.53.0.0  profile.60.0.0
profile.10.0.0  profile.18.0.0  profile.25.0.0  profile.32.0.0  profile.4.0.0   profile.47.0.0  profile.54.0.0  profile.61.0.0
profile.11.0.0  profile.19.0.0  profile.26.0.0  profile.33.0.0  profile.40.0.0  profile.48.0.0  profile.55.0.0  profile.62.0.0
profile.12.0.0  profile.2.0.0   profile.27.0.0  profile.34.0.0  profile.41.0.0  profile.49.0.0  profile.56.0.0  profile.63.0.0
profile.13.0.0  profile.20.0.0  profile.28.0.0  profile.35.0.0  profile.42.0.0  profile.5.0.0   profile.57.0.0  profile.7.0.0
profile.14.0.0  profile.21.0.0  profile.29.0.0  profile.36.0.0  profile.43.0.0  profile.50.0.0  profile.58.0.0  profile.8.0.0
profile.15.0.0  profile.22.0.0  profile.3.0.0   profile.37.0.0  profile.44.0.0  profile.51.0.0  profile.59.0.0  profile.9.0.0

Paraprof screen-shot of user-level performance data from all 'ring' ranks.

Kernel-level Profile

The kernel profiles for all processes corresponding to each node are under the respective "Kprofile.<rank>.0.perprocess" directories and there are 64 of those as well in this example. The files in these directories are numbered by their process-ids. 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls Kprofile.0.0.perprocess/
profile.1.0.0    profile.12.0.0  profile.17.0.0  profile.21.0.0  profile.26.0.0  profile.30.0.0  profile.44.0.0  profile.55.0.0  profile.9.0.0
profile.10.0.0   profile.13.0.0  profile.18.0.0  profile.22.0.0  profile.27.0.0  profile.31.0.0  profile.48.0.0  profile.6.0.0
profile.11.0.0   profile.14.0.0  profile.19.0.0  profile.23.0.0  profile.28.0.0  profile.32.0.0  profile.5.0.0   profile.62.0.0
profile.115.0.0  profile.15.0.0  profile.2.0.0   profile.24.0.0  profile.29.0.0  profile.34.0.0  profile.53.0.0  profile.7.0.0
profile.116.0.1  profile.16.0.0  profile.20.0.0  profile.25.0.0  profile.3.0.0   profile.4.0.0   profile.54.0.0  profile.8.0.0

Paraprof screen-shot of kernel-level performance data from 'ring' rank 0.

Integrated Profile

One method to obtain an integrated profile is to specify which kernel activities need to be closely tracked within the context of user-level application routines. The integration of TAU and KTAU allows specification of those activities through TAU's multiple-counter interface.

For instance, if close tracking of the "timer_interrupt" and "schedule" kernel operations are required then the following cqsub command may be issued: 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> cqsub -p ZeptoOS -k anataraj -t 10 -n 64 \
                -e COUNTER1=GET_TIME_OF_DAY:COUNTER2=KTAU_INCL_timer_interrupt:COUNTER3=KTAU_schedule \
                ./ring

The above command uses environment variables to tell TAU to track the two specific kernel operations in addition to standard elapsed time (gettimeofday). Any instrumented kernel symbol can be used with 'KTAU_' prefixed.

The resulting profile data is placed in "MULTI__*" directories, one for each specified counter. 

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls | grep MULTI__
MULTI__GET_TIME_OF_DAY
MULTI__INCL_timer_interrupt
MULTI__schedule

[email protected]:~/tau-2.18.1p2/examples/pdt_mpi/c> ls MULTI__INCL_timer_interrupt/
profile.0.0.0   profile.16.0.0  profile.23.0.0  profile.30.0.0  profile.38.0.0  profile.45.0.0  profile.52.0.0  profile.6.0.0
profile.1.0.0   profile.17.0.0  profile.24.0.0  profile.31.0.0  profile.39.0.0  profile.46.0.0  profile.53.0.0  profile.60.0.0
profile.10.0.0  profile.18.0.0  profile.25.0.0  profile.32.0.0  profile.4.0.0   profile.47.0.0  profile.54.0.0  profile.61.0.0
profile.11.0.0  profile.19.0.0  profile.26.0.0  profile.33.0.0  profile.40.0.0  profile.48.0.0  profile.55.0.0  profile.62.0.0
profile.12.0.0  profile.2.0.0   profile.27.0.0  profile.34.0.0  profile.41.0.0  profile.49.0.0  profile.56.0.0  profile.63.0.0
profile.13.0.0  profile.20.0.0  profile.28.0.0  profile.35.0.0  profile.42.0.0  profile.5.0.0   profile.57.0.0  profile.7.0.0
profile.14.0.0  profile.21.0.0  profile.29.0.0  profile.36.0.0  profile.43.0.0  profile.50.0.0  profile.58.0.0  profile.8.0.0
profile.15.0.0  profile.22.0.0  profile.3.0.0   profile.37.0.0  profile.44.0.0  profile.51.0.0  profile.59.0.0  profile.9.0.0

Paraprof screen-shot of timer_interrupt activity observed from within the user-level routines of the 64 ranks of the 'ring' application.Paraprof screen-shot of timer_interrupt activity observed from within the user-level routines of Rank-0 of the 'ring' application.

ZOID | Top | Other Packages

Retrieved from "http:/// /zeptoos/index.php?title=(K)TAU&oldid=534"