(K)TAU

Top

---

THIS SECTION UNDER CONSTRUCTION, PLEASE RETURN LATER

Both TAU (Tuning and Analysis Utilities) and its Linux kernel counterpart, KTAU have been ported to ZeptoOS. In addition the build system of the ZeptoOS kernel has been modified to simplify the integration of KTAU patches. The following sections describe how to acquire, configure, build and use TAU and KTAU under ZeptoOS, with Surveyor at ANL as the test platform. The KTAU section appears first as the TAU section has some dependencies on that information.

KTAU

Acquiring KTAU

You can download a gzipped tarball of KTAU (version: 1.7.15-bgpzepto) from http://www.cs.uoregon.edu/research/ktau/downloads.php .

Integrating KTAU into ZeptoOS

For the below examples showing the configuration and build steps, it is assumed that the KTAU tarball has been expanded into the location /home/anataraj/ktau-1.7.15-bgpzepto and that the ZeptoOS tarball has been expanded into /home/anataraj/BGP.

Configure ZeptoOS to point to KTAU path and patch

Begin by configuring the ZeptoOS by pointing it to the correct KTAU path and the name of the patch to be used.

[email protected]:~/BGP> ./configure --edit

Once this is done, the changes will be saved to the Make.rules file as can be checked.

[email protected]:~/BGP> cat Make.rules
#
# Automatically generated make config: don't edit
#

#
# BG/P DIST_DIR
#
DRV_DIR="/bgsys/drivers/ppcfloor/"
BGP_CROSS="$(DRV_DIR)/gnu-linux/bin/powerpc-bgp-linux-"
BGCNS_H_PATH="$(DRV_DIR)/mcp-2.6.16.46/include/"
BGCNS_H="$(BGCNS_H_PATH)/bgcns.h"
OS_DIR="/bgsys/linux/1.3.020081029"

#
# KTAU
#
KTAU_DIR="/home/anataraj/ktau-1.7.15-bgpzepto"
KTAU_PATCH="patch-2.6.19.2-ktau-1.7.15-bgpzepto"

Patching ZeptoOS with KTAU

Step down into the "kernel" directory of ZeptoOS next. Issuing a 'make help' will show all the targets, including the KTAU specific ones.

[email protected]:~/BGP> cd kernel/

[email protected]:~/BGP/kernel> make help

[menu]

bgp-ion-linux                 : Simply use recent built or prebuilt ION kernel (no kernel rebuild)
bgp-ion-linux-prebuilt        : Use prebuilt ION kernel
bgp-ion-linux-menuconfig      : Invoke ION kernel menuconfig
bgp-ion-linux-build           : Rebuild ION kernel and copy
bgp-ion-linux-try-patching    : Try to apply new patch to ION kernel
bgp-ion-linux-update-prebuilt : Copy ION kernel from tmp dir to prebuilt dir (no rebuild)

bgp-cn-linux                  : Rebuild CN ramdisk and merge with recent built or prebuilt CN kernel objs
bgp-cn-linux-prebuilt         : Use prebuilt CN kernel
bgp-cn-linux-menuconfig       : Invoke CN kernel menuconfig
bgp-cn-linux-build            : Rebuild CN kernel objs and CN ramdisk, then merge them into CN Kernel
bgp-cn-linux-try-patching     : Try to apply new patch to CN kernel
bgp-cn-linux-update-prebuilt  : Copy CN kernel objs from tmp dir to prebuilt dir (no build)
bgp-cn-linux-add-ktau         : Patch CN kernel with KTAU and link in KTAU headers & sources
bgp-cn-linux-del-ktau         : Remove KTAU patch and links from CN kernel

NOTE:
- The following targets copy prebuilt or newly built kernel image to the top dir
  bgp-ion-linux, bgp-ion-linux-build, bgp-cn-linux, bgp-cn-linux-build and bgp-cn-linux-ramdisk

- 'make GIT=1 ...' clone repo from public git repo via http and use them as target source tree
- By default, kernel tarballs are extracted and use them as target source tree

Issuing 'make bgp-cn-linux-add-ktau' target patches the ZeptoOS compute node Linux kernel with KTAU the configured patch and then links to headers and sources from the KTAU path.

[email protected]:~/BGP/kernel> make bgp-cn-linux-add-ktau 
[ -d work ] || mkdir work
tar xfj tarball/linux-2.6.19.2-BGP-V1R3.tar.bz2 -C work
if [ -f tarball/linux-2.6.19.2-BGP-V1R3.patch ] ; then \
	( cd work/linux-2.6.19.2-BGP-V1R3 ; patch -p1 < ../../tarball/linux-2.6.19.2-BGP-V1R3.patch ) ; \
	cp tarball/linux-2.6.19.2-BGP-V1R3.patch work/linux-2.6.19.2-BGP-V1R3.patch.applied ; \
fi
patching file Makefile
patching file arch/ppc/Kconfig
patching file arch/ppc/kernel/head_44x.S
patching file arch/ppc/kernel/vmlinux.lds.S
patching file arch/ppc/mm/zepto_flatmem.c
patching file arch/ppc/syslib/bgdd/bluegene_console.c
patching file arch/ppc/syslib/bgdd/bluegene_dma.c
patching file bgp-cn-2.6.19.2-dot-config
patching file bgp-ion-flatmem-2.6.19.2-dot-config
patching file fs/Kconfig
patching file include/asm-ppc/page.h
patching file include/asm-ppc/pgtable.h
patching file include/linux/mmzone.h
patching file kernel/exit.c
touch work/.cn_kernel_prepared
Linking to KTAU headers...
Linking to KTAU sources...
Linking to KTAU make...
Applying KTAU patch...
patching file arch/powerpc/kernel/irq.c
patching file arch/powerpc/oprofile/Kconfig
patching file arch/ppc/kernel/entry.S
Hunk #4 succeeded at 306 (offset -6 lines).
patching file arch/ppc/kernel/smp.c
patching file arch/ppc/kernel/time.c
patching file arch/ppc/kernel/traps.c
Hunk #1 succeeded at 69 (offset -7 lines).
patching file arch/ppc/syslib/bgdd/bluegene_network.c
Hunk #2 succeeded at 496 (offset -34 lines).
Hunk #3 succeeded at 513 (offset -34 lines).
patching file include/linux/sched.h
patching file init/main.c
patching file kernel/fork.c
patching file kernel/irq/handle.c
patching file kernel/Makefile
patching file kernel/sched.c
patching file kernel/softirq.c
patching file kernel/timer.c
patching file kernel/workqueue.c
patching file Makefile
NOTE: Integration of KTAU into ZeptoOS is complete.
 Next, please configure KTAU kernel options using "make bgp-cn-linux-menuconfig".
 After that, please follow the standard ZeptoOS build steps (e.g. "make bgp-cn-linux-build").

Configuring KTAU parameters of the patched ZeptoOS CN Linux

Next the KTAU parameters of the kernel configuration need to be set. This can be done by issuing a 'make bgp-cn-linux-menuconfig'.

[email protected]:~/BGP/kernel> make bgp-cn-linux-menuconfig 

Building ZeptoOS with KTAU

After configuration is performed and saved, the kernel needs to be built.

[email protected]:~/BGP/kernel> make bgp-cn-linux-build

Do not forget to update the prebuilt binaries with the kernel image you just built.

[email protected]:~/BGP/kernel> make bgp-cn-linux-update-prebuilt 
Copying kernel objs from tmp/cnkernelobjs to prebuilt/2.6.19/objs/ ...
done
CN kernel prebuilt objs and kernel image has been updated

Lastly, go back up to the root-level ZeptoOS directory and perform a 'make' to set / update / build any other targets.

[email protected]:~/BGP/kernel> cd ..

[email protected]:~/BGP> pwd
/home/anataraj/BGP

[email protected]:~/BGP> make 
Create root password for I/O Node
   Leave the password field empty if you want to disable root login
   New password:
   Retype new password:
<snipped out>
Zepto Kernel and ramdisk images are ready:
BGP-CN-zImage-with-initrd.elf
BGP-ION-ramdisk.elf
BGP-ION-ramdisk-for-CNL.elf
BGP-ION-zImage.elf

You should now have a KTAU-patched and configured ZeptoOS Compute Node Linux kernel ready for use.

Removing KTAU from ZeptoOS

If you wish to revert back to the original version of ZeptoOS without KTAU integration, please perform the below steps.

[email protected]:~/BGP> pwd
/home/anataraj/BGP

[email protected]:~/BGP> cd kernel/

[email protected]:~/BGP/kernel> make bgp-cn-linux-del-ktau
Reversing the KTAU patch (using a local copy of the applied patch)...
patching file arch/powerpc/kernel/irq.c
patching file arch/powerpc/oprofile/Kconfig
patching file arch/ppc/kernel/entry.S
Hunk #4 succeeded at 287 (offset -6 lines).
patching file arch/ppc/kernel/smp.c
patching file arch/ppc/kernel/time.c
patching file arch/ppc/kernel/traps.c
Hunk #1 succeeded at 69 (offset -7 lines).
patching file arch/ppc/syslib/bgdd/bluegene_network.c
Hunk #2 succeeded at 493 (offset -34 lines).
Hunk #3 succeeded at 505 (offset -34 lines).
patching file include/linux/sched.h
patching file init/main.c
patching file kernel/fork.c
patching file kernel/irq/handle.c
patching file kernel/Makefile
patching file kernel/sched.c
patching file kernel/softirq.c
patching file kernel/timer.c
patching file kernel/workqueue.c
patching file Makefile
NOTE: KTAU integration has been undone (patch reversed and all links removed).
 Next, please reconfigure and build the ZeptoOS CN Linux following standard steps.
 You can "make bgp-cn-linux-menuconfig" and "make bgp-cn-linux-build".

Please remember to reconfigure and rebuild ZeptoOS after undoing the KTAU integration.

TAU

Acquiring TAU

You can get TAU from the downloads page.

Configuring under ZeptoOS

[email protected]:~/tau-2.18.1p2> pwd
/home/anataraj/tau-2.18.1p2

[email protected]:~/tau-2.18.1p2> ./configure -fullhelp | grep Zepto
-zeptodir=<dir> ............................. ZeptoOS install location.

[email protected]:~/tau-2.18.1p2> ./configure -arch=ppc -zeptodir=/home/anataraj/ZeptoReleases/install-BGP -mpi
-------------------- TAU configure script ---------------
  The TAU source code has just been configured to use the
  tau root directory /home/anataraj/tau-2.18.1p2.
  If you move the Tau distribution, you must either
    * set an environment variable TAUROOT containing the new
      location before running any TAU tools
  or
    * run configure again and recompile
-----------------------------------------------------------------
WARNING!! Auto-detect:ppc64 overridden with ppc
-PROFILE and/or -TRACE should be specified. Choosing -PROFILE as the default
Default C++ compiler will be  g++ version 4.1.2 20070115 (prerelease) (SUSE Linux)
Thread-safe MPI interface is defined... 
MPI-2 Interface is defined...
MPI-2 Grequest Interface is defined...
MPI-2 MPI_Datarep_conversion_function Interface is defined...
MPI-2 Comm_create_errhandler Interface is defined...
MPI-2 MPI*attr_functions are defined...
MPI-2 MPI_File functions are defined...
MPI-2 MPI_Type_dup and MPI_Exscan are defined...
MPI-2 MPI_Add_error* functions are defined...
MPI_Status f2c/c2f found...
NOTE: MPI library does not have a threaded _r suffix 
Default C compiler will be /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc
Checking if open takes O_LARGEFILE... yes
Checking for weak symbols... yes
C++ compiler supports Std Runtime Type Information (default)
Testing for Standard C++ Library support for string class
Found support for Standard C++ Library string
NOTE: Compiling Profiling Library and Applications with -DTAU_STDCXXLIB
***********************************************************************
No special modifications found for architecture "ppc"
Checking for bfd.h... no
Checking for shared library compatible libbfd... no
TAU: installing TAU's ParaProf profile browser
jdk version 1.4 or better found
TAU: installed tau_compiler.sh in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed tau_load.sh in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed Java trace reader/writer api in /home/anataraj/tau-2.18.1p2/ppc/lib
TAU: installed tau2slog2, slog2print and jumpshot in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed required slog2sdk jars in /home/anataraj/tau-2.18.1p2/ppc/lib
TAU: installed tauinc.pl in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed TAU's PerfDMF database utilities in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed ParaProf, PerfDMF, PerfExplorer and supporting jars in:
/home/anataraj/tau-2.18.1p2/ppc/lib
Platform does not support JOGL, no 3d displays will be available in ParaProf
TAU: installed tau_analyze in /home/anataraj/tau-2.18.1p2/ppc/bin
TAU: installed Makefile.inc in /home/anataraj/tau-2.18.1p2/tools/src/TauIL
TAU: installed TauIL and supporting jars in /home/anataraj/tau-2.18.1p2/ppc/lib
TAU: installed tau_throttle script
TAU: installing tools in /home/anataraj/tau-2.18.1p2/ppc/bin

NOTE: *** Using ZeptoOS BGP gfortran. ***
NOTE: Enabled Profiling. Compiling with -DPROFILING_ON
NOTE: Using the TAU MPI Profiling Interface
NOTE: Using the TAU MPI_Init_thread Profiling Interface Wrapper
NOTE: Using TAU's MPI-2 extensions
NOTE: Using TAU's MPI-2 Grequest extensions
NOTE: Using TAU's MPI-2 Datarep_conversion extensions
NOTE: Using TAU's MPI-2 Error handler extensions
NOTE: Using TAU's MPI-2 Attr extensions
NOTE: Using TAU's MPI-2 File extensions
NOTE: Using TAU's MPI-2 Type and Exscan extensions
NOTE: Using TAU's MPI-2 Add_error extensions
NOTE: Using default cc compiler.
NOTE: Using the O_LARGEFILE flag to open
NOTE: Using Weak Symbol support
NOTE: Using Standard C++ Library
NOTE: Disabling building of shared objects
Script to modify Makefiles created.
Applying script to all Makefiles, please wait, this may take a while...
.........................................................................................................................................................

Configuring TAU Build scripts...
Modifying Makefiles in the examples subdirectory...
...........

Configuration complete!
   Please add  /home/anataraj/tau-2.18.1p2/ppc/bin  to your path
   Type "make install" to begin compilation

[email protected]:~/tau-2.18.1p2> make install

Using TAU with KTAU : Integrated Profiles

Example Run

---

Top