README for hybrid Linpack, version 1.0
--------------------------------------

 0. Install Cell SDK according to the SDK installation instructions.  This step
    should include install and setup of hybrid DaCS.

 1. Install AMD BLAS.  hpl can be configured to use a variety of BLAS
    implementations.  For hplrr, we've chosen to use AMD's BLAS implementation, 
    the AMD Core Math Library (ACML).  This version of BLAS should be installed
    on the Opteron hosts.  Here are the installation steps:

      - Visit http://developer.amd.com/acml.jsp and download 
        acml-3-6-0-gnu-64bit.tar into /tmp
      - As root, from root's home directory, execute the following commands:

          mkdir acml; cd acml
          cp /tmp/acml-3-6-0-gnu-64bit.tar .
          tar -xvf acml-3-6-0-gnu-64bit.tar
          ./install-acml-3-6-0-gnu-64bit.sh

          echo "/opt/acml3.6.0/gnu64/lib" > /etc/ld.so.conf.d/acml.conf
          ldconfig

      - ACML needs the f2c libraries.  Here are the steps for installing these
        on an F7 system:

          MIRROR=http://www.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/7/Everything/x86_64/os/Fedora
          wget $MIRROR/compat-libf2c-34-3.4.6-7.x86_64.rpm
          rpm -Uvh compat-libf2c-34-3.4.6-7.x86_64.rpm
          cd /usr/lib64
          ln -s libg2c.so.0.0.0 libg2c.so

 2. Install OpenMPI.  Here are the steps to install OpenMPI with TCP/IP as the
    primary transport layer.  (Slightly different configure options are needed
    to use Infiniband as the primary transport layer.) As root:

      wget http://www.open-mpi.org/software/ompi/v1.2/downloads/openmpi-1.2.3.tar.bz2
      tar -xjf openmpi-1.2.3.tar.bz2 
      cd openmpi-1.2.3
      mkdir obj64
      cd obj64
      ../configure --prefix=/usr/local/openmpi64-1.2.3 --disable-mpi-f90 \
                   --enable-ltdl-convenience --with-platform=optimized
      make -j4
      make install

      ln -s /usr/local/openmpi64-1.2.3 /usr/local/openmpi

      echo "/usr/local/openmpi/lib" > /etc/ld.so.conf.d/openmpi.conf 
      ldconfig

      # Create /etc/profile.d/openmpi.sh
      cat > /etc/profile.d/openmpi.sh << EOD
      if ! echo \${PATH} | grep -q /usr/local/openmpi/bin ; then
          PATH=/usr/local/openmpi/bin:\${PATH}
      fi
      EOD
      chmod +x /etc/profile.d/openmpi.sh

      # Now verify the install ...

      # Log out and log back in, then
      echo $PATH        # should contain /usr/local/openmpi/bin

      ompi_info         # should show gobs of stuff

      mpirun -np 2 --host $(hostname) hostname

 3. Grab the HPL 1.0a version of the hpl source from the web

      wget http://www.netlib.org/benchmark/hpl/hpl.tgz
      tar -xzvf hpl.tgz

 4. Patch hpl with changes to add hybrid support and build:

      cd hpl
      patch -p 0 < hpl-hybrid.patch
      export CELL_TOP=/opt/cell/sdk
      make arch=hybrid

 5. Run.  The following steps assume a 4 core host system with 2 attached
    cell blade accelerators, accel1-hostname and accel2-hostname.  Other
    configurations will require changes to the run commands and HPL.dat 
    file.

      # Prep

      # make sure /etc/dacs_topology.config is correct
      scp accel/hpl_accel accel1-hostname:/tmp
      scp accel/hpl_accel accel2-hostname:/tmp
      service hdacsd restart
      ssh root@accel1-hostname service adacsd restart
      ssh root@accel2-hostname service adacsd restart

      # Set up parameters

      The HPL.dat file in hpl/bin/hybrid defines the parameters for a set of tests
      to be executed.  For hybrid hpl, there are some restrictions on the values for
      these settings.  In particular:

      * NB must be 128
      * L1 must be not transposed (1)
      * U must be transposed (0)
      * memory alignment must be 64 (or some multiple of 64)

      # Run (as root)

      cd <path>/hpl/bin/hybrid
      export DACS_START_ENV_LIST=LD_LIBRARY_PATH=/opt/cell/sdk/prototype/usr/lib64
      mpirun -np 4 ./xhpl

=