
 distributed.net client for the PlayStation 3/Cell Broadband Engine
 document revision $Id: readme.cell,v 1.7 2009/05/20 18:17:02 stream Exp $

 Welcome to the distributed.net client.

 This document covers information specific to the Linux client for the Cell
 processor. Refer to other enclosed documentation or browse the online FAQ
 at <http://faq.distributed.net/> for non-platform-specific documentation.

    1.0  Getting started
    2.0  PS3/Cell BE specific notes
    3.0  Troubleshooting and Known issues

 1.0  Getting started ------------------------------------------------
 
    This client runs under the Linux operating system. If you haven't done  
    so already, you will need to install Linux on your PlayStation 3. The
    procedure for installing alternative operating systems is documented
    by Sony at <http://www.playstation.com/ps3-openplatform/>.
    You'll need a Linux distribution for the PowerPC architecture, with a
    reasonably recent kernel offering support for the Cell processor. At the 
    present time, Fedora Core 6 distribution with kernel 2.6.16 and Ubuntu
    Gutsy with kernel 2.6.22 has been tested.

    Just unpack/unzip/untar the client in a directory of your choice and 
    fire it up.

    If you have never run the client before, it will initiate the
    menu-driven configuration. Save and quit when done.

    Then, simply restart the client. From that point on it will use the 
    saved configuration.
    
    The configuration options are fairly self-explanatory and can be run
    at any time by starting the client with the '-config' option.
    A list of command line options is available by starting the client 
    with '-help' or '--help'.

    A complete step-by-step guide to running your first client is 
    available at <http://www.distributed.net/docs/tutor_clients.php>

 2.0  PS3/Cell BE specific notes ------------------------------------

    The PlayStation 3's processor, Cell, was designed by the STI
    (Sony/Toshiba/IBM) joint venture. It is composed of a PowerPC main
    processor (PPE) and up to 8 co-processors, called Synergistic Processing
    Elements or SPEs, all clocked at 3.2 GHz. (SPU is a synonym to SPE,
    meaning Synergistic Processing Unit). As shipped in the
    PlayStation 3, Cell has one of its SPEs disabled, so that only 7 SPEs
    are accessible to most games. Moreover, when running an alternative
    operating system such as Linux, the hypervisor reserves one of the SPEs,
    so only 6 SPEs are accessible to applications running under Linux. Note
    that this issue is specific to the PlayStation 3; blade servers using
    Cell processors should have all 8 SPEs available.   

    We ship optimized cores for the RC5-72 and OGR projects. For RC5-72,
    there are two cores available, "Cell v1 PPE" and "Cell v1 SPE". These
    run on the main processor (PPE) and co-processors (SPEs) respectively.
    As for OGR and OGR-NG, there are three cores available, the two KOGE
    cores from the PowerPC client port, which run on the PPE, and a new
    core, "Cell v1 SPE", which runs on the SPEs. Automatic core selection
    should select the fastest core for the PPE, namely "KOGE 2.0 Hybrid".

    As the current client architecture doesn't support separate core choices
    for different processors, a temporary workaround has been implemented   
    whereby the first crunchers will use the core selected in the config
    file, and further crunchers will always use the "Cell v1 SPE" core. The 
    client will auto-detect the number of physical Cell processors and start 
    1 PPE core for each physical processor and 1 SPE core for each of the 
    usable SPE processors detected.

    It is also possible to manually specify any combination of PPE and SPE
    cores on Cell hardware as follows:

        a) [X] PPE cores and [n] SPE cores (n > 0): select the "Cell v1 PPE"
           core and run [X+n] crunchers simultaneously;
        b) [m] PPE cores and [0] SPE core (m <= X): select the "Cell v1 PPE" 
           core and run [m] crunchers simultaneously;
        c) [0] PPE cores and [n] SPE cores (n > 0): select the "Cell v1 SPE"
           core and run [n] crunchers simultaneously;
        d) [m] PPE cores and [n] SPE cores (m < X; n > 0): run two
           clients using both (b) and (c)

        where X = number of physical processors         

    Note that although Linux reports 2 CPUs per Cell, this is due to the PPE
    being a 2-way simultaneous multi-threading (SMT) design, roughly similar
    to the Hyperthreading technology used in Intel's Pentium 4 CPUs. This
    means the operating system sees each CPU as 2 distinct CPUs, but
    physically these CPUs share the majority of execution resources and
    generally cannot perform at twice the rate of a single CPU. Moreover,
    since our cores already exploit all available PPE execution resources in
    a single cruncher, while relying on precise instruction scheduling, it
    has been observed that running multiple PPE crunchers simultaneously on
    the same physical PPE has a negative effect on performance. Therefore,
    in conjunction with the workaround above for SPE cores, running multiple
    PPE crunchers in a single Cell is disabled in this client.

 3.0  Troubleshooting and Known issues ------------------------------


  3.1   Missing SPU support -----------------------------------------

    Some CellBE Linux distributions (such as old Ubuntu Linux) may receive
    errors such as:
        "spu_create(): Function not implemented" 
        "spu_create(): no such file or directory"

    To have the spu_create() system call working, the spufs kernel
    module must be loaded and the filesystem of the same name must be
    mounted.  This can be done with the following steps:

        echo "none /spu spufs defaults 0 0" >> /etc/fstab
        mkdir /spu
        mount /spu


  3.2   Bugs in SPU scheduler ----------------------------------------

    SPU scheduler, spufs, it a new and very complicated part of linux
    kernel. It's like a kernel inside kernel. Unfortunately, this mean
    that lot of bugs still exist in it. A distributed.net client, which
    fully utilizes all available SPU resources, is very vulnerable to
    these bugs.

    We found that some versions of kernel have critical bug when preemptive
    scheduling of SPU threads. It appears only when number of SPU threads
    in all processes is more then number of physical SPUs. For example,
    if your system is running a single copy of distributed.net client,
    everything will be fine. But if you'll run another program which uses SPU
    (e.g. SPU-accelerated mplayer - "mplayer-ps3", or even a second copy of
    distributed.net client in "-test" or "-bench" mode), bug will eventually
    appear.

    This bug lead to random corruption of memory area used by SPU core. You
    may notice that client quickly completes lot of extremely short stubs -
    usually with length of 0 stat units.

    Speaking technically, look like spufs incorrectly saves DMA state during
    rescheduling. SPU program continues to run thinking that DMA transfer was
    complete. Really, it was not, and memory which should contain core work
    data really contains a garbage.

    Since version 2.9105.511-b2, client is able to detect this problem.
    Signatures in the beginning and end of work are used to check that all
    data really has been transferred to SPU. If not, client will print
    messages like this (the exit code and reason may differ):

        OGRNG-SPE99: non-zero exit code (66) from SPU thread!
        OGRNG-SPE99: possible reason: passed bad tail signature
        This may be caused by kernel bug in SPU scheduler (spufs)
        See readme.cell for more information and possible solutions.

    Affected cruncher process will be terminated (we cannot trust it after
    memory corruption). Since number of SPE threads will be decreased, your
    system will work stable again, but without one cruncher, meaning lesser
    key/node rate. Affected stub will be saved in last known good state
    during termination or restart of client, so it's progress will be not lost.

    If your kernel is affected and you've seen an error message described
    above, first we recommend to upgrade, if possible, either your linux 
    distribution either kernel only (see notes on kernel versions below).

    If upgrade is not possible, we suggest to avoid usage of other SPU-enabled
    programs (like mplayer-ps3) while client is running.

    Advanced users may decrease number of crunchers as described in section
    2.0. Select number of crunchers = 6 and client will run 1 PPE plus 5 SPE
    threads. One SPE will be free for other tasks. It's possible to create
    a special configuration or command file to run this "light" client when
    you need to use other SPE-enabled program.
    
    Please remember that even if client can detect this bug and recover, 
    another application could be affected too with unpredictable results.


    Unfortunately it's not known exactly which kernel versions are affected.
    Kernel version number may be meaningless because latest version of
    PS3-specific parts could be taken directly from developer's repositories
    no matter which version of mainstream kernel is used. The information
    below is a result of my own experiments.

    Ubuntu 7.10 Gutsy, kernel 2.6.22-16 - seems to work, never seen any bogus
    results although I've used mplayer-ps3 very often.

    Ubuntu 8.04 Hardy, kernel 2.6.24-24 - probably affected but not tested
    because whole system is unusable (kernel detects only 96 Mb of memory).

    Ubuntu 8.10 Intrepid, kernel 2.6.25-2 - affected.

    It's expected that systems based on kernel 2.6.26 like Ubuntu Jaunty will
    be not affected. At least official changelog between kernels 2.6.25 and
    2.6.26 contains about 10 spufs bugfixes, and almost each of these errors
    may lead to bug described here. But it wasn't tested yet.
