Enter the Mini-ITX.com online store

Project Browser

September 05, 2017
Choosing the right DC-DC PSU

August 27, 2015
AMD's Project Quantum

August 13, 2015
The Redstone PC is the ultimate Mini-ITX Minecraft Machine

October 09, 2014
The "Restomod TV"

April 09, 2013
Installing NAS4Free

February 28, 2013
Building an XBMC 12 Home Theatre PC

January 25, 2011
XBMC Guide updated to version 10.0

August 06, 2010
Building a Green PC

February 15, 2010
Building an ION powered HTPC with XBMC

October 10, 2008
The "Cambridge Autonomous Underwater Vehicle 2008"

Mini-ITX Online Store

September 12, 2008
"Florian", the DVD burning robot

September 05, 2008
The "i-EPIA"

May 22, 2008
The "GTA-PC"

April 14, 2007
The "Digg" Case

January 19, 2007
The "ITX-Laptop"

December 07, 2006
The "Tortoise Beetle"

October 02, 2006
The "DOS Head Unit"

August 31, 2006
The "Janus Project"

August 05, 2006
The "Leela PC"

June 26, 2006
Nano-ITX in a Football

May 17, 2006
The "EPIA Alloy Mod"

April 11, 2006
Neatorama's Collection of Case Mods

February 18, 2006
The "Rundfunker"

October 24, 2005
The "ITX TV"

October 06, 2005
The K'nex-ITX

August 05, 2005
The "Waffle Iron PC"

July 21, 2005
The "Supra-Server"

July 18, 2005
The "Mega-ITX"

July 07, 2005
The "Encyclomedia"

May 25, 2005
The "Accordion ITX"

Mini-ITX Online Store

May 16, 2005
The "FileServerRouterSwitch"

May 15, 2005
The "Mini Falcon"

May 13, 2005
The "Bender PC"

May 11, 2005

May 10, 2005
The "Frame"

April 20, 2005
The "Jeannie"

March 09, 2005
The "Cool Cube"

January 30, 2005
First Nano-ITX Project?

January 17, 2005
The "iGrill"

January 15, 2005
The "Gumball PC"

December 15, 2004
The "Deco Box"

December 03, 2004

October 06, 2004
The "Coealacanth-PC"

September 17, 2004
The "Gramaphone-ITX-HD"

August 26, 2004
The "C1541 Disk Drive ITX"

August 25, 2004

August 13, 2004
The "Quiet Cubid"

August 06, 2004

July 14, 2004
The "Moo Cow Moo"

July 02, 2004
The "Mini Mesh Box"

Full alphabetical archive on right hand side of page...

The "Mini-Cluster"
By Glen Gardner, USA - Posted on 25 February, 2004

Software Configuration

The cluster consists of a controlling node, with a large capacity hard drive, and several computational nodes, each with their own hard disk drive (these hard drives can be smaller).

The software which performs the parallelization (MPI) is installed on the controlling node, and the computational nodes mount a shared directory on the controlling node via NFS.

Communications between the nodes is established via rsh by MPI, and shared files are found via the mounted NFS file system,

The networking is fast ethernet (100 Mbit) and makes use of a fast ethernet switch switch. Gigbit ethernet is faster (and better for fast file I/O) but 100 Mbit ethernet is quite adequate for number crunching.

The version of MPI used is mpich-

The Operating system for the controlling node and all the computational nodes is FreeBSD MINI 4.8 RELEASE

FreeBSD has moved forward a bit since I began building my cluster, so check with freebsd.org to see what is currently available. Whatever distribution you use, you should be using RELEASE or STABLE versions.

Install and configure the controlling node

Keep it simple. Resist the temptation to add a lot of options. JUST MAKE IT WORK.

Keep all the nodes as identical as possible, they will be running code that is generated on the controlling node.

Setup a firewall between the cluster and the outside world. The cluster needs a high degree of connectivity and has rather poor security.

Assemble the nodes and test them one at a time.

Install Mini-FBSD on the controlling node first (I'm using the mini 4.8 distribution).

Use the same root password on the controlling node and on all the computational nodes.

Configure the controlling node as an NFS server and export /usr to be accessed with root privileges.

Enable inetd, and edit /etc/inetd.conf to allow rlogin.

Setup rsh and ssh so that the controlling node and computational nodes can access each other.

Be sure to edit /etc/ssh/sshd_config to allow root login.

DO NOT allow the controlling node to rsh/ssh to itself. Doing this will not only cause security issues, but can lead to the controlling node getting saturated with rsh connections during a program run , and can cause slowness and program crashes.

Allow only essential external computers to access the controlling node by ssh. Do not allow any external computers to use rsh to access any node. Use ssh instead.

Edit /etc/rc.conf for the appropriate hostname and ip address.

Edit /etc/hosts to include the hostnames and ip's of the controlling node, computational nodes, and any external computers which need to access the controlling node.

Download and install MPI. Be sure to read the documentation on the MPI web site. Install MPI in /usr/local/mpi. I built MPI to run in P_4 mode to keep things simple.

In '/root/.cshrc' add '/usr/local/mpi/bin' to the path. You might also wish to edit '/etc/skel/.cshrc' with the same value so that new users get a working MPI.

Install FBSD on one computation node

Configure it as an nfs client.

Enable inetd and edit /etc/inetd.conf to allow rlogin

Edit '/etc/fstab' to add the nfs mount for /usr and set the mount point as /mnt/usr . Create a symbolic link at /usr/local/mpi that points to /mnt/usr/local/mpi

Add the hostnames and ip addresses for the controlling nodes and all the computational nodes to /etc/hosts

Edit /etc/rc.conf for the appropriate hostname and IP address for the node.

Edit /etc/ssh/ssh_config to configure the node as an ssh client.

Use rcp/scp to copy the /etc/ssh/sshd_config file from the controlling node to the computational node.

Create an empty file with the name of '.hushlogin' and put it in '/root'. You may wish to also put .hushlogin in /etc/skel so new users automatically get a copy of it. This inhibits motd and limits the login text to a prompt. It serves to keep mpi from complaining about getting an unexpected response when it uses rsh to connect to a node.

You may need to have .rhosts in /root, be sure to include all nodes in this, if you use it. You might wish to put a copy of .rhosts in /etc/skel so that new users can use ssh/rsh without being root.

You will need to add each node to '/usr/local/mpi/share/machines.freebsd '. This file is the list of nodes usable by MPI.

Run the test script /usr/locall/mpi/sbin/tstmachines with the -v option. 'sh /usr/local/mpi/sbin/tstmachines -v" It may complain that it can not access the controlling node (this is normal), but it should talk to all the nodes in the nodelist and run some test software to confirm that all is working. The script uses rsh to talk to all the nodes, and if the controlling node cannot rsh to itself, the script will complain. Resist the temptation to allow the controlling node to rsh to itself. MPI will run a process on localhost in addition to any nodes listed in '/usr/local/mpi/share/machines.freebsd', so even if the script complains that it can't find the controlling node, mpi will still work.

Compile and run some of the sample programs that come with mpi to confirm that all is working properly.

Copy the newly configured node to an "empty" hard drive.

If all is well, connect an empty hard drive for the next node to the secondary controller and use dd to copy the configured hard drive to the empty one. Be sure the "empty" drive is configured as slave and does not contain a primary partition. or FBSD might not know what to do with two hard drives at the same time.

Shut down the computer and remove the copied drive and install it in the second node. Don't forget to move the jumper from slave to master.

Configure the new node by booting it, and logging in from a keyboard, and editing /etc/rc.conf for the appropriate hostname and ip address.

Add the new node to '/usr/local/mpi/share/machines.freebsd' on the controlling node.

Reboot the new node and rsh to it from the controlling node to confirm communications.

Run /usr/local/mpi/sbin/tstmachines in verbose mode to assure the new node works properly.

If the new node is working properly, use dd to install copies of the computational node on all the drives for the remaining cluster nodes.


Plan for some odd things to happen. Clustering has a way of exposing "flaky" hardware and software. Usually if a node crashes frequently for no apparent reason , you might want to consider it as having potential hardware problems.

Power up the new cluster and let it idle for a day or two, and check the nodes to see if they spontaneously crash, disconnect, or otherwise misbehave. If the cluster seems stable, you need to begin writing programs designed to stress the machine so that you can expose software bugs, and latent hardware issues. Work through these issues one at a time. Depending on the hardware, the size of the cluster. and it's complexity, it could take from a few weeks to several months to weed out the worst of the quirks and bugs. Replace flaky hardware. One bad node in the nodelist can render a cluster useless, so don't waste your time and money trying to limp along with wounded hardware.


Power it up and leave it up. Cycle the power on a node only when you absolutely must. This reduces failures from inrush currents at power-up as well as reducing thermomechanical stresses that lead to component failures.


You might wish to set aside a node for development, so you can test new kernels or software. Once you are sure your new code is stable, you can migrate it to the other nodes. Exclude this node from the nodelist so the users don't get unhappy surprises when they run their software.


Plan on having about ten percent of the cluster failed or failing at any given time. If you need a machine with 10 nodes operational, you had best plan on having 12 nodes, and some spare parts. The larger the cluster is, the more failed hardware you can expect. Really large clusters have hardware failures on a more or less continuous basis. Alternatively, you can just build a lot of extra nodes and take bad nodes offline as the cluster "burns in" (this seems expensive and wasteful to me). Run the cluster on a good UPS. It is not an option. You need clean power to get good hardware life, and with this many computers the investment in a UPS will pay off in terms of longer hardware life.


Consumer grade electronics is designed with an operational life of two years. Lower quality components have an even shorter design life. This means that once you get all the bugs worked out, and everything is "burned in" you can expect a year or two of fairly trouble-free service. After that, the components age sufficiently that you will begin to see hardware failures rising to the point that you probably will want to consider just building a new machine.

Final words

Building a parallel computing machine is a big investment in time and money. Take your time and plan your project carefully. Make sure all of the components you plan to use are available, and will continue to be available over the several months it is likely to take you to build and test your creation. A little thought will save you a lot in terms of time, money and disappointment, and will pay big dividends in satisfaction.

Useful links

The MPI Home Page. You can download the latest distribution of MPI as well as useful documentation.

The FreeBSD Home Page. Download your favorite distribution of FreeBSD and browse online documentation.

Board Finder
Case Finder
Mini PC Finder
Quick Links
Mini-ITX Online Store

Mailing Lists:
Mini-ITX Store

Mini-ITX 101
Mini-ITX History


Show Random
How to submit
your project

Most Viewed Today

Bender PC

Manga Doll

Cool Cube



Aircraft Carrier
Ambulator 1
AMD Case
Ammo Box
Ammo Tux
Animal SNES
Atari 800 ITX
Attache Server
Aunt Hagar's Mini-ITX
Bantam PC
Bender PC
Biscuit Tin PC
Blue Plate
Borg Appliance
Briefcase PC
C1541 Disk Drive
C64 @ 933MHz
CAUV 2008
Cool Cube
Deco Box
DOS Head Unit
Dreamcast PC
Eden VAX
EdenStation IPX
G4 Cube PC
GasCan PC
Guitar PC
Guitar Workstation
Gumball PC
Humidor 64
Humidor CL
Humidor II
Humidor M
Humidor PC
Humidor V
I.C.E. Unit
ITX Helmet
Jukebox ITX
KiSA 444
K'nex ITX
Leela PC
Lego 0933 PC
Log Cabin PC
Lunchbox PC
Manga Doll
Mantle Radio
Micro TV
Mini Falcon
Mini Mesh Box
Moo Cow Moo
Osh Kosh
Pictureframe PC
Playstation 2 PC
Playstation PC
Project NFF
Quiet Cubid
Racing The Light
Restomod TV
Robotica 2003
Spartan Bluebird
Spider Case
Telefunken 2003
The Clock
Tortoise Beetle
Tux Server
Underwood No.5
Waffle Iron PC
Windows XP Box
Wraith SE/30

How to submit
your project

CF-S688 E-Note
Cubid 2677R
Cubid 2688R
Cubid 3688
GAlantic GA610i
Hush Mini-ITX
Lian Li PC-402A
Jetway B860T
VIA M 10000
VIA MII 12000
Sigma XCard
Travla C137

Guides & Tips:
5.1 EPIA Audio
Cubid Tips
EPIA CL Firewall
Extra USB Ports
IPCop Gateway


Mini-ITX Online Store