Enter the Mini-ITX.com online store

Project Browser

September 05, 2017
Choosing the right DC-DC PSU

August 27, 2015
AMD's Project Quantum

August 13, 2015
The Redstone PC is the ultimate Mini-ITX Minecraft Machine

October 09, 2014
The "Restomod TV"

April 09, 2013
Installing NAS4Free

February 28, 2013
Building an XBMC 12 Home Theatre PC

January 25, 2011
XBMC Guide updated to version 10.0

August 06, 2010
Building a Green PC

February 15, 2010
Building an ION powered HTPC with XBMC

October 10, 2008
The "Cambridge Autonomous Underwater Vehicle 2008"

Mini-ITX Online Store

September 12, 2008
"Florian", the DVD burning robot

September 05, 2008
The "i-EPIA"

May 22, 2008
The "GTA-PC"

April 14, 2007
The "Digg" Case

January 19, 2007
The "ITX-Laptop"

December 07, 2006
The "Tortoise Beetle"

October 02, 2006
The "DOS Head Unit"

August 31, 2006
The "Janus Project"

August 05, 2006
The "Leela PC"

June 26, 2006
Nano-ITX in a Football

May 17, 2006
The "EPIA Alloy Mod"

April 11, 2006
Neatorama's Collection of Case Mods

February 18, 2006
The "Rundfunker"

October 24, 2005
The "ITX TV"

October 06, 2005
The K'nex-ITX

August 05, 2005
The "Waffle Iron PC"

July 21, 2005
The "Supra-Server"

July 18, 2005
The "Mega-ITX"

July 07, 2005
The "Encyclomedia"

May 25, 2005
The "Accordion ITX"

Mini-ITX Online Store

May 16, 2005
The "FileServerRouterSwitch"

May 15, 2005
The "Mini Falcon"

May 13, 2005
The "Bender PC"

May 11, 2005
The "BBC ITX B"

May 10, 2005
The "Frame"

April 20, 2005
The "Jeannie"

March 09, 2005
The "Cool Cube"

January 30, 2005
First Nano-ITX Project?

January 17, 2005
The "iGrill"

January 15, 2005
The "Gumball PC"

December 15, 2004
The "Deco Box"

December 03, 2004
The "TERA-ITX"

October 06, 2004
The "Coealacanth-PC"

September 17, 2004
The "Gramaphone-ITX-HD"

August 26, 2004
The "C1541 Disk Drive ITX"

August 25, 2004
The "SEGA-ITX"

August 13, 2004
The "Quiet Cubid"

August 06, 2004
The "BMWPC"

July 14, 2004
The "Moo Cow Moo"

July 02, 2004
The "Mini Mesh Box"

Full alphabetical archive on right hand side of page...


The "Mini-Cluster"
By Glen Gardner, USA - Posted on 25 February, 2004

Software Configuration

The cluster consists of a controlling node, with a large capacity hard drive, and several computational nodes, each with their own hard disk drive (these hard drives can be smaller).

The software which performs the parallelization (MPI) is installed on the controlling node, and the computational nodes mount a shared directory on the controlling node via NFS.

Communications between the nodes is established via rsh by MPI, and shared files are found via the mounted NFS file system,

The networking is fast ethernet (100 Mbit) and makes use of a fast ethernet switch switch. Gigbit ethernet is faster (and better for fast file I/O) but 100 Mbit ethernet is quite adequate for number crunching.

The version of MPI used is mpich-1.2.5.2

The Operating system for the controlling node and all the computational nodes is FreeBSD MINI 4.8 RELEASE

FreeBSD has moved forward a bit since I began building my cluster, so check with freebsd.org to see what is currently available. Whatever distribution you use, you should be using RELEASE or STABLE versions.

Install and configure the controlling node

Keep it simple. Resist the temptation to add a lot of options. JUST MAKE IT WORK.

Keep all the nodes as identical as possible, they will be running code that is generated on the controlling node.

Setup a firewall between the cluster and the outside world. The cluster needs a high degree of connectivity and has rather poor security.

Assemble the nodes and test them one at a time.

Install Mini-FBSD on the controlling node first (I'm using the mini 4.8 distribution).

Use the same root password on the controlling node and on all the computational nodes.

Configure the controlling node as an NFS server and export /usr to be accessed with root privileges.

Enable inetd, and edit /etc/inetd.conf to allow rlogin.

Setup rsh and ssh so that the controlling node and computational nodes can access each other.

Be sure to edit /etc/ssh/sshd_config to allow root login.

DO NOT allow the controlling node to rsh/ssh to itself. Doing this will not only cause security issues, but can lead to the controlling node getting saturated with rsh connections during a program run , and can cause slowness and program crashes.

Allow only essential external computers to access the controlling node by ssh. Do not allow any external computers to use rsh to access any node. Use ssh instead.

Edit /etc/rc.conf for the appropriate hostname and ip address.

Edit /etc/hosts to include the hostnames and ip's of the controlling node, computational nodes, and any external computers which need to access the controlling node.

Download and install MPI. Be sure to read the documentation on the MPI web site. Install MPI in /usr/local/mpi. I built MPI to run in P_4 mode to keep things simple.

In '/root/.cshrc' add '/usr/local/mpi/bin' to the path. You might also wish to edit '/etc/skel/.cshrc' with the same value so that new users get a working MPI.

Install FBSD on one computation node

Configure it as an nfs client.

Enable inetd and edit /etc/inetd.conf to allow rlogin

Edit '/etc/fstab' to add the nfs mount for /usr and set the mount point as /mnt/usr . Create a symbolic link at /usr/local/mpi that points to /mnt/usr/local/mpi

Add the hostnames and ip addresses for the controlling nodes and all the computational nodes to /etc/hosts

Edit /etc/rc.conf for the appropriate hostname and IP address for the node.

Edit /etc/ssh/ssh_config to configure the node as an ssh client.

Use rcp/scp to copy the /etc/ssh/sshd_config file from the controlling node to the computational node.

Create an empty file with the name of '.hushlogin' and put it in '/root'. You may wish to also put .hushlogin in /etc/skel so new users automatically get a copy of it. This inhibits motd and limits the login text to a prompt. It serves to keep mpi from complaining about getting an unexpected response when it uses rsh to connect to a node.

You may need to have .rhosts in /root, be sure to include all nodes in this, if you use it. You might wish to put a copy of .rhosts in /etc/skel so that new users can use ssh/rsh without being root.

You will need to add each node to '/usr/local/mpi/share/machines.freebsd '. This file is the list of nodes usable by MPI.

Run the test script /usr/locall/mpi/sbin/tstmachines with the -v option. 'sh /usr/local/mpi/sbin/tstmachines -v" It may complain that it can not access the controlling node (this is normal), but it should talk to all the nodes in the nodelist and run some test software to confirm that all is working. The script uses rsh to talk to all the nodes, and if the controlling node cannot rsh to itself, the script will complain. Resist the temptation to allow the controlling node to rsh to itself. MPI will run a process on localhost in addition to any nodes listed in '/usr/local/mpi/share/machines.freebsd', so even if the script complains that it can't find the controlling node, mpi will still work.

Compile and run some of the sample programs that come with mpi to confirm that all is working properly.

Copy the newly configured node to an "empty" hard drive.

If all is well, connect an empty hard drive for the next node to the secondary controller and use dd to copy the configured hard drive to the empty one. Be sure the "empty" drive is configured as slave and does not contain a primary partition. or FBSD might not know what to do with two hard drives at the same time.

Shut down the computer and remove the copied drive and install it in the second node. Don't forget to move the jumper from slave to master.

Configure the new node by booting it, and logging in from a keyboard, and editing /etc/rc.conf for the appropriate hostname and ip address.

Add the new node to '/usr/local/mpi/share/machines.freebsd' on the controlling node.

Reboot the new node and rsh to it from the controlling node to confirm communications.

Run /usr/local/mpi/sbin/tstmachines in verbose mode to assure the new node works properly.

If the new node is working properly, use dd to install copies of the computational node on all the drives for the remaining cluster nodes.

Testing

Plan for some odd things to happen. Clustering has a way of exposing "flaky" hardware and software. Usually if a node crashes frequently for no apparent reason , you might want to consider it as having potential hardware problems.

Power up the new cluster and let it idle for a day or two, and check the nodes to see if they spontaneously crash, disconnect, or otherwise misbehave. If the cluster seems stable, you need to begin writing programs designed to stress the machine so that you can expose software bugs, and latent hardware issues. Work through these issues one at a time. Depending on the hardware, the size of the cluster. and it's complexity, it could take from a few weeks to several months to weed out the worst of the quirks and bugs. Replace flaky hardware. One bad node in the nodelist can render a cluster useless, so don't waste your time and money trying to limp along with wounded hardware.

Operation

Power it up and leave it up. Cycle the power on a node only when you absolutely must. This reduces failures from inrush currents at power-up as well as reducing thermomechanical stresses that lead to component failures.

Development

You might wish to set aside a node for development, so you can test new kernels or software. Once you are sure your new code is stable, you can migrate it to the other nodes. Exclude this node from the nodelist so the users don't get unhappy surprises when they run their software.

Maintenance

Plan on having about ten percent of the cluster failed or failing at any given time. If you need a machine with 10 nodes operational, you had best plan on having 12 nodes, and some spare parts. The larger the cluster is, the more failed hardware you can expect. Really large clusters have hardware failures on a more or less continuous basis. Alternatively, you can just build a lot of extra nodes and take bad nodes offline as the cluster "burns in" (this seems expensive and wasteful to me). Run the cluster on a good UPS. It is not an option. You need clean power to get good hardware life, and with this many computers the investment in a UPS will pay off in terms of longer hardware life.

Lifespan

Consumer grade electronics is designed with an operational life of two years. Lower quality components have an even shorter design life. This means that once you get all the bugs worked out, and everything is "burned in" you can expect a year or two of fairly trouble-free service. After that, the components age sufficiently that you will begin to see hardware failures rising to the point that you probably will want to consider just building a new machine.

Final words

Building a parallel computing machine is a big investment in time and money. Take your time and plan your project carefully. Make sure all of the components you plan to use are available, and will continue to be available over the several months it is likely to take you to build and test your creation. A little thought will save you a lot in terms of time, money and disappointment, and will pay big dividends in satisfaction.

Useful links

The MPI Home Page. You can download the latest distribution of MPI as well as useful documentation.

The FreeBSD Home Page. Download your favorite distribution of FreeBSD and browse online documentation.



Board Finder
Case Finder
Mini PC Finder
Quick Links
Mini-ITX Online Store

Mailing Lists:
Mini-ITX Store

Mini-ITX 101
Mini-ITX History
Advertising

Projects:

Show Random
How to submit
your project

Most Viewed Today

XBMC-ION

ITX-Laptop

Mini-Cluster

NAS4Free

Ammo Box

Accordion-ITX
Aircraft Carrier
Ambulator 1
AMD Case
Ammo Box
Ammo Tux
AmmoLAN
amPC
Animal SNES
Atari 800 ITX
Attache Server
Aunt Hagar's Mini-ITX
Bantam PC
BBC ITX B
Bender PC
Biscuit Tin PC
Blue Plate
BlueBox
BMW PC
Borg Appliance
Briefcase PC
Bubbacomp
C1541 Disk Drive
C64 @ 933MHz
CardboardCube
CAUV 2008
CBM ITX-64
Coelacanth-PC
Cool Cube
Deco Box
Devilcat
DOS Head Unit
Dreamcast PC
E.T.PC
Eden VAX
EdenStation IPX
Encyclomedia
Falcon-ITX
Florian
Frame
FS-RouterSwitch
G4 Cube PC
GasCan PC
Gingerbread
Gramaphone-ITX-HD
GTA-PC
Guitar PC
Guitar Workstation
Gumball PC
Hirschmann
HTPC
HTPC2
Humidor 64
Humidor CL
Humidor II
Humidor M
Humidor PC
Humidor V
I.C.E. Unit
i64XBOX
i-EPIA
iGrill
ITX Helmet
ITX TV
ITX-Laptop
Jeannie
Jukebox ITX
KiSA 444
K'nex ITX
Leela PC
Lego 0933 PC
Legobox
Log Cabin PC
Lunchbox PC
Mac-ITX
Manga Doll
Mantle Radio
Mediabox
Mega-ITX
Micro TV
Mini Falcon
Mini Mesh Box
Mini-Cluster
Mobile-BlackBox
Moo Cow Moo
Mr OMNI
NAS4Free
NESPC
OpenELEC
Osh Kosh
Pet ITX
Pictureframe PC
Playstation 2 PC
Playstation PC
Project NFF
PSU PC
Quiet Cubid
R2D2PC
Racing The Light
RadioSphere
Restomod TV
Robotica 2003
Rundfunker
SaturnPC
S-CUBE
SEGA-ITX
SpaceCase
SpacePanel
Spartan Bluebird
Spider Case
Supra-Server
Teddybear
Telefunken 2003
TERA-ITX
The Clock
ToAsTOr
Tortoise Beetle
Tux Server
Underwood No.5
Waffle Iron PC
Windows XP Box
Wraith SE/30
XBMC-ION

How to submit
your project

Reviews:
CF-S688 E-Note
Cubid 2677R
Cubid 2688R
Cubid 3688
GAlantic GA610i
Hush Mini-ITX
Lian Li PC-402A
Jetway B860T
VIA M 10000
VIA MII 12000
VIA Nano-ITX
VIA Pico-ITX
Sigma XCard
Travla C137

Guides & Tips:
5.1 EPIA Audio
Cubid Tips
EPIA CL Firewall
EPIA COM IR
EPIA SCART
Extra USB Ports
IPCop Gateway
Overclocking
PowerLCD

Drivers:
EPIA  EPIA V
EPIA M  EPIA MII
EPIA CL  EPIA PD
EPIA TC
.

Mini-ITX Online Store

Contact Us

Store: +44 (0) 845 475 8 475

Store enquiries: store@mini-itx.com

Other enquiries: feedback@mini-itx.com

Visit the Store

Click here to enter the online store

Social

Follow us on Twitter!

Join our Mailing List

Copyright: All content on this site is Copyright © 2002-2017 Mini-ITX.com and respective owners, all rights reserved.