Cellular automata were first introduced by John von Neumann and Stanislaw Ulam in the 1940s, and gradually used to solve a wide range of problems, including multicellular biological modeling in which a natural correspondence between each automaton cell and each biological cell is assumed 18192021. Simple local interaction producing complex global behavior is common to a variety of natural phenomena, including cardioelectrical activity. Though traditionally cellular automata are regarded as discrete parallel systems, new functions can be obtained with non-standard implementations 2223. For language-based cellular automata, a program encoded in a language instead of a rule is shared by all cells and describes the behavior of each cell. A compiler translates the cell program into executable files. To enhance the portability of the language, a two-step compilation is usually adopted, with C/C++ files being the intermediate codes, allowing the extension of such cellular automata systems.

Cellular is a cellular automata system based on the language Cellang 23, whose cell program comprises three parts: constant declarations, a cell declaration, and statements. The cell declaration defines a set of fields to store states of each cell between successive steps of computation. The cell array is described in a separate data file, whose output is piped to the cell program to provide the value of fields in each cell so as to enable computation. A predefined unchangeable variable time synchronizes the running of all cells. By conventions, in cellular automata computation quantitative computing is not fully supported and function call is not allowed. To overcome this inherent inadequacy of Cellang in quantitative computing and enable it to solve HH type membrane equations, we have added new language facilities, including the floating-point data type and the mathematical functions provided in the C libraries. Thus, Cellang is able to encode numerical solutions of the HH equations (Figure 1). A built-in function position() is also added to specify the global coordinates of the currently running cell. This function, in combination with the if-then statement, allows position-and time-dependent runtime perturbations to any cells, a function valuable for arrhythmia simulation. Viewing facilities are also extended to monitor and display simulation of electrical activitys at channel, cell, and organ level.

Running a model may be no easier than building it. The large number of cells, the small time step Δt and the long running time for arrhythmia simulation make it impractical to run a whole heart model on any low-end computer. Since large-scale SMP machines are extremely expensive, the prevailing parallel computing platforms are clusters of small-scale SMP or DMP, and the protocols of programming in Fortran, C, and C++ on such computers are OpenMP and MPI. First, an endless while loop is used to realize iterative computation. Second, since for an n-dimensional model each field in the cell program is translated into an n-dimensional data array in the intermediate C program, within the while loop n successive for loops are employed to traverse the n dimensional cell space. Only the codes within the for loops, which are statements in the cell program, need to be parallelized. Instead of defining one data array with an offset for each field in the original Cellular system, we use two data arrays in a flip-flop manner to support parallel write operation. OpenMP provides a group of directives to be inserted into the C program to tell the compiler the region to be executed in parallel. Such directives appear before the for loops to dispatch the outermost for loop into a group of threads, whose number is set dynamically according to the available CPUs. The simulation of cellular activity is thus implicitly parallelized in a shared memory space. To parallelize a Cellang program on a platform with distributed memory, explicit cell space decomposition is inevitable. Every data array is equally divided into several subsets located in distributed memory bodies, and computed by autonomous computing nodes that communicate each other through MPI. Since a cell needs to access its neighboring cells over the maximal radius m to compute transjunctional currents, each subset should enclose m extra layers of cells on each side to ensure that cells at the boundary layer(s) can access correct neighbors that are located in the outermost m layers of the two neighboring subsets, respectively. Field values of cells in these extra layers should be swapped between neighboring subsets after each round of computation to ensure that cells access updated data. Such data exchange is the major factor for the excessive time needed by cellular automata models running on distributed memory platforms. If cells have many fields, the time spent on such operations may even offset the benefits of parallelism.

To reach the maximal flexibility and portability for MPI-parallelization, a master-slave program structure is adopted (Figure 2). The number of slaves is set in the compilation command with an argument. The master is responsible for:

• Reading data from the data file into the cell space and dispatching the cell space at time step 0;

• collecting the value of a selected field in each cell from all slaves and displaying them at each time step;

• updating the variable time and sending its new value to every slave to trigger the next round of computation;

• doing some global, non-cellular automata style computing such as ECG simulation;

Each slave process is responsible for:

• receiving a subset of the cell space at time 0;

• exchanging the boundary layers with neighboring slaves at each time step;

• executing the cell program of cells in its subset at each time step;

• sending the value of a selected field to the master at each time step for display;

• receiving the new value of time at each time step.

Finally, to run the model on a cluster of SMP nodes, a two-tier parallelism – a coarse-grained setting among nodes and a fine-grained setting within nodes – is applied. The hybrid programming is straightforward – insert OpenMP directives into the master and slaves, as in the case of pure OpenMP programming to spawn a group of threads within each node to implement cell level parallelism. Due to the stereotyped appearance of these codes, we let the compiler generate them each time we compile a model, leaving the number of slaves and threads within each slave as command line arguments, so as to make parallel computation entirely transparent to users.

The anatomical model, which is a data file describing the distribution of the cell array and the initial value of fields in each cell, is independent of the cell program. The data file can be manually edited or generated by a program coded in C/C++. Due to the correspondence between each automaton cell and each biological cell, the building of the anatomical model is straightforward, even if a model has an irregular structure and a heterogeneous cell population. Usually, the Moore neighborhood is adopted, as is the case in our model.

The data used to build the 3-D heart model are the axial images of the Visible Human Project (VHP) digital male cadaver 17, which contains about 125 thoracic slices at a 1 mm interval. Using an image processing program we developed, we enter a 128 × 128 coordinate system on each slice and use a computer mouse to mark the cells of different tissue with different colors (Figure 3). This process digitalizes each slice into a data file that reports the coordinates and type of each cell. The current heart model contains six kinds of cardiac tissues: sinoatrial node (SAN), atrioventricular node (AVN), atrium (AT), ventricle (VT), and trunk conduction bundle in the atrium (CBA) and in the ventricle (CBV). The distribution of trunk conduction bundles in each slice is determined by clinical experts, but the terminal distribution of the Purkinje fibers is generated by an algorithm at runtime. After processing all 125 slices, we use a program to merge the generated 2-D data files into a 3-D data file, which constitutes the anatomical model of the heart that occupies about 280,000 cells in the 128 × 128 × 128 cellular automata cell space. We also build an illustrative 2-D model containing the same kinds of cells and the same action potential models to evaluate the ECG simulation algorithms (Figure 6). The simulated normal and some abnormal ECG waveforms support the validity of the algorithm (Figure 7).

The heterogeneity of a model is described in two ways. First, a specific field type, indicating cell type, is defined in the cell program, but whose value is initially stipulated in the anatomical model when we process tissue slices, such as:

[x, y, z] = type, ......

x, y, and z are the coordinates. Values of other fields can be defined or not in the data file. With a nested if-then statement on the value of type, the cell program is divided into several parts, each being executed by cells of the specific type. Second, type can be modified at runtime to simulate pathological changes. After a cell type is changed, its program and therefore its behavior also change.

In a tissue or organ, aside from heterogeneity, cells often show anisotropy and inhomogeneity that cannot be conveniently described while processing the raw data. To describe these two properties needs a bit of pattern formation programming 24, an interesting and challenging issue of biological modeling with cellular automata. In building the whole-heart model, the first relevant issue is cells at different layers in cardiac walls have different electrical properties 1625. To express this inhomogeneity, we developed a resolution-and dimension-independent algorithm to stratify cardiac walls into layers at the initial stage of runtime. The algorithm is comprised of three steps, and is run by all cells:

Let the layer number of SAN cells (at least one SAN cell is on the epicardium) be 0 and the layer number of AVN cells (at least one AVN cell is on the endocardium) be 50.

• If the current cell is neither a ventricular nor an atrial cell, then: if it connects to a cell whose layer is 0, its layer is 0; if it connects to a cell whose layer is 50, its layer is 50.

• If the current cell is a ventricular or an atrial cell, then: visit all its neighboring cells and find out their minimal layer number, Min. The layer of this cell is Min+1.

A slice of the stratified 3-D model and of the stratified 2-D model is shown in Figure 6. After stratification, the layer number of each ventricular cell is added to its HH equations as a special parameter to control and adjust its action potential duration (we do not use the layer number of atrial cells). This procedure ensures that epicardial cells have shorter action potential duration than endocardial cells, and the epicardium-to-endocardium repolarization is naturally and faithfully established. By assigning a large layer number to cells in the middle layers of the ventricular walls, an unusually long action potential duration is created, and the specific electrical property of M cells can be naturally simulated 25.

The slices of the digital cadaver do not provide any useful information on the distribution of terminal Purkinje fibers. Even if we know that most of them are located in the subendocardium, it is impractical to make a slice-by-slice manual description. Based on the stratified cardiac walls, this issue can be easily solved by using a lateral inhibition algorithm applied to cells on the ventricle subendocardium to generate a mesh-like Purkinje network. On completion of running the algorithm, some of the cell types are changed from ventricular cell to Purkinje cell.

The more difficult engineering issue is the fine structure of cardiac cells and conduction fibers. Simulation with the 2-D model demonstrates that, even if the HH action potential model of Purkinje cells produces a very fast upstroke, which means a quick transjunctional conduction, yet a one-cell by one-cell conduction can never give normal propagation profiles as observed in Durrer's experiment, due to a too long transjunctional delay 26. Nevertheless, our simulation shows that the rapid conduction in conduction fibers can be implemented by n-by-n cell communication among automaton cells of conduction fibers without physically building the fiber structure; n = 4 gives very satisfactory simulation results. Again, we propose that by using a pattern formation algorithm the physical construction of fiber structures is also feasible. In this scenario, a fiber is assembled by n successive automaton cells sharing the same, unique identity number, among which there is no conduction delay. Currently, with the 2-D model we find that even if letting each automaton cell stand for a discrete ventricular cell, the model works quite well in ECG simulation.

Each automaton cell is a computing unit for action potential and ECG simulation. The electrical activity of each automaton cell of a specific cell type is described by the corresponding HH type action potential model. Five action potential models are employed to simulate activities of different cardiac cells (The cells of trunk conduction fiber in atriuma and ventricles use the action potential model of the Purkinje cell) 2728293031. Since the models are based on experimental data from different animal species, parameters of K⁺ channels are slightly changed to produce action potential duration of human cardiac cells. Time constraints are the only reason for us to adopt early published, simpler action potential models. In the given simulations, all action potential models are solved using explicit Euler integration with a time step, Δt, of 0.01 ms 32. An asynchronous adaptive time step method has also been developed to speed up simulation 49. Electrical activities at channel (Figure 4), cell (Figure 5), and organ (Figures 6, and 7) levels are monitored and captured at runtime.

Each automaton cell in the 3-D model has 26 neighbors, linked by gap junctions. When a cell depolarizes, driven by the potential difference between it and its neighboring cells, transjunctional currents generate and propagate to neighbors through gap junctions. Simulations with the 2-D model show that the simple static gap junction model, in which the resistance of the gap junction is a constant and the transjunctional current follows Ohm's law, works quite well. The dynamic gap junction models, in which the resistance of gap junction changes in a nonlinear manner with membrane potential, will significantly increase computational time 3334 and create an extra burden for large-scale modeling.

Values of gap junction resistance between different cells and in different directions are initially set according to available experimental data, and then tuned via simulation according to Durrer's experimental observations 26. Excitation conduction between cells of the same layer follows an end-to-end propagation, and between cells of different layers is a side-to-side process. Parameters are also adjusted to fix the ratio of side-to-side conduction speed vs. end-to-end conduction speed to be one-third in ventricular cells and one-tenth in atrial cells 35. Gap junction resistance can be modified at runtime to examine its effect on excitation propagation. For each cell in each round of computation, before solving the HH equations, the transmembrane potentials of all neighboring cells are checked and the transjunctional currents computed and summed to get the stimulating current, I_stim, that the cell receives. The direction and speed of electrical propagation within the heart are jointly controlled by: (1) the stratification of the ventricular walls, (2) the speed of end-to-end and side-to-side conduction, (3) the ratio of end-to-end conduction speed to side-to-side conduction speed, and (4) the distribution of the trunk conduction bundle.

In the cardiac model built with the extended cellular automata, abnormal electrical activities are grouped into four classes. The first class is based on anatomical defects in the heart that can be described by changing the type of some cells, either in the anatomical model or in the cell program. The Wolff-Parkinson-White syndrome caused by atrium-ventricle bypasses is a representative case. The second class is resulted from aberrant environments, especially abnormal ionic concentration (e.g. hyperkalemia) which significantly influences action potential. Abnormality of transjunctional conduction constitutes the third class, and the fourth class is is comprised of anomalous dynamics of cellular electrical activity itself. Changes in the HH equations can simulate these aberrations. In most cases, pathological changes affect more than one aspect. For example, in addition to providing an abnormal cell environment, ischemia also results in changed cellular electrical activity and altered gap junction resistance. To simulate the effect of ischemia, a special field blood, with normal value 1.0 is defined in the cell program, and introduced as an extra parameter in action potential and in gap junction models. To multiply the conductivity of the gap junction by an abnormal value of blood (e.g., blood 0.7) can simulate a lowered gap junction conduction speed. A small value for blood, when inserted into the equation computing Ca⁺⁺ current, affects the generation of the action potential. Dynamically modified blood can naturally simulate progressive ischemia (Figure 5 and Figure 7) 3637.

Combining these factors, a variety of arrhythmias can be simulated. For example, by partially or completely blocking the activity of cells, or the conduction of gap junctions at specific locations, various conduction blocks such as AV (atrium-ventricle) block, LBBB (left bundle branch block) and RBBB (right bundle branch block) can be conveniently simulated. Many rapid arrhythmias are triggered by successive ectopic beats. Such abnormal beats can be produced in the model by providing cells at specific locations with extra stimulation or by changing them from atrial/ventricular to SAN cells. For abnormal propagations such as long QT syndrome, caused by the extra long refractory period of M cells, we can adjust, either dynamically or statically, the layer number of cells at the middle layers of the ventricular walls. Complex spatiotemporal patterns of arrhythmias are formed through discrete cell-cell communication. By these means, arrhythmic electrical activities can be simulated in flexible ways, different from those models created by cable equations or other partial differential equations (PDE).

Linking ECG simulation directly with cellular and channel electrical activity is crucial for the understanding and treatment of arrhythmias. Since each automaton cell in our model is a computing unit, a cell level ECG simulation algorithm is developed to compute each cell's field point potential. The algorithm, implemented as a backend function coded in C language, shows strengths, because such cell level simulation builds a link between the action potentials of cells and the ECG waveforms. The limitation here is that this heart model does not include the chest, so that the contribution of thoracic tissues to body surface potential is much simplified. The model reads the membrane potential of every automaton cell and computes its field potentials at the standard lead locations.

Several assumptions are made in the computations on account of established physical laws. (a) Each (automaton) cell has a spherical shape. Thus, the transmembrane potential, V_m, is uniform on all parts of the cell membrane, except at gap junctions. (b) Gap junctions occupy the same area and are symmetrically distributed on each cell. (c) The distance from cell to field point is long enough so that the two solid angles subtended by the positive and negative sides of a cell membrane to a field point are equal. (d) Dielectric effects on field potential are neglected. σ_media is the average conductivity of the 1-D tissues between a source cell and a field point; σ_e the conductivity of the intercellular matrix; and σ_i the conductivity of the intracellular cytoplasm. (e) For any neighboring cells, j and k, σ_je = σ_ke, σ_ji = σ_ki and φ_je = φ_ke, φ_e is the potential in intercellular space. For convenience, we present the algorithm for the 2-D model here; the 3-D case is similar. The equation computing the field potential of an isolated cell at field point P is 38

σ_i φ_i - σ_e φ_e is the double layer strength of the cell membrane. When there is no propagation on the cell membrane, the field potential is zero. If the cell connects with eight adjacent cells, equation (1) becomes:

Here, σ_i φ_i - σ_ji φ_ji is the double layer strength of the gap junction between the cell and its jth neighbor. The transjunctional potential difference between the two cells then becomes:

φ_i - φ_ji = φ_i - φ_e + φ_e - φ_ji = φ_i - φ_e + φ_je - φ_ji = V_m - V_jm     (3)

S1 to S8 are areas of the eight gap junctions; Ω₁ to Ω₈ are solid angles subtended by them; and S0 denotes the remaining part of the cell membrane (Figure 8). Let the membrane area positive to P be S_B and the area negative to P be S_A. If: (a) S_B = S_A, (b) the same and even number gap junctions distribute symmetrically on S_B and S_A, and (c) all gap junctions have an equal area, then it can be proved that:

Here, Σ K_A is the sum of areas of gap junctions on S_A, and Σ K_B is the sum of areas on S_B. In this circumstance, the first term in equation (2) is zero, and only the second term makes a contribution to field potential. The equation now becomes:

Considering all Sj (j = 1...8) to be equal, and cos α₁ = cos α₅ = cos 90° = 0, we have:

Here, is a constant that affects only the baseline, but not the shape of the ECG waveform. The field potential at point P, generated by the whole heart, is the superposition of all cells' contributions:

Here, n is the number of automaton cells, and φ_j(P) is computed with equation (5).

Due to the time needed for the numerical solution of massive HH equations, the computing of σ_media R² has to be much simplified, as described in equation (7), where σ_{media_i} is the conductivity of tissue i, and R_i is the length of tissue i on the line between the source cell and the field point P.

As noted above, both the boundary effect and order of dielectric distribution are neglected. The 2-D (3-D) boundary between two tissues is treated as a 1-D boundary, and the ordered dielectric distribution is treated as an unordered distribution. Because of the huge number of cardiac cells and various possible field points on body surface, epicardium, endocardium and cardiac cavities, it is impossible to deal with numerous situations of boundary conditions between cells and field points. The method we adopt here is to make a runtime traverse from the source cell to the field point and meanwhile check the conductivity of each tissue and determine the 1-D space it occupies. The scaled distance is 1.0 between two perpendicular-connected cells, and 1.414 between two oblique-connected cells. Although the electrical property of non-cardiac tissues is much simplified, the electrical property of cardiac cells is intensively considered. Cardiac cells under different conditions have different conductivity 39. It is σ_Depo = 0.4 mhos/m in depolarization, σ_Repo = 0.25 mhos/m in repolarization, σ_Rest = 0.2 mhos/m in resting state, σ_Infarct = 0.1 mhos/m in acute infarcted area, and σ_Ischemia = 0.16 mhos/m in ischemic area. We let σ_Blood = 0.6 and σ_Chest = 0.1 be the average conductivity of tissues in chest 4041. We find ECG simulation is not clearly impaired by this simplified dielectric description, but benefits from the intensified cardiac tissue description.

For the physically parallelized whole-heart electrophysiological model built with the extended cellular automata, several factors impair the simulation efficiency on a cluster of SMP.

The first difficulty is the graphic display of the simulation. The main display window, as the critical resource in the parallelized program, can only be sequentially accessed by the cells. Thus, to display the updated state of all cells leads to a significant decrease of running speed. We mitigate this problem by displaying the cell state every 20 or even 50 steps, instead of every single step. The second problem is the overhead of communication among slaves located on different nodes. The time cost of communication, spent on the exchange of boundary sheets between neighboring slaves in each round, rises with the increase of slave number. An extreme case for a 128 × 128 × 128 model is that there are 128 nodes, and each slave deals with just one sheet, requiring every sheet to be exchanged in each round. The physical link between nodes also significantly affects performance. An additional limitation is that the overhead of fork/join operations in the OpenMP parallelized program degrades performance, although not very significantly. The OpenMP directive parallel can be inserted either before the endless while loop, or more simply, before the three for loops. If it is inserted before the for loops, the fork and join operations, which create and delete threads and allocate and collect memory for temporary variables in each thread, will be repeatedly executed in each round. The preferable way, which eliminates this unnecessary expenditure, is to insert the directive omp parallel before the while loop, and the directive omp for before the for loops. Between the omp parallel and the omp for,omp master is used to limit parallelism to only the cell program part. Finally, load balance becomes a limitation when a model has an irregular and/or heterogeneous structure. The heart, with four chambers and an uneven shape, is a typical case. Usually, on a cluster consisting of m identical nodes each containing n CPUs, a program is evenly dispatched to distribute the cell space across all nodes. We find that using this strategy, the 3-D heart model cannot reach the best load balance and performance because different nodes deal with different numbers of cardiac cells. Furthermore, even if each node contains the same number of cardiac cells, since different cells run different action potential models, the burden of computation remains unequal. Only a solid cubic model with homogeneous cells occupying the full cell space can ensure a best performance. Thus, if a model needs to run many simulations, an important issue is to find the best cell space partition.

Pilot runs with the 3-D model have been made on a 4-node SUN computer cluster. The head node has eight UltraSparc CPUs, and the other nodes each have four identical CPUs. Five processes, 1 master and 4 slaves, are created at each run. The master and a slave are assigned to the head node, and each remaining slave is assigned to a node. Within each slave, four threads are created using the OpenMP parallel directive. Figure 9 gives the performance results for an even partition strategy. With this partition, due to the irregular structure of the heart and the heterogeneity of cardiac cells, the combination of 4 slaves-4 threads does not give the best performance. The 4 slaves-2 threads setting behaves better because the overall cost is lower. . However, when the whole cell space is occupied by ventricular cells, the 4 slaves-4 threads version provides the best performance, as predicted.