A 4-fJ/Spike Artificial Neuron in 65 nm CMOS Technology

Abstract

As Moore’s law reaches its end, traditional computing technology based on the Von Neumann architecture is facing fundamental limits. Among them is poor energy efficiency. This situation motivates the investigation of different processing information paradigms, such as the use of spiking neural networks, which also introduce cognitive characteristics. As applications at very high scale are addressed, the energy dissipation needs to be minimized. This effort starts from the neuron cell. In this context, this paper presents the design of an original artificial neuron, in standard 65nm CMOS technology with optimized energy efficiency. The neuron circuit response is designed as an approximation of the Morris-Lecar theoretical model. In order to implement the non-linear gating variables, which control the ionic channel currents, transistors operating in deep subthreshold are employed. Two different circuit variants describing the neuron model equations have been developed. The first one features spike characteristics, which correlate well with a biological neuron model. The second one is a simplification of the first, designed to exhibit higher spiking frequencies, targeting large scale bio-inspired information processing applications. The most important feature of the fabricated circuits is the energy efficiency of a few femtojoules per spike, which improves prior state-of-the-art by two to three orders of magnitude. This performance is achieved by minimizing two key parameters: the supply voltage and the related membrane capacitance. Meanwhile, the obtained standby power at a resting output does not exceed tens of picowatts. The two variants were sized to 200µm2 and 35µm2 with the latter reaching a spiking output frequency of 26kHz. This performance level could address various contexts, such as highly integrated neuro-processors for robotics, neuroscience or medical applications.

Introduction

Computing technology, based on binary coding, Von Neumann architecture and CMOS technology, is currently reaching certain limits (Waldrop, 2016). Traditional computers, the champions for the resolution of complex equation systems, have difficulties to classify/organize data, something that the human brain seems to accomplish effectively. For this reason, research in the field of Artificial Neural Networks (ANNs) is attracting much attention and is quickly developing worldwide. At the bottom of these efforts lies the ultimate goal to realize machines that could surpass the human brain, in some aspects of cognitive intelligence. In that sense, brain research and ANNs bear the promise of a new computing paradigm.

Currently, traditional, discrete-time, digital ANNs, fueled by the unprecedented computational capability of modern Graphics Processing Units (GPUs), represent the state-of-the-art for addressing cognitive tasks (Oh and Jung, 2004; LeCun et al., 2015) such the ones encountered in computer vision applications. However, it is the more recent class of Spiking Neural Networks (SNNs), often referred to as the third generation of neural networks, that are known to be bio-realistic and more computationally potent compared with their predecessors (Maass, 1997). The functional similarity with the actual biological networks permits envisioning, apart from interfacing or reproducing brain processes, the implementation of circuits and systems with cognitive characteristics without explicit programming tasks. This would endow the modern generation of computers with the capacity to learn from input data.

In SNNs, neuronal communication is carried out in the form of noise-robust, signal pulses or “spikes.” SNNs try to reproduce the physical characteristics of the brain, through highly connected neurons dendrites and axons. At present, two main methodologies fulfill neuro-inspired computing tasks: digital simulation and hardware implementation.

In digital simulations, the dynamics of neuronal models are coded in software and calculated on general-purpose digital hardware. Digital simulations have the advantage that they can be reliably programmed using numerical operations of very high precision. However, their reliability comes at the cost of high circuit complexity, which is necessary for the data transfer, exchange and processing (Cao et al., 2015). Accordingly, the energy consumption remains still very high, especially as one juxtaposes biological data for comparison. For instance, the brain of the cat is emulated at the cost of a power dissipation in the megawatt range (Ananthanarayanan et al., 2009), while the animal brain actually consumes only a couple of watts.

As far as hardware implementations of SNNs are concerned, the alternative, “neuromorphic,” approach consists of employing VLSI circuit technology, namely CMOS fabrication processes which can be possibly associated with more advanced device technology such as memristors (Kim et al., 2012). The analog hardware approach consists of a large-scale integration of silicon artificial neurons (AN) and synapses, in an attempt to produce low power neuro-inspired architectures compatible with the current electronics technology.

The efficiency of such architectures can be revealed in contrast to the energy consumption of biological neurons (BN). Brain activity needs a continuous exchange of ions through the cell membrane and these exchanges correspond to the charge and discharge of the neuron capacitance (soma, dendrite, and axon). As a consequence, the important parameters for energy dissipation are the membrane capacitance and the voltage swing. Membrane capacitance varies considerably according to the type of neuron cell, ranging from picofarads to nanofarads for the largest ones (Amzica and Neckelmann, 1999; Golowasch et al., 2009; Rössert et al., 2009; Tripathy and Gerkin, 2012). Interestingly, a recent estimation of the capacitance that could be involved for computation in the human cortex is proposed in Hasler and Marr (2013). The calculations are based on a digital power model and suggest a biological system of 10¹² neurons with a 0.5 Hz average firing rate. The total capacitance value is calculated at 245 pF that is high when compared with the femtofarad order common in integrated circuits. Indeed, energy savings could be envisioned in silicon AN by aiming at reducing the capacitance and/or the voltage swing.

Next to a reduced capacitance, low power operation in silicon neurons can be facilitated by the physics of the MOS transistors. Indeed, as it has been observed (Mead, 1989, 1990) the nervous system uses, as its basic operation, a current that increases exponentially with the membrane voltage, similar to the current-voltage characteristic of a MOS transistor operating in subthreshold. However, the physical origins of these exponential dependencies are very different: a non-linear voltage controlled conductance in biological membrane against a current controlled by an energy barrier in the transistor. Due to this, the MOS transistor can only asymptotically approach a slope of kT/q per e-fold of current change, while the biology is not limited as such (Mead, 1989, 1990). Even if I-V characteristics show different slopes, a bridge between the physics of biological membrane and the one of electronic devices has been established, especially when the energy and power properties are considered. This led to the advent of neuromorphic silicon neurons, which allowed neuronal spiking dynamics to be directly emulated on analog large-scale integration chips. So far, several generations of SNNs have been proposed and the reader could refer to the relevant works (Misra and Saha, 2010; Indiveri et al., 2011; Hasler and Marr, 2013) to obtain more information.

Based upon these previous works, this paper describes the design and measurement results of a new family of silicon AN. It was designed under the guidelines of (i) a biophysically meaningful model, (ii) a minimum energy dissipation, (iii) an analog circuit that would allow a complete time variation modeling of the membrane potential and (iv) a resulting topology, that when implemented in CMOS technology, it would occupy a minimum area in order to enable large scale integration. This unique combination of characteristics resulted in a neuron topology that was measured to consume several orders of magnitude less energy than the values encountered either for BN or the AN reported so far.

The rest of this paper is organized as follows: The “Materials and methods” section will be devoted to a discussion on neuron energy efficiency, the selection of the mathematical model and the circuit topology and functionality. The circuit proposed in this paper was fabricated and characterized experimentally. Both simulation and experimental results are described in the “Results” section. The “Discussion” section presents a comparison with the state of the art and highlights issues regarding noise, supply voltage sensitivity and temperature impact. Finally conclusions are drawn in the eponymous last section.

Materials and Methods

Energy Efficiency of Biological and Artificial Neurons

In BN, total power dissipation consists of a static and a dynamic part. The major factor of static power dissipation in neuron cells is due to Na-K pumps maintaining a constant resting potential value, despite the sodium and potassium leakage currents. In the circuit design domain, optimizing static power consumption imposes low DC voltage supply and transistors featuring small gate widths.

The dynamic part of energy dissipation is due to the Action Potential (AP) or “spike” generation and propagation. In BN, the AP corresponds to a rapid voltage change between extra and intra-cellular spaces, which are separated by the (quasi-) insulating lipid bilayer. According to the reference model of charge transfer, the two-ion model (Hodgkin and Huxley, 1952), the rising part of the spike is mainly due to an inward sodium current, while the decreasing part is due to an outward potassium current. Changing the voltage across the membrane capacitance Cm by ΔV, will need, at least, an amount of charge ΔQ = CmΔV to move from one side of the membrane to the other. Since the driving potentials on both sides of the membrane (the Nernst potentials) are constant, the minimum energy dissipated for this change will be ΔE = ½CmΔV². If we consider the generation and propagation of a spike (charge and discharge), the total energy dissipated by the cell will be at least CmΔV².

The driving force for the inward and outward currents are the Nernst potentials Ena and Ek, which are almost constant in time. In living beings, Ek varies from −80 to −100 mV, while for Ena it is from +30 to +55 mV (Hodgkin and Huxley, 1952; Morris and Lecar, 1981; Wei et al., 2014). The membrane potential always stays between Ek and Ena and in vivo measurements showed that the resting potential is close to Ek while the ΔV for a spike is close to 100 mV (Hodgkin and Huxley, 1952). Using ΔV = 100 mV and Cm = 245pF (Hasler and Marr, 2013) means at least 1.5 × 10⁸ sodium (resp. potassium) ions are entering (resp. leaving) the neuron at each AP and the corresponding energy loss is about CmΔV² = 2.45 pJ.

Starting from these simple considerations, it is clear that decreasing the dissipated energy in AN passes from setting Cm and ΔV to the smallest possible values. For artificial neuron circuits, the spike amplitude is proportional or strongly related to the applied DC voltage (Indiveri et al., 2011). Usually, the DC voltage of these circuits is about 1V. Since the energy loss varies with ΔV², the smallest possible value for ΔV is desired. Meanwhile, maintaining compatibility with the biological neuron spike amplitude, a feature of interest for some applications, entails fixing ΔV to about 100 mV. This implies a DC supply voltage of about 150 to 200 mV and constitutes a strong constraint on the active devices comprising the circuit. In particular, the transistors should operate in deep subthreshold regime characterized by a gate voltage far below the threshold, which currently varies between 0.3 to 0.5 V for advanced fabrication nodes. In this regime, however, MOS transistors show some interesting characteristics, such as the exponential control of the drain current by the gate voltage, a low current and therefore low energy dissipation, but they also present some drawbacks, such as a limited transconductance leading to extremely low voltage and power gains.

The other parameter targeted for low energy dissipation is the membrane capacitance. In artificial neuron circuits three main components can be distinguished: the discrete capacitance added in the circuit, and the parasitic capacitance of interconnects. The gate capacitance of transistors was found negligible for the small transistor sizes used in this work. Since the membrane capacitance value has a strong impact on several circuit parameters (energy loss, membrane charging and discharging times, spiking frequency, membrane voltage noise), its value is of utmost importance. For this reason, a mathematical model for the neuron is necessary to provide the necessary guidelines for circuit design.

Neuron Model Selection: The Morris-Lecar Model

The implementation of an artificial neuron circuit is defined by the selection of a corresponding mathematical model. An analysis of the various advantages and drawbacks of the different mathematical neuron models was proposed in Izhikevich (2004). The Leak Integrate and Fire (LIF) model is one of the simplest and most widely used for such implementations (van Schaik, 2001; Indiveri et al., 2011). However, it is not biophysically realistic: it does not accurately represent the dynamics of ion transport through the neuron membrane. In the analysis of Izhikevich (2004), it was shown that from a biophysical standpoint two models were meaningful: The Hodgkin-Huxley (HH) model (Hodgkin and Huxley, 1952) and the Morris-Lecar (ML) model (Morris and Lecar, 1981). The HH and refined-HH models (Wei et al., 2014) are considered as reference for in depth analysis of the neuron non-linear behavior. Nevertheless, the connection between the HH model and transistor physics is not easily discernible. When transferred to the circuit domain, the use of four non-linear differential equations and multiple non-linear parameters would lead to a relatively complicated topology, demanding considerable silicon surface (Yu and Cauwenberghs, 2010). On the contrary, the ML model is much more attractive for this purpose (Patel and DeWeerth, 1997). It is compiled as a system of two non-linear differential equations associated with exponential functions (Equations 1–5).

CmdVmdt=Iex-GCamss(Vm).(Vm-ECa) - GK.n.(Vm-EK)- GL(Vm-EL) (1)

dndt= λ (Vm). (nss(Vm)-n) (2)

mss(Vm)=12.[1+Tanh (Vm−V1V2)] (3)

nss(Vm)=12.[1+Tanh (Vm−V3V4)] (4)

λ(Vm)= λ0. Cosh (Vm-V32V4) (5)

In the above equations the symbols are explained as follows: Cm is the membrane capacitance, Iex is the excitatory current, EK, ECa, and EL, are the ion equilibrium potentials, GCa, GK, GL are the Ca, K and leak conductances, nss and mss are the steady-state potassium and sodium gating variables, λ0 is the reference frequency and V1−4 are fitting parameters that can serve for tuning the dynamic properties in order to represent different systems of interest.

The ML model features only two state variables: the membrane voltage Vm and the potassium gating variable n. It is well suited to perform phase plane analysis, an efficient tool for analyzing the non-linear dynamics of neurons in terms of stability, spiking conditions and spiking modes (Izhikevich, 2007). Acknowledging this fact, the next section details analytically the design of a circuit response approximating the ML equations. Though exact 1-1 circuit representation of the ML equations is not pursued, this approach yields an energy-efficient design, whose response achieves a good correlation with a higher-order cortical neuron model (Section Biomimetic neuron). As the ML model was initially conceived referring to a barnacle muscle, the following analysis uses Na and K ions (instead of Ca and K) aiming to indicate in the notation the neuronal context, which is ultimately addressed.

The Artificial Neuron Circuit Design

Expressing the system of equations of the ML model as a circuit response can be carried out through a series of considerations for the various components of the system. The most apparent involves the two differential equations to be represented by expressions based on current node summation, as per Kirchhoff's Current Law (KCL). Also, Na and K channel dynamics can be represented by single transistors biased in subthreshold, as it has been previously shown (Farquhar and Hasler, 2005; Hynna and Boahen, 2007).

Figure 1 shows the proposed neuron circuit, based on the ML model described above. The membrane capacitance, Cm, is charged through MPNa (a PMOS transistor modeling the sodium channel) and discharged through MNK (an NMOS transistor modeling the potassium channel) and the leakage conductance GL. The membrane voltage node is the summing point of the excitation current, leak currents as well as two feedback loops: A positive feedback loop through MPNa and MP1/MN1, which implements a pull-up network for Vm, and a negative feedback loop, which implements a pull-down network through MNK and the two cascaded inverters MP2/MN2 and MP3/MN3. The time constant of the negative-feedback loop is set by capacitance CK.

FIGURE 1

Figure 1. Proposed biomimetic artificial neuron circuit based on the ML model.

The excitation current Iex is assumed to be provided by a synaptic circuit, which in the most rudimentary form can be implemented by a single transistor current source.

The circuit analysis for the proposed circuit is divided in three parts: the drain current model, the static properties of inverters in subthreshold operation and the circuit response equation.

Drain Current Model in the Sub-Threshold Regime

All the transistors used in the circuit are assumed to operate in the deep subthreshold regime, with a supply voltage lower than 200 mV. The most important effect to be observed from a modeling standpoint is the basic exponential relation for the gate controlled drain current. For this reason, the corresponding expression has been chosen as:

Ids=G0.exp(VgsηVt).Vds (6)

G0 is the device conductance, that will be referred in the following as Gn0 for NMOS and Gp0 for PMOS, Vgs and Vds are the gate-to-source and drain-to-source voltages, η is the subthreshold slope factor and Vt is the thermal voltage. For the sake of simplicity, it is assumed that the NMOS and PMOS ideality factors are the same.

As it will be proved, the use of such a simple model provides a comfortable medium for demonstrating the correlation between the circuit response and the ML model equations sufficiently. In addition to that, it allows a straightforward circuit analysis that is essential for the implementation effort.

Static Properties of Inverters in Sub-threshold

The role of inverters is central to circuit performance as the inverter is the means of realizing the necessary Tanh functions in Equations (3,4). We assume a CMOS inverter like the one in Figure 2, operating in the sub-threshold regime. Supply voltages are +/−Vd and input/output voltages are Vin and Vout. By applying KCL for the output node of the inverter in Figure 2, and expressing currents as in Equation (6) (assuming same slope factors for NMOS and PMOS), the output voltage can be found as:

Vout=-Vd.Tanh[ Vin ηVt+12ln(Gn0Gp0)] =-Vd.Tanh(Vin-Visv ηVt)=-Vd.Tanh(Vq ηVt) (7)

In Equation (7) we simplify the operand of the Tanh function with the use of Visv, that is the input voltage, provided that Vout = 0, then:

Visv≡Vin|Vout = 0=-η. Vt2.ln(Gn0Gp0) (8)

According to Equation (8), Visv can be either positive or negative depending on the conductance ratio. For further simplification, we define the parameter Vq as:

Vq=Vin-Visv (9)

The above expression shows two important properties of the proposed circuit. Primarily, the control of the conductance ratio allows adjusting the inverter switching voltage. With a ratio of 10 between the NMOS and PMOS conductance, the switching threshold can be shifted around 50 mV. With this technique, the required voltage shifts V1 and V3 in Equations (3,4) can be tuned sufficiently. Secondarily, the presence of voltage gain is essential to match the behavior of BN. As it has been shown by Mead (1989), the high sub-threshold slope of MOS transistors (larger than 60 mV/dec) yields a much lower current-voltage slope than the one in biology. So, correct circuit operation is associated with the presence of inverter gain.

FIGURE 2

Figure 2. Inverter biasing conditions used in the analysis.

In Equation (7), the maximum voltage gain will be obtained for Vout = 0. This can be expressed through differentiation of as:

AV=dVoutdVin|Vin = Visv=-VdηVt (10)

According to Equation (10), the magnitude of Av can be larger than one for Vd > ηVt. For an ηVt about 35 to 40 mV, a significant voltage gain can be provided when Vd > 70–80 mV. It is worth mentioning that this supply voltage value is close to the Nernst potentials encountered in biological cells. This last observation reveals that the neuron circuit can be implemented to operate under extremely low supply bias.

The Circuit Response

To show the connection between the Morris-Lecar equations and the circuit response, we use the circuit shown in Figure 3. In order to simplify the equations and without loss of generality, a symmetrical supply voltage is considered.

FIGURE 3

Figure 3. Artificial neuron circuit analysis.

Neglecting transistor capacitances and assuming only the capacitances shown in the schematic, two circuit equations are derived from the application of KCL on the two summing nodes VGK and Vm:

{ICm=INa−IK+Iex−IL ICk=Ip2−In2 (11,12)

In the equations above, INa and IK are the drain currents of MPNa and MNK, Iex and IL the excitation and leak currents while Ip2 and In2 are the drain currents in the MP2/MN2 inverter.

Employing Equation (6) for the drain currents, gives:

{CmdVmdt=GNa. exp(Vd − VGNaηVt).(Vd−Vm)− GK. exp(VGK + VdηVt). (Vm+Vd)+ Iex−IL(Vm) CkdVGKdt=Gp2.exp(Vd −Vout3ηVt).(Vd−VGK) − Gn2.exp(Vout3 + Vd ηVt).(Vd+VGK) (13,14)

The VGNa and VGK are the gate voltages of MPNa and MNK, and Vout3 is the output of MP3/MN3 inverter. VGNa is the output of the MP1/MN1 inverter and Vout3 the output of MP3/MN3 both of which are related to Vm by an input/output relation such as Equation (7).

It can be easily proven that the last equation could also be rewritten as:

dVGKdt=λ(Vm)[Vd.Tanh(Vq2ηVt)-VGK] (15)

With:

λ(Vm)=2.Cosh (Vq2ηVt).exp (VdηVt).Gn2.Gp2Ck (16)

and Vq2, as seen above:

Vq2=Vin2-Visv2=Vout3+ηVt2ln(Gn2Gp2) (17)

The differential equation for VGK is similar to that of the Morris-Lecar model. In particular, the time dynamics of VGK are defined by the reference frequency λ(Vm), which is related to the circuit parameters through Equation (16).

Introducing the parameters mss(Vm) and n, Equation (13) could be rewritten as the first differential equation of the Morris-Lecar model:

CmdVmdt=Gna.exp (2VdηVt).mss(Vm).(Vd-Vm) -Gk.exp (2VdηVt).n.(Vm+Vd)+Iex- IL(Vm) (18)

With:

n=exp(VGK-VdηVt) (19)

and

mss(Vm) = exp(−VGNa(Vm)+VdηVt) (20)

It is easy to see that, mss(Vm) and n are between 0 and 1 because VGNa and VGK are between −Vd and + Vd. The mss(Vm) function is very well correlated with the Tanh function of the Morris-Lecar model and n, the potassium gating variable, is simply related to VGK.

Assuming a small VGK, the first order approximation of n would be:

n=exp ( -VdηVt) exp (VGKηVt)≈exp ( -VdηVt). (1+VGKηVt) (21)

This linear relation between n and VGK allows us to rewrite Equation (15) as:

dndt=λ(Vm)[nss(Vm)-n] (21)

which matches the second differential equation (Equation 2) of the Morris-Lecar model.

Equations (13,14) show that the dynamic behavior of the circuit is characterized by two time constants, which are associated with the capacitances Cm and Ck and the currents delivered by the transistors MPNa, MNK, and MN2/MP2. The static power of the AN is simply given by the product of the supply voltage (+/−Vd) by the leakage currents flowing through the inverters, the leak conductance (if present), and the transistors MPNa and MNK. The additional energy dissipation during spiking (dynamic power) is related to the charge and discharge of the capacitances Cm and Ck.

To conclude this section, it has been formally shown how the proposed circuit response approximates the Morris-Lecar model equations. The main components of this approach have been the use of a simple subthreshold drain current model (Equation 6), the assumption of equal slope factors for PMOS and NMOS and a linearization step. This analytical model was used as a guideline for circuit design and was complemented by more rigorous circuit simulations.

A Simplified Neuron Circuit

If biological accuracy is not of prime concern it could be traded-off for a reduced silicon footprint and a high output frequency performance. Such a modification is suggested in Figure 4. In this circuit, the characteristics of the feedback loops (dynamics, threshold voltage etc.) cannot be set independently, because of the presence of a common inverter stage MP1/MN1. Further simplification is obtained by the removal of the leak conductance. The resulting circuit response might not reach the biological relevance of the initial circuit, yet it is still interesting, especially when combined with low capacitance values. Such a lightweight configuration, referred as “simplified neuron” in the following, could adhere to a wider range of frequency specification, aiming at potential deployment in SNNs for computational applications. As it is exhibited below, though the original spike form is not retained and initial design flexibility is compromised, the spiking nature of the output remains intact and maximum output frequency is increased by two orders of magnitude with respect to the biology. Obviously, the approach in circuit analysis seen above is still valid.

FIGURE 4

Figure 4. Simplified neuron circuit.

Circuit Fabrication and Measurement Setup

The ML neuron circuits described in the preceding sections have been designed and fabricated using TSMC 65 nm CMOS process in the LP option. The test chip size was 1.2 × 2.1 mm and comprised a variety of neuron circuits including variants of the aforementioned biomimetic and simplified neuron topologies. The area occupied by a single neuron circuit ranges from 35 μm² for the simplified neuron to 200 μm² for the biomimetic version without accounting for the output driving buffer or protection circuitry. The capacitors typically dominate the area utilization: 65% for the simplified and 70% in the biomimetic version. A photograph of the fabricated chip is shown in Figure 5. The designs presented here are highlighted.

FIGURE 5

Figure 5. Die photograph of the fabricated chip. The highlighted DUTs, each one sized 450 × 250 [mu]m, refer to the two measured neuron circuit variants: biomimetic (core size: 300 [mu]m2) and simplified (core size: 35 [mu]m2).

All the neuron circuits were designed with external biasing. The excitation was implemented with on-chip externally biased transconductance. The output was monitored through an on-chip unity gain output buffer that was designed ensuring that the frequency response of the neuron circuit would not be affected. The output buffer featured independent DC supply to enable accurate power consumption measurements while the neuron's bias pins were not ESD protected. Concerning the results presented in the following section: static power refers to a zero excitation condition, while dynamic power consumption is deduced from the consumption under constant excitation, which induces a spiking output.

The experimental set-up was composed of a waveform generator that generated step signals for the neuron excitation. The supplies used for biasing enabled measurement of the average current with a nominal 500 fA accuracy. The output signal issued from the buffer was monitored by an oscilloscope, in order to perform spike width, frequency and amplitude measurements.

Results

In this section, we will focus on the performance of the proposed biomimetic and simplified neuron circuits. After describing the basic functionality, the measurement results are presented, regarding spike shape and frequency, standby power and energy dissipation per spike.

Basic Circuit Operation

To supplement the mathematical analysis above with more intuition, circuit operation can be clarified by following the circuit dynamics for the membrane voltage (Figure 6A) and the drain currents of MPNa and MNK transistors (Figure 6B). The following description applies to both presented variants equally.

FIGURE 6

Figure 6. (A) Membrane potential and excitation current; (B) sodium and potassium currents during the action potential peak. The circuit parameters are: VDD = 100 mV, VSS = −100 mV, Cm = 20 fF and CK = 60fF.

When excitation is not present, a constant membrane voltage Vm is obtained, assuming a stable circuit. This constant voltage is known in neuron physiology as the resting potential Vrest. In BN, this value is close to the potassium Nernst potential. For the circuit, this means that Vrest should be essentially pulled down close to the negative supply. This is achieved by proper sizing of the transistors representing the ion channels. The conductance of MNK should be much higher than the one of MPNa, in order to achieve a small voltage drop across MNK source and drain terminals.

While in a rest state, with Vm close to the negative rail, there exists a small current through MNK and MPNa. Adding an excitation current Iex will induce further charge accumulation in Cm. However, if this current is not sufficient to overcome the current that can be sustained in the MNK branch, then due to its partial compensation, Vm will be slightly increased on a higher level than the initial resting potential, but still remain constant. On the contrary, if the excitation current exceeds this limit, this will cause Vm to increase with a slope proportional to the excess current: this situation is illustrated in Figure 6A by the quasi-linear increase of membrane potential after 100 μS. If the duration of this excitation is enough to reach the switching threshold of the first inverter (MN1/MP1), the output will exhibit a steep ascent through the positive feedback action of MPNa. This happens in the vicinity of −40 mV in Figure 6A. This steep increase is due to the exponential relationship between the drain current of MPNa and its gate voltage, VGNa.

Following the increase, as Vm approaches the positive supply value, the source-drain voltage of MPNa decreases. This will effectively limit the rise of Vm, because the current though MPNa diminishes as shown in Figure 6B. In the mean-time, another phenomenon takes place. As Vm rises, capacitor Ck is charged and the drain current of transistor MNK is increased. This action acts antagonistically to the reason creating it, i.e., to the rise of Vm, therefore constituting negative feedback. Vm reaches its maximum value when the drain currents of MPNa and MNK are equal. After this point the increase in the MNK drain current will discharge the membrane capacitance, returning Vm eventually back to its resting value or, depending on the excitation, to a value where the current charge accumulation in Cm recommences.

As mentioned before, the above described dynamics and the output spike form rely mainly on circuit characteristics such as the conductance ratio of MPNa and MNK, the switching threshold of the inverters, as well as the values of the associated capacitances for Cm or Ck. For the biomimetic version of the proposed circuit, the facility to independently configure the gains and the time constants for the antagonizing loops ensures tuning of the output form parameters, such as the resting potential, the spike amplitude and width, the spiking frequency and the range of the excitation current. This flexibility permits approximating the behavior of biological neuron models, as it is shown in the following paragraph.

Biomimetic Neuron

For various applications (robotics, medicine…), the neuron circuit dynamics should be compatible with the ones of BN. To account for their different characteristics, a reference model proposed in Wei et al. (2014) was used. This model can be considered as an improved Hodgkin-Huxley model, which unifies neuronal dynamics from spikes to seizures using four state variables, aiming at a better understanding of some pathological states. To achieve a compact circuit and low energy dissipation for the biomimetic circuit variant, capacitance values of Cm = 50 fF and CK = 100 fF were used with no leak conductance introduced to the circuit. The leak component of the ML equations, however, exists in the form of the parasitic leakage current of the transistor MNK. As it can be easily observed, an extra component modeling the leak has the same effect as increasing the size of the transistor MNK (KCL at Vm node in Figure 3). The absence of an explicit component does not compromise the approximation of the mathematical model, but rather limits the available parametrization options. The “ionic” transistor widths were set as WNa = 0.6 μm and WK = 1.83 μm and circuit time constants in the millisecond range were achieved. The rest of the circuit parameters can be found in Table 1.

TABLE 1

Table 1. Circuit parameters.

The comparison between the circuit simulation (A), the reference model (B) and the experiment (C) is illustrated in Figure 7. The Nernst potentials (+55 mV, −92 mV) were applied for circuit biasing, while the reference model plots were created with the code offered with Wei et al. (2014). For the measured results, VSS was set to ground and the supply voltage to 200 mV. The neuron circuit and output was monitored through a buffer. In this comparison, an excitatory pulse current is applied for Ts = 10 ms. Since the reference model is based on normalized quantities per unit area (Iex = 20 μA/cm² and Cm = 1 μF/cm²) a common normalization coefficient, ξ = Iex/Cm, is used. Figure 7 shows that very similar responses are obtained. For each case, three spikes are generated during the excitation. A slightly decreasing spike amplitude is observed for the reference model, while a constant amplitude is observed for the circuit. The form of the first spike in the measurement differs due to the initial condition imposed by the excitation. Below the supply value of 200 mV, no spikes were observed. This is attributed to the lower transistor currents observed in the experiment, giving rise to a reduced voltage gain (a sufficient voltage gain is required for spiking, as explained in Section Static properties of inverters in subthreshold).

FIGURE 7

Figure 7. Biomimetic neuron response to a pulse input current. Output waveform from (A) biomimetic circuit simulation (B) reference model and (C) experimental response to an excitatory pulse current. The step voltage shown on top, controls the generation of the excitation current.

In order to analytically compare the simulation and the experimental results with the reference model, three parameters have been considered: the spiking time St (defined as the latency between the maximum of each AP and the beginning of the excitation), the peak to peak amplitude, Vpp, and the spike width, SW, defined at Vpp/2. These parameters are given in Table 2.

TABLE 2

Table 2. Spiking characteristics for a 10 ms excitation current.

In the reference model, Vpp varies from 116 to 99.7 mV while the biomimetic neuron generates spikes with constant peak-to-peak amplitude of 120 mV. This difference is due to the use of the Morris-Lecar model based on two state variables, instead of the four state variables for the Wei model. Nevertheless, a good agreement is observed between the biomimetic neuron circuit and the reference model as far as excitability and spike shape are concerned.

Figure 8 shows the excitability of the biomimetic neuron. The minimal observed frequency is around 20 Hz. As expected, the firing rate and power consumption increase with the excitatory current. For Iex = 120 pA, a maximal spike frequency of 1.2 kHz was obtained with a total dissipated power of 90 pW. According to Izhikevich (2007) this biomimetic neuron is categorized as a Type I neuron.

FIGURE 8

Figure 8. Spike frequency and power consumption over the excitation current for the biomimetic neuron.

The energy efficiency of the circuit is deduced from the power dissipation and frequency measurements. In Figure 9 it is plotted as a function of the excitation current for two cases: (i) when the whole average power (static and dynamic) is taken into account and (ii) when the standby power (i.e., power consumption at zero excitation current) is subtracted. For an excitation current higher than 30 pA, the dissipated energy per spike is roughly constant and therefore independent of the spike frequency. An energy efficiency value of 40 fJ/spike is obtained when considering the dynamic power consumption.

FIGURE 9

Figure 9. Energy efficiency vs. excitation current for the biomimetic neuron when the DC power is included or not from energy estimation.

It is worth mentioning that this value is several orders of magnitude lower than the energy efficiency of actual BN, as it is estimated in the 10 pJ range, from the ATP consumption in Attwell and Laughlin (2001), Lennie (2003) and Poon and Zhou (2011). This low energy dissipation obtained for the proposed AN can be interesting for SNN applications needing the integration of a large number of neurons.

Simplified Neuron

In this part, the acceleration factor compared to the biological real time (BRT) and the energy dissipation are investigated for the simplified neuron design. Indeed, an important aspect of the simplified neuron, described above, is the ability to operate at accelerated biological time i.e., at higher frequency than the one of the BN. In this context, a circuit was sized to reach the highest spike frequency combined to the lowest energy dissipation. For this purpose, the supply voltage VDD = 200 mV and VSS = 0 V was applied and special emphasis was made to downscale the transistor size: the gate widths were chosen in the sub-micron range (few 100 nm). According to the aforementioned constraints, ultra-low capacitance values were chosen for Cm and Ck values, (4 fF and 8 fF, respectively including parasitic capacitances associated to the interconnections).

The output waveform resulting from an excitatory pulse current is presented in Figure 10, demonstrating the ability of this neuron to be externally stimulated.

FIGURE 10

Figure 10. Example of output waveform under constant current excitation. Excitation current = 150 pA, Vpeak-peak = 112 mV, Spike frequency = 26 kHz.

In order to analyze the AN performance, both the spike frequency and the corresponding average power consumption (including the static and dynamic parts) have been measured as a function of the excitation current. For the maximum excitation current, a spike frequency as high as 26 kHz was obtained with a total power consumption of 105 pW. The plot of power consumption and spike frequency is shown in Figure 11 as a function of the excitation current.

FIGURE 11

Figure 11. Spike frequency and power consumption vs. excitation current for the simplified neuron variant.

As already discussed, the main objective of this work was to minimize the energy dissipation: a value is in the fJ/spike range was predicted by the circuit simulation. The energy efficiency was determined from measurements of the total dissipated power along with the spike frequency.

The energy efficiency is plotted as a function of the excitation current in Figure 12 in two cases as previously mentioned for the biomimetic variant of the circuit. The curves demonstrate that the energy efficiency does not significantly depend on the output spiking frequency and the experimental dissipation reaches values as low as 3 fJ/spike when only the dynamic power is considered.

FIGURE 12

Figure 12. Energy efficiency vs. excitation current measured for the simplified neuron variant.

Discussion

The most important feature of the proposed neuron circuit is the ultra-low energy consumption. This is mainly attributed to the operation under a minimal supply voltage, which entails low drain to source voltage and deep subthreshold operation. Reduced to one sixth of the standard value for this technology node, it actually approaches the membrane potential values met in biology. Meanwhile, the design achieves the desired characteristics with minimum explicit capacitance load. This combination leads to a dynamic power consumption of a few femtojoules per spike for both circuit variants and specifically as low as 4fJ/spike for the simplified neuron. The characterization described in the previous sections refers to neuron circuits that were realized as stand-alone. Performance is benchmarked against similar implementations in Table 3. The values reported for the power consumption include both static (DC) and dynamic (AC) power; hence, the energy efficiency is calculated accounting for the total dissipated power. An improvement of two orders of magnitude below the state of the art energy efficiency in Cruz-Albrecht et al. (2012) is observed, along with a silicon area decrease by more than one order of magnitude. This is a very encouraging result in the road to diminish the power consumption of SNNs. Following, we shortly discuss the effects of noise, temperature and supply voltage variation as well as relevant synapse technology.

TABLE 3

Table 3. State of the art performance of reported stand-alone artificial neurons.

Firstly, despite the quite low peak to peak spike amplitude (~ 100 mV), the noise, while being observable in Figures 7C, 10 did not seem to strongly perturb the output. If we consider an equivalent circuit for output stage of the neuron circuit composed of a conductance Gm and a capacitance Cm in parallel, the Root Mean Square (RMS) of the generated noise voltage is:

vm=kTnCm (23)

where Tn is the noise temperature of the conductance Gm and k the Boltzmann constant. Evidently, in contrast to energy dissipation, the noise voltage increases when the membrane capacitance decreases. This fact can pose an issue in performance for a low capacitance circuit. Biased under threshold, the transistor drain current is low and the main noise component is thermal. For an order of a few femtofarads (as in the simplified neuron design), a pure thermal noise (Tn = 300 K) will induce an RMS noise voltage around 1 mV. Such a value was confirmed experimentally, and the first conclusion is that the noise has little impact. It is also worth mentioning that in biology, RMS voltage fluctuations are also in the mV range (Rudolph and Destexhe, 2004) but these fluctuations are obtained for a much higher capacitance (several 10's of pF, see the discussion in the introduction of this paper). This could be explained by the fact that the noise temperature Tn in biology is higher than the one considered for the neuron circuit.

Secondly, considering the behavior of the neuron circuit with the variation of ambient temperature. The use of subthreshold operation denotes an exponential variation of the drain current versus temperature. For SNN applications, the stability of the resting potential versus temperature is essential. For that matter, let us consider the circuit shown in Figure 4, with no excitatory current. In steady state, (assuming VSS set to 0 V), MPNa and MNK are off and the resting membrane potential Vrest can be calculated as:

Vrest= GNaGNa+GK. VDD= 11+GKGNa . VDD (24)

By carefully looking at Equation (24), the effect of temperature variation mainly depends upon the ratio between the potassium and sodium conductance. Assuming roughly the same impact of temperature for each transistor (Alioto, 2010), a smooth variation of the membrane resting potential is expected.

Thirdly, the impact of the supply voltage variation is investigated. As clearly indicated in Equation (18), the spike frequency is proportional toexp (2VdηVt)=exp (VDD-VSSηVt) . Figure 13 shows the measured results of the output spike frequency as the supply voltage (VDD) is increased. A straight line is obtained for the logarithmic plot that demonstrates the exponential relationship that exists between the output time constant and the supply voltage. It is also worth mentioning that the acquired slope of 100 mV/dec is very close to the sub-threshold slope of the transistor. This plot is a good indication of the validity of the analytical model presented above.

FIGURE 13

Figure 13. Output spike frequency measurement vs. supply voltage variation for the simplified neuron variant.

Finally, we would like to emphasize the challenge related to compatible synapse technology. In Figure 14 an indicative synaptic circuit is shown, that can be used to connect the proposed neuron circuits. PMOS transistors can model excitatory synapses through additional sodium channels, while conversely inhibitory synapses would be represented by additional potassium channels and NMOS transistors. As widely supported by many works (Arthur and Boahen, 2006), in order to introduce plasticity, the excitatory and inhibitory post synaptic currents can be controlled by interposing “weight” transistors. In the context of synaptic plasticity, SNNs can feature unsupervised learning through the so-called “Spike Timing Dependent Plasticity” (STDP) rule (Bi and Poo, 1998; Markram et al., 2012). Though use of emergent technologies such as memristors to realize plastic synapses featuring STDP is promising (Yang et al., 2013; Prezioso et al., 2015), issues as cost, reliability and large volume production advocate for the well-established CMOS technology. The STDP scheme proposed in Arthur and Boahen (2006), is fully compatible with our artificial neuron technology and can be readily employed.

FIGURE 14

Figure 14. Indicative synaptic circuit.

Conclusion

In conclusion, this work presented the properties and experimental results of an artificial neuron circuit based on the Morris-Lecar model that was fabricated in 65 nm standard CMOS technology. Circuit design was inspired from the ML neuron model equations and an analysis is offered in order to formally demonstrate the analogy between the mathematical model and the circuit response. The design of the artificial neuron has been conceived to perform closely to the biological neuron and allow high-level integration. From a circuit design perspective, the choice of using transistors in deep subthreshold operation exposed three key points: (i) the exponential control of the drain current by the gate voltage, (ii) the possibility to easily realize the required non-linear functions (Tanh) by using inverters, and (iii) the operation under minimal supply voltage and related capacitance, which enabled important energy consumption savings. The circuit analysis, through closed form equations, points out several important characteristics. Primarily, achieving voltage gain is essential to match the high slope of current change met in biology. Secondly, the output frequency is expressed as a function of key circuit parameters. Though the ML model is well suited to produce an architecture having a biological relevance, it soon became clear that the resulting topology can be extended to various performance optimization goals. For instance, the design could be simplified to achieve further reduction in power consumption and silicon area. The measured results confirm the systematic approach and the orders of magnitude improvement in energy efficiency over prior state of the art motivate the use of the proposed circuit in energy-efficient spiking neural network architectures.

Author Contributions

AC initiated and supervised this study. VH, SH, FD, CL performed the circuit simulations and measurements. IS designed the chip and EM managed the tape-out submission. All authors discussed the results and contributed to the refinement of the manuscript. IS finalized the manuscript.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors acknowledge the “Métropole Européenne de Lille (MEL),” IRCICA and SATT Lille for the financial support. The personnel of the “Centrale de Caractérisation” of IEMN for its significant contribution to the electrical measurements of the chips. The authors also acknowledge the contribution of CEA-Leti staff for its permanent support to this work.

References

Alioto, M. (2010). Understanding DC Behavior of Subthreshold CMOS Logic Through Closed-Form Analysis. IEEE Trans. Circ. Syst. I Regul. Papers 57, 1597–1607. doi: 10.1109/TCSI.2009.2034233

Amzica, F., and Neckelmann, D. (1999). Membrane capacitance of cortical neurons and glia during sleep oscillations and spike-wave seizures. J. Neurophysiol. 82, 2731–2746.

Ananthanarayanan, R., Esser, S. K., Simon, H. D., and Modha, D. S. (2009). “The cat is out of the bag: cortical simulations with 10⁹ neurons, 10¹³ synapses,” in Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis. (Portland, OR). doi: 10.1145/1654059.1654124

Arthur, J. V., and Boahen, K. (2006). Learning in silicon: timing is everything. Adv. Neural Inf. Process. Syst. 18:75. Available online at: https://papers.nips.cc/paper/2859-learning-in-silicon-timing-is-everything.pdf

Attwell, D., and Laughlin, S. B. (2001). An energy budget for signaling in the grey matter of the brain. J. Cereb. Blood Flow Metabol. 21, 1133–1145. doi: 10.1097/00004647-200110000-00001

Basu, A., and Hasler, P. E. (2010). Nullcline-based design of a silicon neuron. IEEE Trans. Circuits Syst. I Regul. Papers 57, 2938–2947. doi: 10.1109/TCSI.2010.2048772

Bi, G. Q., and Poo, M. M. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464–10472.

Cao, Y., Chen, Y., and Khosla, D. (2015). Spiking deep convolutional neural networks for energy-efficient object recognition. Int. J. Comput. Vis. 113, 54–66. doi: 10.1007/s11263-014-0788-3

Cruz-Albrecht, J. M., Yung, M. W., and Srinivasa, N. (2012). Energy-efficient neuron, synapse and STDP integrated circuits. IEEE Trans. Biomed. Circuits Syst. 6, 246–256. doi: 10.1109/TBCAS.2011.2174152

Farquhar, E., and Hasler, P. (2005). A bio-physically inspired silicon neuron. IEEE Trans. Circuits Syst. Regul. Papers 52, 477–488. doi: 10.1109/TCSI.2004.842871

Golowasch, J., Thomas, G., Taylor, A. L., Patel, A., Pineda, A., Khalil, C., et al. (2009). Membrane capacitance measurements revisited: dependence of capacitance value on measurement method in nonisopotential neurons. J. Neurophysiol. 102, 2161–2175. doi: 10.1152/jn.00160.2009

Hasler, J., and Marr, H. (2013). Finding a roadmap to achieve large neuromorphic hardware systems. Front. Neurosci. 7:118. doi: 10.3389/fnins.2013.00118

Hodgkin, A. L., and Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. 117, 500–544.

Hynna, K. M., and Boahen, K. (2007). Thermodynamically equivalent silicon models of voltage-dependent ion channels. Neural Comput. 19, 327–350. doi: 10.1162/neco.2007.19.2.327

Indiveri, G., Chicca, E., and Douglas, R. (2006). A VLSI array of low-power spiking neurons and bistable synapses with spike-timing dependent plasticity. IEEE Trans. Neural Netw. 17, 211–221. doi: 10.1109/TNN.2005.860850

Indiveri, G., Linares-Barranco, B., Hamilton, T. J., van Schaik, A., Etienne-Cummings, R., Delbruck, T., et al. (2011). Neuromorphic silicon neuron circuits. Front. Neurosci. 5:73. doi: 10.3389/fnins.2011.00073

Izhikevich, E. M. (2004). Which model to use for cortical spiking neurons? IEEE Trans. Neural Netw. 15, 1063–1070. doi: 10.1109/TNN.2004.832719

Izhikevich, E. M. (2007). Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting. Cambridge, MA: MIT Press.

Joubert, A., Belhadj, B., Temam, O., and Héliot, R. (2012). “Hardware spiking neurons design: analog or digital?,” in The 2012 International Joint Conference on Neural Networks (IJCNN). (Brisbane, QLD). doi: 10.1109/IJCNN.2012.6252600

Kim, K.-H., Gaba, S., Wheeler, D., Cruz-Albrecht, J. M., Hussain, T., Srinivasa, N., et al. (2012). A functional hybrid memristor crossbar-array/CMOS system for data storage and neuromorphic applications. Nano Lett. 12, 389–395. doi: 10.1021/nl203687n

LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521, 436–444. doi: 10.1038/nature14539

Lennie, P. (2003). The cost of cortical computation. Curr. Biol. 13, 493–497. doi: 10.1016/S0960-9822(03)00135-0

Maass, W. (1997). Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10, 1659–1671. doi: 10.1016/S0893-6080(97)00011-7

Markram, H., Gerstner, W., and Sjöström, P. J. (2012). Spike-timing-dependent plasticity: a comprehensive overview. Front. Synap. Neurosci. 4:2. doi: 10.3389/fnsyn.2012.00002

Mead, C. (1989). Analog VLSI and Neural Systems. Reading, MA: Addison-Wesley.

Mead, C. (1990). Neuromorphic electronic systems. Proc. IEEE 78, 1629–1636. doi: 10.1109/5.58356

Misra, J., and Saha, I. (2010). Artificial neural networks in hardware: a survey of two decades of progress. Neurocomputing 74, 239–255. doi: 10.1016/j.neucom.2010.03.021

Morris, C., and Lecar, H. (1981). Voltage oscillations in the barnacle giant muscle fiber. Biophys. J. 35, 193–213.

Oh, K.-S., and Jung, K. (2004). GPU implementation of neural networks. Pattern Recognit. 37, 1311–1314. doi: 10.1016/j.patcog.2004.01.013

Patel, G. N., and DeWeerth, S. P. (1997). Analogue VLSI Morris-Lecar neuron. Electron. Lett. 33, 997–998. doi: 10.1049/el:19970686

Poon, C.-S., and Zhou, K. (2011). Neuromorphic silicon neurons and large-scale neural networks: challenges and opportunities. Front. Neurosci. 5:108. doi: 10.3389/fnins.2011.00108

Prezioso, M., Merrikh-Bayat, F., Hoskins, B. D., Adam, G. C., Likharev, K. K., and Strukov, D. B. (2015). Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64. doi: 10.1038/nature14441

Rössert, C., Straka, H., Glasauer, S., and Moore, L. E. (2009). Frequency-domain analysis of intrinsic neuronal properties using high-resistant electrodes. Front. Neurosci. 3:64. doi: 10.3389/neuro.17.002.2009

Rudolph, M., and Destexhe, A. (2004). Inferring network activity from synaptic noise. J. Physiol. Paris, 98, 452–466. doi: 10.1016/j.jphysparis.2005.09.015

Tripathy, S., and Gerkin, R. C. (2012). Electrophysiological Values of Cell Capacitance Across Neuron Types From Literature. Available online at: http://neuroelectro.org/ephys_prop/1/data/

van Schaik, A. (2001). Building blocks for electronic spiking neural networks. Neural Netw. 14, 617–628. doi: 10.1016/S0893-6080(01)00067-3

Waldrop, M. M. (2016). The chips are down for Moore's law. Nat. News 530:144. doi: 10.1038/530144a

Wei, Y., Ullah, G., and Schiff, S. J. (2014). Unification of neuronal spikes, seizures, and spreading depression. J. Neurosci. 34, 11733–11743. doi: 10.1523/JNEUROSCI.0516-14.2014

Wijekoon, J. H., and Dudek, P. (2008). Compact silicon neuron circuit with spiking and bursting behaviour. Neural Netw. 21, 524–534. doi: 10.1016/j.neunet.2007.12.037

Yang, J. J., Strukov, D. B., and Stewart, D. R. (2013). Memristive devices for computing. Nat. Nanotechnol. 8, 13–24. doi: 10.1038/nnano.2012.240

Yu, T., and Cauwenberghs, G. (2010). Analog VLSI biophysical neurons and synapses with programmable membrane channel kinetics. IEEE Trans. Biomed. Circuits Syst. 4, 139–148. doi: 10.1109/TBCAS.2010.2048566