Practical Implementation of an Analogue and Digital Electronics System for a Modular Cosmic Ray Detector—MCORD

Abstract

A Modular COsmic Ray Detector (MCORD) was prepared for use in various physics experiments. MCORD detectors can be used in laboratory measurements or can become a part of large measurement sets. MCORD can be used as a muon detector, a veto system, or a tool supporting the testing and calibration of other detectors. MCORD can also work as a stand-alone device for scientific and commercial purposes. The basic element of MCORD is one section consisting of eight oblong scintillators with a double-sided light reading performed by silicon photomultipliers (SiPMs). This work presents a practical description of testing, calibration, and programming of analogue and digital electronics modules. The characterisation and calibration methods of the analogue front-end electronic modules, the obtained results, and their implementation into an operating system are presented. In addition, we describe the development environment and the procedures used to prepare our kit for practical use. The architecture of the FPGAs is also presented with a description of their programming as a data-collecting system in a simple coincidence circuit. We also present the possibilities of extending the data analysis system for large experiments.

1. Introduction

The idea of building a new cosmic ray detector was born in 2018 in connection with the construction of the MPD (Multi-Purpose Detector) [1,2] for the NICA collider (Dubna, Russia). The currently designed system is being prepared for use in other large experiments such as NA61/SHINE, for educational and training purposes, and for commercial use in the industry. From the very beginning, the detector was designed as a universal and modular system. Its design is adapted to work in simple laboratory systems and is easily expandable to multi-channel measurement systems in large physics experiments. The MCORD detector (Modular COsmic Ray Detector) [3,4,5] can be used as a muon and charged particle detector, or a muon veto system. It can be used for calibrating and testing other sub-detectors, as a muon identifier, trigger for complex measurement systems, and as a tool for astrophysical measurements. The detailed design and operation of the detector have been described in the CDR (Conceptual Design Report) documentation and published in [6]. The present paper describes practical issues related to analogue and digital electronic systems designed for this detector.

2. The MCORD Detector

2.1. Detection System

The detection system is based on plastic scintillators with a built-in single optical wavelength shifting fibre type Y-11(200) from Kuraray [7], with a diameter of 2 mm placed along the scintillator in the middle of one of the surfaces. The double-sided readout of scintillation light and its conversion into an electrical signal is performed by Hamamatsu S13360-3075FE [8,9] silicon photomultiplier (SiPM) with a matrix size of 3 × 3 mm${}^2$, 1600 pixels with a pixel size of 75 $\mathsf{\mu }$m. The choice of a relatively large pixel size, with a smaller number of pixels in the matrix, was supported by the fact that we did not expect a lot of light (the number of photons detected is about 50–100), and we wanted to maximise detection efficiency, keeping the risk of saturation at a low level. The scintillators with a reflection layer and a paper cover made by NUVIA company [10] are placed in aluminium covers (Figure 1). Inside the aluminium profiles are also boards with analogue front-end (AFE) electronics modules. The size of a single scintillator is 1620 × 72 × 22 mm${}^3$; whereas, the size of an aluminium profile is equal to 1744 × 80 × 30 mm${}^3$. Each profile has two aluminium feet, enabling the detector covers to be attached to an installation frame. Both ends of aluminium profiles are closed with plastic end-caps specially printed on a 3D printer. The end cap is additionally secured against penetration by light using rubber seals and a sprayed layer of aluminium. The signals from both SiPMs are passed from one end of a detector using a dust- and light-tight USB-C connector, and further to MCORD HUB using a USB-C cable rated for 20 Gb/s (four high-speed differential pairs available).

Sixteen signals from eight detector slabs that make up one MCORD section are transmitted to one MCORD HUB. The HUB passively transfers the signals over two SAS (Serial Attached SCSI) cables to the digital part of the MCORD electronics. SAS cables can have a length of up to 100 m without any significant deterioration in signal quality. The MCORD section is the basic unit of our detector with which we can construct any measurement system, e.g., a set of two sections operating in a coincidence mode (Figure 2). Such a system can be used for testing and calibrating another detector placed between these sections using cosmic ray particles (muons). This simple example, together with the method of reading and analysing physical data, is described in detail in Section 5.4. However, MCORD is designed with much more complex and demanding measurement systems in mind.

The specific shape and size of scintillators in any given system result from particular measurement conditions. The ACORDE [11] cosmic ray detector for the ALICE experiment used long double-layer scintillators placed in parallel at considerable distances. This system was meant primarily to work as a trigger for the other large-sized detectors. It is triggered by cosmic muons coming mainly from the zenith direction. However, if the main constrain is the time resolution, and the planned detector is to be small in size, the aim is to make the scintillators as small as possible, such as in the miniBeBe [12] detector, where four SiPMs are to be installed on each square (2 × 2 cm${}^2$) scintillator. In yet another experiment, during which the muongraphy of the Egyptian pyramid is to be carried out using cosmic muons, the most critical parameter is the exact determination of the particle trajectory. For this purpose, two thin and relatively large square scintillators with a fibre optic grid and signal reading using SiPM sensors were designed [13]. In turn, if muons are used to scan small compact objects, several layers of longitudinal scintillators with a triangular cross-section can increase the accuracy of the muon scanning of objects [14].

In our case, the length and shape of scintillators were designed for a large cylinder-shaped coincidence system consisting of 84 MCORD sections surrounding the MPD detector [1,2] (see Figure 3). Long scintillators lower the cost of the entire structure by reducing the number of readout electronics at the cost of deterioration of the time resolution of the system. For this reason, we used thick (over 2 cm) scintillators to increase the amount of light generated during the passage of the particle. For the same reason, we equipped them with optical fibres to make them independent of signal attenuation in a long object. This design and size allows our detector to be used in many different situations and configurations, but, if justified, it is possible to adjust the size and shape of scintillators with focus on a given parameter. Such an adjustment does not require changes in the readout system and associated electronics.

2.2. Electronic System

Several complete acquisition commercial off-the-shelf (COTS) systems are available from vendors such as CAEN [15], and Vertilon. CERN and other research laboratories also develop multi-channel ASICs for detector purposes [16,17]. Moreover, dedicated multi-channel DAQ chips are available from vendors such as Texas Instruments [18] and Analog Devices, which were designed for ultrasound imaging applications but were shown to work with such detectors. However, they were designed for dense, multi-channel applications where the readout system resides close to the detector and usually lacks the precise time-stamping (a dozen of ps) required for ToF applications. MCORD application makes such acquisition systems unfeasible due to the physical distances between the SiPM and the AFE (several metres). Such long cables add several tens or hundreds of pF to the input of the amplifier, which significantly reduces the system’s SNR, bandwidth, and time resolution. Moreover, not all COTS solutions can inject calibration signals directly at the input of the AFE. This is essential for ToF applications such as MCORD. For this reason, the amplifiers were placed physically close to the SiPMs and relatively simple electronic boards were designed. Moreover, standard cables such as USB-C and SAS were re-purposed to lower the system cost. The data acquisition path relies mostly on COTS solutions (commercially available open hardware modules).

The electronic system of the MCORD detector consists of two basic parts: the analogue electronics system (AFE (Analogue Front End) with MCORD HUB), and digital electronics system for the analysis of the physical signal based on FPGA (Field Programmable Gate Array) modules, and using the microTCA (Micro Telecommunications Computing Architecture) platform. In Figure 4, we see a block diagram of the entire system. The light signal from the scintillator is converted into an electric one in the SiPM photomultiplier, amplified in the AFE systems and through the HUB (it collects the signal from eight detectors, i.e., 16 channels), and sent to a DAC (digital/analogue converter) on the FMC (FPGA Mezzanine Card) board. The digitised physical signal is analysed with the help of FPGAs on the AMC boards. One microTCA crate contains up to 12 AMC (Advanced Mezzanine Card) boards. The results of the physical data analysis are sent to the Data Acquisition database (DAQ) or the trigger system. The control data of the detector are stored in the Equipment Database (EqDb).

2.2.1. Analogue Front-End (AFE)

The MCORD AFE system consists of three types of electronic boards. They are all installed in close proximity to the scintillator inside an aluminium profile. The first and the smallest board is only for the SiPM photomultiplier and the temperature sensor. The board is attached using a short FFC cable to make adjustment possible. On the second (AFE External), there is an amplifier for one of the SiPMs. The third electronic board (Main AFE), in addition to the second SiPM amplifier, contains power supplies, a control processor, and a USB-C connector (Figure 5). The amplifiers had to be installed very close to the SiPM to limit the wiring capacitance. For this reason, commercial off-the-shelf solutions with remote amplifiers could not be used. Long wires and additional capacitance significantly reduce the SNR and timing resolution of the entire detector system.

The main task of the AFE system is to control the power supplied to the SiPM sensors via the LDO (Low Dropout regulator) system (Figure 6). This circuit converts the 70 V supply from the HUB to an appropriate level in the range from 48 V to about 64 V. Voltage regulation can be automatic based on the measured temperature (LM45 temperature sensor) with accuracy to a few mV. In addition, AFE enables SiPMs calibration and monitors and transmits the sensor’s operating parameters on an ongoing basis using a 12-bit converter (voltage, current, and temperature). All processes are controlled by the STM 32F072CBU6 processor [19]. The AFE system communicates with the HUB system using the CAN (Controller Area Network) protocol. The procedure for checking and calibrating the AFE systems will be described in detail in Section 3. The programming of the AFE devices is described in detail in Section 4.

2.2.2. HUB

The other part of the analogue electronics is the MCORD HUB (Figure 7). This system acts as an intermediary in the transmission of the physical signal. The physical signal is not modified, but instead it is converted from eight USB-C cables to two SAS cables. The main task of the HUB is to support the management of AFE systems by generating 5 V and 70 V supply voltages, generating calibration signals and synchronising the operation of eight AFEs (Figure 8). The HUB is also controlled by an STM32 type processor (F767ZIT6), and the procedures are programmed in a micro Python environment. The HUB communicates internally and with AFE systems via the CAN protocol. Each AFE system has its own individual CAN ID number. The HUB is managed via a LAN cable. The HUB can be supplied from a 12 V DC source or via a LAN cable from a PoE switch (Power over Ethernet). In addition to the LAN port, we can communicate with the HUB directly using micro-USB, RS-485 and CAN ports and protocols. In addition, the HUB is equipped with a micro SD card port, which can expand its memory or the way it is started. A more detailed description of the MCORD system electronics analogue part can be found in reference [20].

2.2.3. Digital System Modularity and Expandability

The physical signals are transmitted over SAS cables to the digital electronics part of the MCORD. The system is designed as modular and scalable, and it can be easily expanded. In the basic configuration, the system of two MCORD sections (2 × 8 scintillators) together with two HUBs is supported by one FPGA board (AMC), on which two systems with D/A converters (FMC boards) are installed. Such a set can handle up to 32 channels. The AMC board is installed in a miniature mTCA chassis (Figure 9 left). In the case of supporting more sections (more than 32 channels), the AMC boards are installed in the standard micro TCA system housing (Figure 9 right). A description of the architecture and programming of the digital part can be found in Section 5.

3. AFE Calibration and Stability

3.1. Motivation

After designing and manufacturing the AFE electronics, a series of measurements were performed to test the correctness of operation and to prepare calibration of A/D and D/A converters operating in these systems. This calibration is necessary so that when the user is setting or reading the physical quantities (voltage, current, and temperature), he/she can work on SI units, not ADC counts and DAC codes. The results described in this chapter are based on measurements made with eight detectors, i.e., eight sets of AFE electronics (Main and External) with sixteen SiPM sensors.

We started testing the properties of our AFE systems by checking the stability of operation, both short and long-term. We investigated the behaviour of AFE systems at different temperatures. To provide data traceability over the whole lifetime of a large experiment, there is a need for parameter monitoring and, possibly, a recalibration that should be easy to carry on, and is coherent with the previous settings. To this end, we develop a protocol that covers the following:

• calibration in SI units (V, A, °C);

• long-term tracking of the SiPM performance and degradation;

• online temperature compensation.

3.2. Materials and Methods

An AFE module is based on an STM32 microprocessor. It is controlled by a computer via MCORD HUB. Each AFE is controlling two SiPMs (Main—primary and External—secondary), which are realised by two power suppliers with a voltage range of 48–65 V, two voltmeters with 18.4 mV resolution and two ammeters with 2.43 nA resolution. Each SiPM is mounted on a small PCB board with a temperature sensor (see Figure 5) that provides the possibility to stabilise SiPM’s gain by adjusting the operation voltage according to temperature fluctuations (see Section 3.8).

Figure 10 shows visualisations of the prepared measuring stations. In the first case (left), it is a system with a complete detector slab (scintillator, SiPM sensors, AFE systems in an aluminium housing), connected with a dedicated USB-C cable to the MCORD HUB and controlled by a computer. In this case, the measurements are performed using the internal voltage, current, and temperature meters built into the MCORD AFE systems. Additionally, the ambient temperature near the detector slab was measured as a comparative value. In the second case (Figure 10, right), the AFE boards were removed and placed in a special measuring box. Thanks to this, it was possible to operate the traced system, such as by connecting an external Keithley electrometer (Model: 6517A), disconnecting the SiPM sensor, and connecting a precise laboratory resistor in its place. During the calibration procedure, the SiPM sensor was disconnected, and an external voltmeter or calibrated resistor was connected instead (see Figure 11).

3.2.1. Stability

First, a series of measurements were made to check the stability of the detector control system (in particular AFEs, SiPMs, and scintillator). These measurements examined the behaviour of our electronics just after turning on the power, the study of transients, and the time of temperature and voltage stabilisation. After connecting the power supply, the AFE electronics boards warm up themselves and the SiPM sensors. Since the stabilisation of the temperature of these detector slab elements is of decisive importance for the stability of the measurement results, the time needed to stabilise the operating parameters of these elements of the system was investigated. In addition to these measurements, the long-term behaviour of our system was investigated. This was to check whether the electronics maintained operating parameters during multi-day measurements. Based on these results, the warming-up time of the system was set to 30 min.

3.2.2. Voltage Calibration

The voltage of AFE power supply is set using DAC and controlled using an internal meter coupled with ADC. Both systems were calibrated to determine conversion coefficients from ADC count and DAC codes to volts, and back. Calibration was made without load (external resistor R was disconnected) by an external high-precision electrometer/voltmeter, Keithley 6517A. The internal impedance of the electrometer can be neglected (200 T$\Omega$).

3.2.3. Current Calibration

The current calibration procedure should be preceded by the voltage calibration. By connecting the external calibrated resistor R instead of SiPM, and generating already calibrated voltages 48–64 V, the internal current meter (10 $\mathsf{\mu }$A range with 12-bit resolution) may be tested. The external meter (Keithley) should be disconnected during the procedure as it generates high electronic noise and disturbs the calibration process. A series of calibrated resistors R${}_{\mathrm{s}}$ were tested (with a precision better than 1% in the 50–60 V range), namely 983 M$\Omega$, 478 M$\Omega$, and 10.48 M$\Omega$. Finally, the 10.48 M$\Omega$ resistor was chosen for the calibration measurements because its use gives the effective measured current falling in the middle of the range of the tested A/D converter. Precise measurements of resistors’ resistances were performed using Keithley electrometer 6517A.

During regular operation, SiPMs would be connected in the place of the resistor R, but during the voltage calibration, the SiPM socket remains empty (there is an open circuit, R = R${}_{\infty }$). During the current calibration, a well-defined standard resistance is attached to the SiPM socket, R = R${}_s$ = 10.48 M$\Omega$. In Figure 11, one can see the schematic diagram of the test stand for calibration measurements. Only one part of the AFE system (either main or external) was measured simultaneously.

3.2.4. Current–Voltage Characteristics

The measurements of the current–voltage characteristics of SiPM sensors were performed in two ways. The first reference measurement using an external meter was checked by the SiPM sensor itself, without the remaining components (AFE, scintillator). The second, less precise measurement was made using the internal ADCs of AFE. We then compared these two measurements to test whether the AFE system is sufficient to determine such characteristics and thus calibrate our detectors in the future during regular experiments.

A noteworthy difference between the laboratory setup and the detector system (see Figure 10) is the location of the AFE main (with an amplifier, power supplies, and control processor) near the slab end with a USB-C connector. The external AFE is located on the other end of the scintillator slab. This poses two important issues:

• The main AFE and its SiPM are exposed to heat from a more complicated electronic circuit, which raises their temperatures significantly above ambient temperature. The external AFE is not affected, so these two parts work in different environmental conditions.

• The length of the external AFE connection wires is about 2 m longer than the main AFE, which implies a signal degradation. While it is of no importance for the actual physical signal, the noise is significantly increased on the temperature sensor signal. Therefore, to achieve a reliable measurement, signal averaging is needed to filter out noise.

In the following sections, the results of the measurements will be presented and discussed.

3.3. Warm-Up and Long-Term Voltage Stability

The entire system (scintillator, SiPMs, and AFEs) is enclosed in an aluminium profile and closed at both ends using plastic endcaps. Therefore, the startup procedure requires time for the system to warm up to working temperatures. After turning on the power, the electronics slowly heat up and warm up the air inside the aluminium housing, which also works as a heat sink and dissipates excess heat. At the same time, the scintillator and other components are heated up. So, we have to wait until the entire detector board stabilises its internal temperature, taking into account the ambient temperature. It should be noted that the SiPM sensor is located about 5 cm from the electronic board, and placed in an additional plastic cover (box). This box, however, is not designed to isolate you from the ambient temperature change, but to protect it from mechanical injuries. Nevertheless, it also acts as a stabiliser that protects SiPM from rapid and short-term temperature changes. The SiPM active temperature stabilisation system was not placed on purpose, as it would significantly increase the system’s cost, complexity, size, and failure rate. Since there is no room for a complex system, and a simple one would not be effective, a better solution is to use temperature change compensation by changing the supply voltage, which is the solution provided in our system.

In the test conditions, AFEs needed about 30 min to warm up from around 22 °C of ambient temperature to 39 °C, where it stabilises (see Figure 12 and Figure 13). After about 30 min, when there is no significant change in the ambient temperature, the temperature inside the detector slab, and thus the temperature at which the SiPM sensor works, is constant within one degree.

The long-term voltage stability after starting the system is achieved after about 2 h, when it settles at a level of 1.2 mV of standard deviation (see Figure 14). After the startup, the voltage fluctuates by several mV and then slowly rises to the stable voltage (see Figure 14). Fluctuations at these levels are below the resolution of the internal voltmeters of the AFE. From its point of view, the voltage fluctuates just by one count of the ADC (Figure 15). In this figure, we can see the readings of the internal voltmeter during a very long period of time. It is visible that the voltage value fluctuates in the range of ±1 ADC count over a few days. This translates into voltage variations well below the threshold that is important for our SiPM-based circuit. On this basis, we can say that after the AFE operating conditions stabilise (maximum 2 h), the system maintains very high stability.

3.4. Voltage Supply Calibration

The next step in the AFE system analysis is to calibrate the D/A converters. As a result, the requested values may be provided by the user in SI units instead of DAC codes. Each AFE system is equipped with several converters with which, on the one hand, we read the measured values of voltage, current, and temperature on the SiPM sensor and, on the other hand, we set the voltage supply on a SiPM sensor. Despite the fact that converters on different boards are identical, they have some inherent differences due to the manufacturing process Therefore, we decided to carry out measurements and calibrations of each converter separately.

The measuring system for determining the calibration is shown in Figure 10 and Figure 11. In all cases, the value set or measured by the AFE was controlled by an external high-precision meter. In Figure 16, we plot the value expressed in bits as read with a given converter versus the voltage set for an exemplary AFE. One can see two curves, separately for the transducer for the main and external systems. Both characteristics are linear, except for a small kink at the edges of the voltage range. Both curves are characterised with the same slope; they differ slightly with offset. Figure 16 shows the result of a typical measurement, only slightly affected by the fact of whether or not a standard resistor (10.48 M$\Omega$) was connected in the place of the disconnected SiPM sensor. The voltage drop was estimated to be 0.1 V per 5 $\mathsf{\mu }$A of current, which corresponds to 20 k$\Omega$ of internal resistance of the power source. The calibration of voltage converters was performed without the resistive load using a meter, the resistance of which could be treated as infinite.

In Figure 17, one can see a comparison of the calibration coefficients (a, b) of the linear function $U_{\mathrm{AFE},\mathrm{set}}=a{U}_{\mathrm{req}}+b$ determined for many AFE systems. $U_{\mathrm{AFE},\mathrm{set}}$ is the DAC code that needs to be sent to the power supply for it to deliver the requested voltage $U_{\mathrm{req}}$. Values of $U_{\mathrm{req}}$ were determined using the Keithley voltmeter during calibration for the whole range of $U_{\mathrm{AFE},\mathrm{set}}$. The parameters are fitted to the linear part of the calibration curve shown in Figure 16.

Based on the measurements, a file with a database of obtained coefficients was prepared, along with information to which AFE they should be attributed. The file will become part of the AFE electronics software; so, the calibration coefficients for the converters will be automatically checked each time our detectors are started. If the program fails to find a value for a given AFE board (based on the CAN ID or a serial number of the STM processor on that AFE), the average value will be used. As can be seen from this graph (Figure 17), this is justified and will not contribute to a significant increase in error.

3.5. Internal Voltmeter Calibration

As was written in the previous section, the calibration coefficients were measured separately for the value of the voltage set on the SiPM sensor and the measured one. In this section, we show the results of the calibration measurements for the voltage read by the internal AFE voltmeter. Figure 18 shows the calibration result for an exemplary AFE board set. The fitted calibration curve corresponds to a linear function $U=a{U}_{\mathrm{AFE},\mathrm{read}}+b$, where U is the real voltage value given in volts, and $U_{\mathrm{AFE},\mathrm{read}}$ is the ADC code received from the internal AFE voltmeter. The distribution of those coefficients for individual AFEs is shown in Figure 19. As one can see, the variation of both coefficients is small (one should pay attention to the scale in the figure). As in the previous section, the determined average value will be used for AFE calibration in cases where the software fails to recognise the connected detector slab.

3.6. Internal Ammeter Calibration

The AFE ammeter calibration is carried out by using a model resistor with a precisely known value. The measuring stand was built as shown in Figure 11. The resistor was placed instead of the SiPM sensor, and the Keithley electrometer was disconnected. The ammeter calibration is performed only after the power supply calibration.

In Figure 20, one can see the calibration curves of the converter for exemplary AFEs, which performs an internal current measurement (values in ADC counts) compared with the reference value determined with a precise voltmeter. Again, the characteristics of this transducer are linear and very similar for all AFEs. In Figure 21, we can see a comparison of the coefficients a and b of the linear calibration function, $I=a{I}_{\mathrm{AFE}}+b$, for many AFEs. $I_{\mathrm{AFE}}$ is the ADC code received from the internal AFE ammeter, and I is the reference current measured in amperes. During calibration, the value of I is derived from the (already calibrated) voltage supplied on the calibrated resistor.

The red mark shows the mean calibration coefficients, which will be used if the AFE system does not have its own values listed in the database.

3.7. Breakdown Voltage Determination

The current–voltage (I–V) characteristic is the basic quantity describing the properties of the silicon photomultiplier (SiPM). To perform the model characteristics of SiPM sensors, we used an external, precise electrometer, in which, after setting the supply voltage, the current was immediately measured. In Figure 22, I–V curves measured for one SiPM sensor using a precise external Keithley meter (red dots) and internal meters of the AFE system (black dots) are superimposed to present the relative capabilities and compatibility of both systems. As one can see, the measuring range of the AFE system is limited; however, the I–V curves overlap in the operational range of our system (53.5–59.5 V). The maximum differences in this range do not exceed 10%, which is sufficient for the needs and confirms the possibility of using the AFE system to determine SiPM sensors’ operational parameters.

Using the precise Keithley meter, we determine the breakdown voltage $V_{\mathrm{BR}}$ by fitting a line to the part (three points) of the characteristic curve with the fastest relative increase (the steepest part shown in Figure 22 as blue line). The intersection of this line with the dark current level (the horizontal blue line) measured far below $V_{\mathrm{BR}}$ is the searched value of $V_{\mathrm{BR}}$.

Unfortunately, the limitations of the AFE system make it impossible to determine $V_{\mathrm{BR}}$ in the same manner as with the precise meter. Firstly, the initial range of the AFE ADC is non-linear, so the measurement of ADC count equal to 1 is very uncertain. Secondly, there are a few counts of noise in the AFE system. For this reason, a safe current measurement level would be about 10 counts. For arbitrary reasons, we doubled this value to 20 counts. This principle is often used in electronics to determine a safe operating range. A total of 20 counts corresponds to a current of about 150 nA for the AFE, with the highest offset of 110 nA (see values of coefficient b in Figure 21), which is too high to determine the dark current for voltages below $V_{\mathrm{BR}}$. Therefore, we defined an operational breakdown voltage $\widetilde{V_{\mathrm{BR}}}$ determined for the fixed value of 150 nA (green line in Figure 22) that can be reliably measured using all tested AFEs. Figure 23 presents the relation between $\widetilde{V_{\mathrm{BR}}}$ and $V_{\mathrm{BR}}$ for several SiPM sensors using the same AFE for all of them (the data are also presented in Table 1). As one can see, the value of $\widetilde{V_{\mathrm{BR}}}$ is higher than the $V_{\mathrm{BR}}$ by ∼1.2 V in most cases. However, in two cases, the difference is as high as 2 V.

Summarising, the measurement of $\widetilde{V_{\mathrm{BR}}}$ with the internal meter is performed when a reliable reading of the current value is achieved. Then, on this basis, we estimate $V_{\mathrm{BR}}$ based on differences from Table 1. The operating voltage $V_{\mathrm{op}}$ of the SiPM sensor is defined 3 volts above $V_{\mathrm{BR}}$ (in the range recommended by the manufacturer [9]):

(1)$V_{\mathrm{op}}=V_{\mathrm{BR}}+3\mathrm{V}=\widetilde{V_{\mathrm{BR}}}-\Delta V_{\mathrm{BR}}+3\mathrm{V}$

Figure 24 shows current–voltage characteristics of several different SiPM sensors. We can see that they are grouped into groups, which, in fact, correspond to different production batches of sensors. SiPMs designated with numbers in the range 0101–0153 have different breakdown voltages determined at the factory than SiPMs with numbers 0154 and above. The difference is also visible in Table 1, especially in values of $I_{\mathrm{FOP}}$.

In Figure 25, we can see the current–voltage characteristics for a single SiPM sensor made at different temperatures by the precise ammeter. The horizontal green line indicates the current corresponding to $\widetilde{V_{\mathrm{BR}}}$ breakdown voltage determined using internal AFE meters. It should be noted that this value is around the intersection of all curves, i.e., at a voltage for which the current is similar for a large range of temperatures. Therefore, $\widetilde{V_{\mathrm{BR}}}$ has a very small dependence on the sensor temperature.

3.8. Temperature Monitoring

Temperature compensation of the SiPM gain is a challenging procedure. Therefore, the current requirement is to place the MCORD detector in a well-controlled environment. As the detector signals are not meant for spectroscopy, but for the discrimination between different radiation components and precise timing (for track positioning), the demand for the precision and stability of the gain is not very high. Therefore, the daily change of the ambient temperature by 2–4 °C is acceptable. Temperature fluctuations in such a small range do not translate into substantial changes in the breakdown voltage; therefore, the bias voltage does not require frequent adjustment. According to the factory data, the correction of the SiPM sensor supply voltage should be 52 mV per one degree of temperature change. Our own measurements in the climatic chamber gave us a value of 48 mV per 1 °C. Thus, with such small assumed changes in temperature (measurements are carried out in special air-conditioned rooms), and without a working voltage correction system, we can assume that the working conditions of SiPM sensors are stable.

Ultimately, our system will be equipped with an automatic voltage calibration loop with bias adjustment. For this purpose, each SiPM sensor has its own temperature sensor (LM45) mounted on the electronic board at a very short distance (less than 1 cm). This small electronic board, together with the SiPM and the temperature sensor, are closed in a small plastic box, which protects the system against short-term temperature changes. The whole thing, together with the activator and the AFE system, is enclosed in an aluminium profile, which ensures effective, quick, and even heat dissipation to the outside. Thanks to this, the temperature inside the profile stabilises. Below, we present preliminary results of measurements examining the temperature stability of our system. The influence of temperature fluctuations on the operating parameters of the SiPM sensor are shown in Figure 26 and Figure 27. The data were collected during three consecutive days. The room was exposed to changes in the outdoor temperature in the day–night cycle. The main parameter that was monitored was the breakdown-voltage $\widetilde{V_{\mathrm{BR}}}$ of SiPM sensors, determined using internal AFE electronics meters. The measurement was performed in the automatic mode, where, at a given time interval, the AFE system performed measurements which set the current–voltage curve of a given SiPM sensor. On this basis, the voltage $\widetilde{V_{\mathrm{BR}}}$ of the SiPM sensor was determined, which corresponded to 150 nA of current. In Figure 26, we can see a comparison of the changes in the value determined in this way $\widetilde{V_{\mathrm{BR}}}$, with the change in temperature measured by the internal AFE meter, and an independent external meter showing the temperature of the housing. We can see that the changes in these three parameters correspond and are proportional to each other. The observed temperature change by 16–17 °C corresponds to a change in the $\widetilde{V_{\mathrm{BR}}}$ voltage value of 0.3–0.4 V. In Figure 27, one can see the distribution and dependence of all $\widetilde{V_{\mathrm{BR}}}$ measurements made during these three days, with respect to the corresponding temperature. A temperature change of 100 ADC counts corresponds to a temperature change of about 8 °C. This dependence is a quadratic function, not a linear one like the dependence of the SiPM sensor itself [9], because here the temperature effect on the SiPM sensor and the AFE electronics, and, above all, D/A converters are added together.

The obvious conclusion resulting from these measurements is that the MCORD detector should work in a room with a stabilised temperature, or the automatic correction loop of the SiPM sensor supply voltage should be added to compensate for temperature-related changes in its gain. We can also conclude that the temperature measurement system in our AFE works correctly and can be used for the operation of the automatic correction loop. However, a detailed investigation of the dependence of the SiPM gain on the SiPM temperature measured by the AFE system must be completed first.

4. AFE Firmware Functionality

The AFE (Analogue Front End) module is directly connected to the two SiPMs located around one scintillator. The role of the module is to form the output pulses from SiPMs. These pulses have a reduced bandwidth and, thus, increased duration. The bandwidth limitation reduces hardware requirements for the readout equipment and minimises RF interferences [20].

The AFE module additionally controls the SiPMs parameters, such as the voltage, current, and temperature. Based on them, it ensures electronic protection of the SiPMs. The basic functionality of the AFE is depicted in the schematic in Figure 6.

The AFE module uses an STM32 microprocessor—STM32F072CBU6 [19]. The microprocessor with the core Cortex-M0 contains 128 kB of RAM, 16 kB of Flash, 12-bit ADC with 16 channels, 12-bit DAC with 2 channels, and supports communication interfaces such as SPI and CAN. It is clocked with a 48 MHz internal oscillator. The software is written in ANSI C language using HAL libraries. GCC toolchain is used for development.

The main functionality of the AFE board includes control of

• two temperature sensors of SiPMs (located on the same board, a few mm from each SiPM);

• two LDOs (Low-DropOut regulators) dedicated to SiPMs voltage controllers;

• two SiPMs calibration blocks;

• two SiPMs signal transmitters;

• CAN network driver;

• external memory interface for remote firmware update.

The AFE is connected to the HUB by the connector, where the following signals are available:

• CAN interface (both directions);

• calibrations signals (to AFE);

• SiPMs analogue signals (to HUB);

• power delivery (to AFE).

4.1. Can Communication with Hub Module

The AFE uses a CAN communication interface to exchange data with a HUB. CAN interface is used in version 2.0A and works with a 100 kHz baud rate. The CAN Standard Data Frame can be seen in Figure 28. In one frame up to 8 bytes can be sent.

Although the CAN interface allows working in multi-node topology, for our purposes, only the HUB is treated as a primary node. HUB always starts a messaging action, and AFE answers. One HUB is connected to eight AFE modules.

The CAN interface provides access to all AFE functionalities, settings, and data. The designed communication protocol contains a definition of all transactions between AFE and HUB. In the CAN frame, the first two bytes of the data field define the request (function individual code), and the next words define the arguments. When AFE receives a frame based on values in the first two data bytes, it decodes it and prepares a response.

The CAN bus node identification is based on IDs. Every node connected to the bus has its own unique ID. The ID of AFE contains information about the location of the section, HUB number, and channel number of HUB where AFE is connected. Additionally, filters were defined for the CAN interface, which allows for the processing of frames with specific IDs only.

4.2. Basic Functionality and Can Transactions

Below are, described in detail, all functionalities of the AFE module with corresponding CAN frame requests.

4.2.1. AFE Control Flow

The AFE module contains two LDO regulators, which deliver voltage to both SiPMs. The LDOs can generate a voltage from the 70 V rail, which comes directly from the HUB module. The converters are enabled when the signal enable is high. The enable signals of DC/DC converters are controlled by the CAN frames coming from the HUB.

The AFE module verifies signal path parameters such as gain or bandwidth by connecting as source calibration pulses instead of SiPMs. Calibration pulses can have two sources—internal and external. Internal pulses are generated by the HUB processor; external pulses are generated by an external source connected to the HUB via an SMA connector. Switching the source of signals from SiPMs to the test pulses can be performed by sending the correct CAN frame and setting proper bits in the control register. The definition of bits in the control register is listed in Table 2. The definitions of CAN frames for setting bits in the control register are listed in Table 3.

4.2.2. SiPM Voltage Setting

The initial SiPM voltage value can be set only by the CAN protocol. D/A converter used in the design is 12-bit width which allows for setting 4096 voltage levels. The output of the D/A converter is in the range of 0 V to 3 V. Then, this value is used in the feedback loop in the DC/DC converter, which, based on the feedback value, generates a final high voltage in a range between 48 V and 64 V. The value of 0 V at the output of the D/A converter corresponds to 64 V, and 3 V at the output of the D/A converter corresponds to 48 V. Finally, the resolution of the output SiPM voltage is around 3 mV. For easier manipulation, the CAN frame takes, as an argument, voltage values instead of raw binary DAC codes. Then, the HUB recalculates it and sends it in the frame to the AFE binary raw value. The definition of the CAN frame used for setting voltage values in the control register is listed in Table 4.

4.2.3. Analogue Data Readout

The microprocessor on the AFE uses eight independent channels (inputs) of an internal 12-bit A/D converter to control SiPM parameters. Channel 0 and 1 are used to verify the voltage delivered to both SiPMs. The raw binary value is read and can be transferred to the HUB. In the HUB, the binary value is recalculated to the voltage value. Recalculation is based on a linear function where coefficients were chosen experimentally. Channels 2 and 3 are used to verify offset voltage in the signal path.

Channels 4 and 5 measure the current of the SiPMs to verify if they are in the limits range. The range of the measurement is configured by hardware. The value of 0 V at the ADC input means no current. This is equal to 0 in the raw binary value scale. The value of 3 V, which is equal to 4095 in a raw binary value scale, means the current flow is around 10 $\mathsf{\mu }$A.

Channels 6 and 7 measure the temperature of SiPMs. The temperature verifies the conditions of the SiPMs and has an impact on the delivered voltage. The LM45 sensor is used as a temperature sensor. The output value is linear with a slope of 10 mV/°C. The value of 0 V is equal to 0 °C, and 1 V is equal to 100 °C.

All transactions used to send measured values are listed in Table 5.

4.2.4. Offset Setting Procedure

The digital potentiometer is used to set the offset of the SiPMs’ signal paths. The microprocessor communicates with the potentiometer by SPI interface. The offset has to be set to a voltage level, which ensures processing pulses from SiPMs without losing the quality of the signals. Negative phenomena such as overshoots, saturation, and amplification beyond the range should be removed by setting the correct offset. Additionally, with temperature floating, parameters of the measurement path can change, so, in such cases, offset should be changed to keep stable conditions for the chain. The transaction dedicated to manipulating the offset values is listed in Table 6.

4.2.5. Version

The firmware for the AFE microprocessor is still under development. The possibility to read the AFE firmware version gives knowledge about implemented functionalities in the module. The transactions which read the version number are listed in Table 7.

5. Data Acquisition (Sub) System

MCORD Data Acquisition Subsystem (DAQ) is responsible for digitisation and data processing. It is designed modularly, enabling it to address different use cases of MCORD, ranging from laboratory test setups to full-scale systems with hundreds of detectors.

5.1. Hardware Architecture Overview

A basic building block of MCORD DAQ is an FPGA Mezzanine Card (FMC)—FMC ADC100M 10B TDC 16cha [22]—that holds essential digitisation circuitry. For scalability, that card is used with a carrier module equipped with an FPGA and where application logic is implemented. A single FMC card can support up to 8 detector units (16 channels).

Functional separation into FMC and carrier allows for the adjustment of carrier complexity according to the application requirements. For example, testing and development can be performed using a standard, widely-available development kit such as Digilent Genesys 2 [23] (Figure 29).

If a setup consisting of 16 detector units is required, an AMC FMC Carrier Kintex (AFCK) [24] carrier board can host 2 FMC mezzanines and run in a single module mTCA Chassis (Figure 30).

For a scalable solution, a microTCA-based approach with AFCK carriers is envisioned. It enables support for up to 192 detector units per mTCA crate with the possibility of implementing inter-crate communication, allowing for the simultaneous operation of hundreds of channels. A dedicated microTCA Carrier Hub (MCH) mezzanine, Metlino, was developed for that purpose. It acts as a data concentrator, allows low-latency communication between modules, and is a source of timing synchronisation.

For applications requiring a lower number of channels, MCORD can be implemented using hardware developed for CERN Distributed I/O Tier (DIOT). This approach uses DIOT Kintex-Ultrascale Peripheral Board FMC Carrier [25] as an FMC carrier, along with DI/OT Zynq Ultrascale-based System Board [26]. Such a configuration will allow it to host up to 64 detector units with a single crate.

The choice of the scalable platform is application-dependent and based on factors such as required channel density, processing power, and funding constraints. The use of universal platforms such as mTCA or DIOT also enables the integration of different kinds of devices that may be necessary for a particular application (e.g., digital I/O cards, frequency synthesisers, etc.).

It is worth mentioning that the hardware infrastructure used in the MCORD DAQ subsystem is designed and published as Open Hardware. Consequently, all design files are publicly available on the online repositories [22,24,25,26]. It significantly lowers a vendor lock-in risk, makes custom modifications more affordable, and gives a significant insight into the operation internals of the whole application.

5.2. DAQ Hardware Building Blocks

5.2.1. FMC ADC 100M 10B TDC 16cha

FMC ADC100M 10B TDC 16cha (shown in Figure 30) is a basic building block of the MCORD DAQ. It is used in every implementation, despite the choice of the carrier and scaling scheme. The board is equipped with 16 channels. Every channel operates separately and consists of 100 MSPS ADC channels (Texas Instruments ADS5296) and TDC (ScioSense TDC-GPX2). Two SAS connectors connect the FMC module to MCORD HUB with SAS-SAS cables.

TDC is preceded by a constant fraction discriminator (CFD) that allows the generation of time-tagging on a constant fraction of the total impulse peak height rather than on threshold crossing. It is achieved by comparing the original impulse with added DC offset to its delayed copy. The required delay depends on the rising time of the impulse and is constant for a specific type of detector. For that reason, this parameter is fixed during module manufacturing. DC offset is runtime adjustable, individually for every channel using onboard DAC.

TDC chip, ScioSense TDC-GPX2, offers up to 35 MSPS with 20 ns pulse-to-pulse spacing. Depending on the chosen settings, offered temporal resolution can be as low as 15 ps. Noteworthy, the sampling rate and resolution are interchangeable, and, to have the lowest latency (and thus pulse-to-pulse spacing), it is required to decrease the resolution to 30 ps. This trade-off can be partially mitigated by combining several TDC channels together, which reduces system channel density. In such configuration, every MCORD HUB and every FMC module will serve eight detector units.

Clock circuitry allows using various reference signals for the clock distribution network. An onboard oscillator can be a clock source for a single FMC operation without any external reference. However, for multi-board operation, clock signals must be locked to a common clock. An onboard clock generator can be disciplined with either an external clock delivered to the MMCX clock input, or supplied from the FMC carrier via the FMC connector. The latter option is particularly useful when the multi-carrier operation is necessary (e.g., in a crate). A clock delivered with clock input allows for the synchronisation of several carriers that are used outside of scalable approaches such as stand-alone carriers or development boards.

5.2.2. AFCK Carrier and microTCA Chassis

MCORD DAQ, implemented with AFCK FMC carrier and mTCA Chassis, combines a hardware platform that is scalable with a cost-efficient and compact entry point setup. AMC FMC Carrier Kintex (AFCK) [24] (shown in Figure 31) is a microTCA module initially developed for high-energy physics applications and equipped with Xilinx Kintex 7 FPGA.

From the MCORD DAQ perspective, the most important features of AFCK are two FMC HPC connectors, comprehensive clocking circuitry, support for multigigabit communication and LVDS signalling via backplane connector and capable FPGA, allowing the implementation of vital system interfaces as well providing resources for data processing. The presence of two FMC connectors allows it to support up to sixteen detector units (thirty-two analogue channels) per carrier. Two FMC modules can work synchronously, having their clocking circuits disciplined with the carrier’s onboard high-quality digitally-controlled oscillators.

For multi-carrier crate operation, AFCK can forward mTCA clocks, distributed via the backplane, to FMC modules and thus provide a crate-wide synchronous operation. Full synchronisation requires length-matched backplane connections, and not every crate available on the market meets that requirement.

AFCK is an AMC module designed to be used in mTCA crates which makes AFCK-based designs easier to scale up. For systems with a channel number lower or equal to 64, it is more cost efficient to use a dedicated mTCA chassis. To address this issue, a special mTCA chassis was designed. It is a single AMC module chassis (Figure 30) that provides all necessary power supplies, exposes communication interfaces in SFP format, and allows remote device management using an Ethernet connection.

5.3. Firmware and Control Software

MCORD DAQ Subsystem uses an ARTIQ (Advanced Real-Time Infrastructure for Quantum physics) control system [27], both for firmware and high-level software. ARTIQ originates from quantum information experiments and is distributed as an open-source software, freely available to any interested party and founded by several scientific institutions worldwide.

ARTIQ features a high-level, Python-based domain-specific language (DSL) for a description of complex control systems and experiments (see an example in Figure 32). The code is compiled and executed on dedicated soft processors embedded in FPGA. The architecture of the control system provides nanosecond timing resolution and sub-microsecond operation latency. The execution of operations is guaranteed to happen in a precise moment by the FPGA logic design. Therefore, ARTIQ can be included in a group of hard real-time control systems.

Furthermore, output events can be precisely scheduled in time, and input events can be time tagged. Time tagging resolution can be as low as 1 ns for IO events and 8 ns for all others. This feature is used in MCORD DAQ to provide event timestamping.

ARTIQ also provides a consistent way to implement FPGA firmware along with its software, a Python-level counterpart. For managing experiments, there is a dedicated graphical user interface and a methodology for experiment version control and management of results. External laboratory equipment such as oscilloscopes, generators, or spectrum analysers can be easily integrated into experiment flow and controlled from ARTIQ DSL.

Despite an extensive list of features, a significant advantage and reason for utilising ARTIQ in MCORD DAQ is its heritage in many experiments, and its vibrant and qualified community.

5.3.1. FPGA Firmware Overview

Firmware for MCORD DAQ is an open-source project and is available on GitHub [28]. It is written using Migen [29] Fragmented Hardware Description Language (FHDL) and is expressed in Python. Migen is a toolbox for generating complex designs in programmable logic, and provides the benefits of modern and widely used language (e.g., mathematical libraries) to the process of firmware design.

Although firmware design using Migen is a relatively new concept, a note must be taken that it effectively generates hardware descriptions in well-known and recognised Verilog HDL. Therefore, Migen design uses standard FPGA vendor tools for synthesis and analysis. It can also be simulated and tested using renowned, commercially available simulators. Because of Verilog’s representation, Migen can easily integrate vendor-specific IP cores and protected IP cores from third-party vendors.

MCORD DAQ firmware uses ARTIQ as its implementation framework. As ARTIQ is a real-time control system, it provides predefined tools to ensure real-time operation. One such tool is the RT-Link interface. It acts as a bridge between user-defined logic and experiment. Operations to the logic can be scheduled with a base resolution of 8 ns and IO outputs with a resolution of 1 ns. RT-Link also provides a timestamped input interface with the same resolution.

Following good design patterns and having regard for scalability, MCORD DAQ firmware is divided into several modules (see Figure 33).

The FMC module provides support for the hardware interface to FMC ADC100M 10B TDC 16cha. Due to the use of hardware primitives specific to a particular FPGA family, this module is portable between FPGAs of the same family. Specifically, the existing implementation is dedicated to Xilinx Kintex 7. FMC data channels are combined to form logical groups representing a single detector unit.

Every detector connected to DAQ is handled by a single MCORD DAQ Channel module. The acquisition is triggered by signals generated by the Trigger Collector that implements the main part of the triggering subsystem and is responsible for the coincidence detection scheme. Application logic encapsulates application-specific logic, e.g., computation of statistical data or detection of specific combinations of events. Generated data are transferred to the PC or data concentrator (in the case of multi-carrier systems) for further processing using the External Communication module, which is implemented as the Ethernet connection (although different means can be easily integrated).

5.3.2. MCORD DAQ Channel

A single detector unit provides an analogue signal separately for each end (here denoted as A and B, which were previously named “main” and “external”). This signal is sampled with ADC, and impulse time is converted to digital form using TDC. Therefore, in a digital domain implemented in an FPGA, every detector unit provides four data streams: two streams of samples and two streams of timestamps.

ADC sample stream is used to detect physical events on the detector unit. The flow of data is depicted in Figure 34. Baseline Trigger Generators generate impulses based on the signal deviation from baseline, separately for every side of the detector unit. The coincidence between those signals indicates that a physical event took place at the detector unit, and this information is transferred to the triggering subsystem (Trigger Out). Such coincidence is detected with a configurable detection window (see Figure 35) that effectively is an elongated baseline trigger. The rising edge of the product of extended triggers from both ends of the detector unit defines the coincidence detection signal.

Because the decision of whether to use an event or not is computed in a triggering subsystem, ADC samples must be stored in a circular buffer until a decision on acquisition is available. Acquisition triggers will always arrive after the event takes place. Therefore, Circular Buffer modules define a pretrigger parameter that describes the start point of sample acquisition and the number of samples to acquire (see Figure 35).

Samples from Circular Buffer are transferred to Statistics modules. Those modules can calculate measures required by an application, e.g., impulse energy. For diagnostic purposes, raw samples can be stored using Samples Acquisition modules. Generated data, along with the trigger timestamp, are passed to the packetizer, where a packet describing an event is formulated. The Data packet is finally transferred from the detector support module to the upper layer, where it can be further processed or transferred to a different processing system (e.g., PC).

Data flow for TDC samples is similar to that of ADC (see Figure 36). Data collected from the converters is stored in a Circular Buffer module, and when the acquisition trigger is received, samples are transferred to the packetizer and then to the upper layers of the system. Additionally, a difference between TDC samples is calculated and appended to the transmitted data. For a properly configured system, only a single TDC sample is expected from each detector’s end. However, for debugging purposes, all samples appearing in the acquisition window are stored.

5.3.3. Triggering Subsystem

The main goal of the MCORD DAQ firmware is to detect the physical event and acquire data related to that event. Most basic event detection is performed in the scope of a single detector slab. Data from both ends of the scintillator unit are sampled with ADCs. Then FPGA firmware generates impulses based on the signal deviation from baseline, separately for every side of the detector unit. The coincidence between those signals indicates that a physical event occurred at the detector slab.

The events identified for a single detector can be used to detect events engaging multiple detector units with more complex coincidence rules, depending on the application requirements.

Inter-detector coincidence detection takes place in a module called Trigger Collector. Its implementation encapsulates application-dependent logic and, according to the scale of the system, can combine signals from a single carrier or multiple carriers distributed in a number of crates using high-speed, low-latency links.

Every event identified by the Trigger Collector is given an ID that, combined with a timestamp, allows for unique event identification. All data relating to the particular event are then tagged with the same ID and timestamp, allowing the compilation of information from different detectors.

5.4. 16-Detector Slab Setup

For limited applications, demonstration, and development purposes, we developed a 16-detector setup (2 MCORD sections). Such configuration can be supported by two MCORD HUBs and single AFCK carrier equipped with two FMC modules in a single-slot mTCA chassis. The demonstration setup arrangement is given in Figure 37.

In this configuration, a clock required for FMC operation is supplied from the FMC carrier’s onboard digitally-controlled oscillator. The clocking signal is fed to the FMC’s onboard trigger distribution circuit and distributed to ADCs and TDCs. For consistent time tagging, a common reset is delivered to the TDCs of both FMC modules. In effect, both FMC modules operate with the same time frame.

As only a single carrier is used, all data are processed by the carrier’s FPGA and no inter-carrier communication is implemented.

The operation of this demonstration setup is aimed at detecting particles crossing both detection planes (upper and bottom). Therefore, a dedicated, simplified triggering mechanism was implemented, and an example data flow was depicted in Figure 38. Channels are grouped in a way that every group represents a detection plane.

Every triggering impulse coming from the MCORD channel starts a counter that counts from some specified number (in Figure 38 it is 20) to 0. Those counters establish a coincidence detection window for events taking place on different detector planes.

Trigger output is enabled when the event is detected on both detection planes. In the example shown in Figure 38, this takes place when IDX is three and five. When the trigger is activated, corresponding down counters are cleared to zero and channels that take part in the trigger action are denoted with channel mask variable (CHMASK). That variable is associated with the trigger event and stored for future analysis. It is also an acquisition triggering mask for MCORD channels (i.e., only channels denoted with CHMASK will acquire data in the particular event).

The TRIGID variable is passed along the triggering signal to MCORD channel modules and is embedded into data packets to, along with the time tag, identify data from a single event.

Data from MCORD channels are downloaded to the host PC using an Ethernet connection and stored in a database for further analysis.

It must be noted that this simplified triggering system is intended only for demonstration purposes. It is not scalable to multi-carrier operation and has limited configuration capabilities (only coincidence window lengths are configurable, and the detection scheme is hard-coded).

For future multi-carrier applications, a different, packet-based triggering subsystem was designed. However, as implementation is heavily based on particular timing requirements of the target application, it was suspended until such information was available.

6. Conclusions

In this manuscript, we present the detector construction and electronics architecture of the the MCORD detection system for registering muons originating from cosmic ray interactions with the atmosphere. The methods for calibration of A/D converters and for determining the stability of the system operation are also given. Low-level programming of analogue AFE systems, as well as programming of FPGA systems dealing with the analysis of the measured signals and obtaining the trigger signal from our detector are described. In the simplest exemplary setup, presented in Section 5.4, the triggering signal is computed as a coincidence of the events detected on the upper and lower sections of the detector. For different setups, more sophisticated algorithms can be implemented.

The system requires a certain time (minimum 0.5 h, optimally 2 h) to stabilise the performance. The AFE systems require measurements to calibrate the A/D converters. Since the calibration functions of the transmitters differ very little, it is possible to use averaged calibration coefficients. The AFE’s built-in current meter sensitivity is insufficient to determine the value of the current, corresponding to the breakdown voltage $V_{\mathrm{BR}}$ of the SiPM sensors. However, we have shown that we can set operational breakdown voltage $\widetilde{V_{\mathrm{BR}}}$ that allows us to define the operating voltage $V_{\mathrm{op}}$ using a simple method and applying a Formula (1).

Using the STM processor and a programmable system ensures basic functionality, control, and safe commissioning of the AFE systems and the entire detector. Additional functionality provides error handling through appropriate system messages or LEDs (e.g., incorrectly connected cables or lack of communication with the module), and the implementation of advanced functions such as the SiPM operating voltage setting or temperature fluctuation recalibration loop. The temperature measurement circuits (sensors and ADCs) used in the AFEs are sufficient for the proper operation of the MCORD system in an environment with unstable ambient temperature. The communication between the AFE and HUB systems, and the control of the STM processor are carried out using the CAN protocol. The concept of frames was used, in which the first two bytes define the command, and the next words define the arguments. When AFE receives a frame based on values in the first two data bytes, it decodes it and prepares a response. The interface provides access to all AFE functionalities, settings, and data.

An FPGA Mezzanine Card (FMC)—FMC ADC100M 10B TDC 16cha—is a basic building block of MCORD DAQ that holds essential digitisation circuitry. For scalability, that card is used with a carrier module equipped with an FPGA and where application logic is implemented. A single FMC card can support up to 8 detector units (16 channels, one MCORD section). The analysis of recorded signals is based on the fulfilment of two basic conditions: the signal detected at the given end of the detector slab exceeds the low-level threshold set, and the signals from both ends of the detector slab are recorded in coincidence within the given time window of a few tens of nanoseconds.

Thanks to its modular construction, the MCORD can be used in simple detection systems, but it can also be extended to large scale instruments in various geometries, depending on the requirements of specific applications.

Author Contributions

Conceptualisation, M.B., G.K. and K.P.; methodology, A.B., P.K, M.S. (Mikolaj Sowinski) and W.M.Z.; software, M.K. (Michal Kiecana), P.K., M.K. (Michal Kuc), M.K. (Michal Kuklewski), M.S. (Mikolaj Sowinski) and J.S.; validation, M.B., M.P. and Ł.Ś.; formal analysis, A.B. and M.K. (Michal Kiecana); investigation, A.B., J.G., M.K. (Michal Kiecana) and M.K. (Michal Kuc); resources, A.D., J.G. and M.S. (Maciej Sitek); data curation, M.K. (Michal Kiecana); writing—original draft, M.B., A.B., P.K., M.K. (Michal Kuklewski), M.P. and M.S. (Mikolaj Sowinski); writing—review and editing, M.B., G.K., M.P., Ł.Ś. and J.S.; visualisation, E.J., M.K. (Michal Kiecana) and M.P.; supervision, M.B.; project administration, M.B.; funding acquisition, M.B. and A.S.-K. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data available upon request.

Acknowledgments

We would like to emphasize and thank our colleague Arek Chlopik (National Center for Nuclear Research, Poland), who died at the beginning of 2021, for his participation in the work on this detector.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ADC or A/D	Analogue to Digital Converter
AFCK	AMC FMC Carrier Kintex
AFE	Analogue Front End
AMC	Advanced Mezzanine Card
ARTIQ	Advanced Real-Time Infrastructure for Quantum physics
CERN	Conseil Européen pour la Recherche Nucléaire
CAN	Controller Area Network
CDR	Conceptual Design Report
CFD	Consant Fraction Discriminator
COTS	Commercial Off-The-Shelf
CPU	Central Procesor Unit
DAC or D/A	Digital to Analogue Converter
DAQ	Data Acquisition
DC	Direct Current
DIOT	Distributed IO Tier
DSL	Domain-Specific Language
EqDb	Equipment Database
FHDL	Fragmented Hardware Description Language
FMC	FPGA Mezzanine Card
FPGA	Field-Programmable Gate Array
HPC	High Pin Count FMC
LAN	Local Area Network
LDO	Low DropOut regulator
LVDS	Low-Voltage Differential Signalling
MCH	microTCA Carrier Hub
MCORD	Modular Cosmic Ray Detector
MMCX	Micro Miniature CoaXial
MPD	Multi Purpose Detector
NICA	Nuclotron-based Ion Collider fAcility
mTCA or microTCA	Micro Telecommunications Computing Architecture
PoE	Power over Ethernet
SAS	Serial Attached SCSI
SCADA	Supervisory Control And Data Acquisition
SCSI	Small Computer Systems Interface
SFP	Small Form Pluggable
SiPM	Silicon Photo Multiplier
SNR	Signal to Noise Ratio
SPI	Serial Peripheral Interface
TDC	Time To Digital Converter
USB-C	Universal Serial Bus Type-C

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Generate image description

Figure 1. Scintillator in an aluminium profile 1744 × 80 × 30 mm[Forumla omitted. See PDF.].

Generate image description

Figure 2. Two MCORD sections (grey) in coincidence mode. The setup used for testing another detector (red).

Generate image description

Figure 3. A cylindrical detector constructed of 84 MCORD sections.

Generate image description

Figure 4. Block scheme of the whole MCORD electronic system.

Generate image description

Figure 5. MCORD AFE electronic boards viewed from both sides. SiPM socket with installed SiPM detector (left, purple), main AFE (centre, blue), and external AFE (right, orange).

Generate image description

Figure 6. MCORD AFE layout.

Generate image description

Figure 7. MCORD HUB electronics board.

Generate image description

Figure 8. MCORD HUB idea scheme.

Generate image description

Figure 9. mTCA chassis for one AMC board (left) and standard mTCA rack for up to 12 AMC (right).

Generate image description

Figure 10. Visualisation of the measuring station using the complete MCORD detector slab (left), and when the AFE plates are separated from the detector slab to be used with an external precision Keithley meter (right).

Generate image description

Figure 11. Schematic diagram of the measuring station for calibration measurements (visualisation shown in Figure 10, right).

Generate image description

Figure 12. The heat-up curve of AFE no. 10 measured inside the housing using a thermocouple and Keithley 6517A electrometer.

Generate image description

Figure 13. Photographs of AFE electronics taken with a thermal imaging camera (Flir One by Teledyne) superimposed on regular image (with small misalignment due to parallax). The black crosses show the temperature measurement points. One can see the connected USB-C cable.

Generate image description

Figure 14. Voltage U during heat-up of main AFE measured using the Keithley voltmeter.

Generate image description

Figure 15. Long-term voltage stability measured using AFE internal voltmeter. Voltage is given in ADC counts.

Generate image description

Figure 16. DAC code that sets supply voltage, x-axis, vs. main and external AFE SiPM voltage (with and without 10.48 M[Forumla omitted. See PDF.] of resistive load), y-axis.

Generate image description

Figure 17. Distribution calibration coefficients across many AFEs for AFE voltage supply. The scale is limited to a narrow range to show small differences in presented values. The average values are marked with a red marker and given as an example below the formula.

Generate image description

Figure 18. AFE internal voltmeter calibration comparison between Main and External AFEs.

Generate image description

Figure 19. Internal voltmeter calibration coefficient distribution. The scale is limited to a narrow range to show small differences in presented values. The average value is marked with a red marker.

Generate image description

Figure 20. Main and external AFE ammeter calibration. The scale is limited to a narrow range to show small differences in presented values.

Generate image description

Figure 21. AFE ammeter calibration coefficient distribution. The scale is limited to a narrow range to show small differences in presented values. The average value is marked with a red marker.

Generate image description

Figure 22. Current—voltage characteristics measured using Keithley electrometer and AFE internal meters. The green line represents current level at which [Forumla omitted. See PDF.] is determined. The intersection of the blue lines determines [Forumla omitted. See PDF.].

Generate image description

Figure 23. Breakdown voltage [Forumla omitted. See PDF.] (determined using Keithley meter) vs. operational breakdown voltage [Forumla omitted. See PDF.] (determined using AFE meters).

Generate image description

Figure 24. Current–voltage characteristics for various SiMPs.

Generate image description

Figure 25. Current—voltage characteristics of SiMP under different temperatures.

Generate image description

Figure 26. Long-term correlation of the breakdown voltage [Forumla omitted. See PDF.], ambient temperature measured with a Keithley electrometer on the detector slab casing, and temperature measured by AFE near the SiPM. The ambient temperature was intentionally increased by placing the detector in sunlight during hot summer days. The calibration parameters of the AFE thermometer are taken from the vendor documentation.

Generate image description

Figure 27. Dependence of the [Forumla omitted. See PDF.] breakdown voltage on the SiPM temperature measured by AFE. The calibration parameters of the AFE thermometer are taken from the vendor documentation.

Generate image description

Figure 28. Fields of Standard CAN Frame. A detailed description of every field can be found in [21].

Generate image description

Figure 29. Minimal MCORD DAQ testing setup using FMC ADC100M 10B TDC 16cha and Digilent Genesys 2.

Generate image description

Figure 30. Setup of 2 FMC–ADC–TDC modules (green electronic boards) mounted on AFCK board inside mTCA Chassis (black box). AFCK and modules are temporarily ejected for presentation/maintenance purposes.

Generate image description

Figure 31. AMC FMC Carrier Kintex (AFCK) [24].

Generate image description

Figure 32. Fragment of the experiment described in ARTIQ DSL.

Generate image description

Figure 33. General overview of MCORD DAQ Firmware for single-carrier application.

Generate image description

Figure 34. ADC data flow.

Generate image description

Figure 35. Coincidence detection for single detector unit.

Generate image description

Figure 36. TDC data flow.

Generate image description

Figure 37. Demonstration setup.

Generate image description

Figure 38. Example data for triggering subsystem of demonstration application.

Table 1

The relation between normal and operational breakdown voltages ($V_{\mathrm{BR}}$ and $\widetilde{V_{\mathrm{BR}}}$, respectively), determined in this study. Values of breakdown voltage $V_{\mathrm{FBR}}$ and current at operational voltage $I_{\mathrm{FOP}}$ determined at the factory are presented for comparison. Last two rows present SiPMs from the second production batch.

SiPM ID	$\widetilde{{\mathit{V}}_{\mathbf{BR}}}$ [V]	${\mathit{V}}_{\mathbf{BR}}$ [V]	$\mathbf{\Delta }{\mathit{V}}_{\mathbf{BR}}=\widetilde{{\mathit{V}}_{\mathbf{BR}}}-{\mathit{V}}_{\mathbf{BR}}$ [V]	${\mathit{V}}_{\mathbf{FBR}}$ [V]	${\mathit{I}}_{\mathbf{FOP}}$ [A]
0101	53.93	52.64	1.29	52.43	0.566
0107	53.51	52.27	1.24	52.07	0.601
0110	53.12	51.88	1.23	51.67	0.589
0134	53.69	52.58	1.11	52.39	0.691
0137	54.12	52.84	1.27	52.63	0.552
0139	53.93	52.53	1.40	52.40	0.532
0145	53.21	51.98	1.23	51.78	0.604
0150	53.78	52.42	1.36	52.31	0.557
0181	53.91	51.87	2.04	51.64	0.283
0205	53.62	51.65	1.96	51.43	0.263

Table 2

Definition of bits in the Control Register.

Bit Number	Function
0	Enable/Disable DC/DC converter for SiPM A
1	Enable/Disable DC/DC converter for SiPM B
2	Enable/Disable signal path calibration pulses path for SiPM A
3	Enable/Disable signal path calibration pulses path for SiPM B

Table 3

List of CAN transactions for setting bits in Control Register.

Function	Function Individual Code	Arguments	Return Data
Set bits in a Control Register	0x0040	Bits to set	–
Clear bits in a Control Register	0x0041	Bits to clear	–
Get bits in a Control Register	0x0042	–	Bit values in the Register

Table 4

CAN transaction for setting SiPM Voltage.

Function	Function Individual Code	Arguments	Return Data
Set voltage for SiPM A and B	0x0012	DAC binary values to set voltage for SiPM A (2 bytes) and SiPM B (2 bytes)	–

Table 5

List of CAN transactions for reading Voltage, Current, and Temperature values.

Function	Function Individual Code	Arguments	Return Data
Get ADC data Reg. 1	0x0010	–	SiPM A Voltage (2 bytes), SiPM B Voltage (2 bytes), SiPM A Current (2 bytes)
Get ADC data Reg. 2	0x0011	–	SiPM B Current (2 bytes), SiPM A Offset Voltage (2 bytes), SiPM B Offset Voltage (2 bytes)
Get temperature values	0x0013	–	SiPM A Temperature (2 bytes), SiPM B Temperature (2 bytes)

Table 6

CAN transaction for setting offset for SiPM signal path.

Function	Function Individual Code	Arguments	Return Data
Set digital resistance	0x00A0	2 × RAW Digital Resistance values	–

Table 7

CAN transaction for reading AFE firmware version.

Function	Function Individual Code	Arguments	Return Data
Get version	0x0001	–	Firmware version (6 bytes)

References

1. Golovatyuk, V.; Kekelidze, V.; Kolesnikov, V.; Rogachevsky, O.; Sorin, A. The Multi-Purpose Detector (MPD) of the collider experiment. Eur. Phys. J. A; 2016; 52, 212. [DOI: https://dx.doi.org/10.1140/epja/i2016-16212-1]

2. Abgaryan, V.; Acevedo Kado, R.; Afanasyev, S.V.; Agakishiev, G.N.; Alpatov, E.; Altsybeev, G.; Alvarado Hernández, M.; Andreeva, S.V.; Andreeva, T.V.; Andronov, E.V. et al. Status and initial physics performance studies of the MPD experiment at NICA. Eur. Phys. J. A; 2022; 58, 140. [DOI: https://dx.doi.org/10.1140/epja/s10050-022-00750-6]

3. Bielewicz, M.; Grodzicka-Kobylka, M.; Mianowski, S.; Sibczynski, P.; Swiderski, L.; Szczesniak, T.; Linczuk, M.; Wielanek, D.; Kisiel, A.; Kasprowicz, G. et al. MCORD: MPD cosmic ray detector for NICA. Proceedings of the Photonics Applications in Astronomy, Communications, Industry, and High-Energy Physics Experiments 2018; Wilga, Poland, 3–10 June 2018; Romaniuk, R.S.; Linczuk, M. SPIE: Bellingham, WA, USA, 2018; Volume 10808, 1080847. [DOI: https://dx.doi.org/10.1117/12.2501720]

4. Bielewicz, M.; Milewicz-Zalewska, M.; Grodzicka-Kobylka, M.; Mianowski, S.; Sibczynski, P.; Swiderski, L.; Szczesniak, T.; Linczuk, M.; Wielanek, D.; Kisiel, A. et al. MCORD—MPD Cosmic Ray Detector a new features. EPJ Web Conf.; 2019; 204, 07016. [DOI: https://dx.doi.org/10.1051/epjconf/201920407016]

5. Bielewicz, M.; Kankiewicz, P.; Chlopik, A. Minimum Resolution of MCORD as a Consequence of Astrophysical Observation Requirements. Acta Phys. Pol. B Proc. Suppl.; 2021; 14, 543. [DOI: https://dx.doi.org/10.5506/APhysPolBSupp.14.543]

6. Bielewicz, M.; Bancer, A.; Barabanov, M.; Chlopik, A.; Czarnynoga, M.; Dabrowski, D.; Dudzinski, A.; Dziedzic, A.; Grodzicka-Kobylka, M.; Grzyb, J. et al. Conceptual design report of the MPD Cosmic Ray Detector (MCORD). J. Instrum.; 2021; 16, P11035. [DOI: https://dx.doi.org/10.1088/1748-0221/16/11/P11035]

7. Kuraray Plastic Scintillating Fibers. Available online: http://kuraraypsf.jp/psf/ws.html (accessed on 10 March 2023).

8. Grodzicka-Kobylka, M.; Szczesniak, T.; Moszyński, M. Comparison of SensL and Hamamatsu 4 × 4 channel SiPM arrays in gamma spectrometry with scintillators. Nucl. Instrum. Methods Phys. Res. Sect. A Accel. Spectrometers Detect. Assoc. Equip.; 2017; 856, pp. 53-64. [DOI: https://dx.doi.org/10.1016/j.nima.2017.03.015]

9. Hamamatsu. MPPC S13360-3050CS. MPPC for Precision Measurement, Photosensitive Area: 3.0 × 3.0 mm, Pixel Pitch: 50 μm. Available online: https://www.hamamatsu.com/eu/en/product/optical-sensors/mppc/mppc_mppc-array/S13360-3050CS.html (accessed on 19 October 2022).

10. NuviaTech Instruments. Nudet Plastic—Plastic Scintilation Detectors. Available online: https://www.nuviatech-instruments.com/product/nudet-plastic/ (accessed on 19 October 2022).

11. Fernández, A.; Gámez, E.; Herrera, G.; López, R.; León-Monzón, I.; Martínez, M.; Pagliarone, C.; Paic, G.; Román, S.; Tejeda, G. et al. ACORDE a cosmic ray detector for ALICE. Nucl. Instrum. Methods Phys. Res. Sect. A Accel. Spectrometers Detect. Assoc. Equip.; 2007; 572, pp. 102-103. [DOI: https://dx.doi.org/10.1016/j.nima.2006.10.336]

12. Kado, R.A.; Hernández, M.A.; Ayala, A.; Torres, M.A.; Bietenholz, W.; Chaires, D.; Cuautle, E.; Domínguez, I.; Guirado, A.; Maldonado, I. et al. The conceptual design of the miniBeBe detector proposed for NICA-MPD. J. Instrum.; 2021; 16, P02002. [DOI: https://dx.doi.org/10.1088/1748-0221/16/02/P02002]

13. Shereen, A.; Michael, T.; Basma, E.; Yasser, A.; Richard, T.K.; Ayman, M.; Alain, B.; Azaree, L.; Isar, M.; Ryan, P. et al. Simulation Studies of a Novel Muography Detector for the Great Pyramids. J. Adv. Instrum. Sci.; 2022; 2022, [DOI: https://dx.doi.org/10.31526/jais.2022.306]

14. Zhao, Y.; Luo, X.; Qin, K.; Liu, G.; Chen, D.; Augusto, R.; Zhang, W.; Luo, X.; Liu, C.; Liu, J. et al. A cosmic ray muons tomography system with triangular bar plastic scintillator detectors and improved 3D image reconstruction algorithm: A simulation study. Nucl. Eng. Technol.; 2023; 55, pp. 681-689. [DOI: https://dx.doi.org/10.1016/j.net.2022.10.031]

15. A5202 - 64 Channel Citiroc unit for FERS-5200-CAEN—Tools for Discovery. Available online: https://www.caen.it/products/a5202/ (accessed on 10 March 2023).

16. Calò, P.P.; Ciciriello, F.; Petrignani, S.; Marzocca, C. SiPM readout electronics. Nucl. Instrum. Methods Phys. Res. Sect. A Accel. Spectrometers, Detect. Assoc. Equip.; 2019; 926, pp. 57-68. [DOI: https://dx.doi.org/10.1016/j.nima.2018.09.030]

17. Callier, S.; Taille, C.D.; Martin-Chassard, G.; Raux, L. EASIROC, an Easy & Versatile ReadOut Device for SiPM. Phys. Procedia; 2012; 37, pp. 1569-1576. [DOI: https://dx.doi.org/10.1016/J.PHPRO.2012.02.486]

18. AFE5832 Data Sheet, Product Information and Support | TI.com. Available online: https://www.ti.com/product/AFE5832 (accessed on 10 March 2023).

19. STMicroelectronics. STM32F072CB - Mainstream Arm Cortex-M0 USB Line MCU with 128 Kbytes of Flash Memory, 48 MHz CPU, USB, CAN and CEC Functions. Available online: https://www.st.com/en/microcontrollers-microprocessors/stm32f072cb.html (accessed on 23 October 2022).

20. Kolasiński, P.; Poźniak, K.; Kasprowicz, G.; Zabołotny, W.; Sowinski, M.; Bielewicz, M.; Chlopik, A. Electronic readout system designed for MCORD in NICA experiment. Proceedings of the Photonics, Applications in Astronomy, Communications, Industry, and High Energy Physics Experiments 2020; Wilga, Poland, 31 August – 2 September 2020; Romaniuk, R.S.; Linczuk, M. SPIE: Bellingham, WA, USA, 2020; Volume 11581, 115811H. [DOI: https://dx.doi.org/10.1117/12.2580615]

21. Texas Instruments. Introduction to the Controller Area Network (CAN). Available online: https://www.ti.com/lit/an/sloa101b/sloa101b.pdf?ts=1666318983922 (accessed on 23 October 2022).

22. Electronics for High Energy and Quantum Physics Research Group. FMC ADC100M 10B TDC 16cha. Available online: https://github.com/elhep/FMC_ADC100M_10B_TDC_16cha (accessed on 23 October 2022).

23. Digilent, National Instruments. Genesys 2. Available online: https://digilent.com/reference/programmable-logic/genesys-2/start (accessed on 26 October 2022).

24. Open Hardware Repository. AMC FMC Carrier Kintex (AFCK). Available online: https://ohwr.org/project/afck/wikis/home (accessed on 26 October 2022).

25. Open Hardware Repository. DIOT Kintex Ultrascale-based Peripheral Board FMC Carrier. Available online: https://ohwr.org/project/diot-pfc-ku/wikis/home (accessed on 26 October 2022).

26. Open Hardware Repository. DI/OT Zynq Ultrascale-Based System Board with White Rabbit Support. Available online: https://ohwr.org/project/diot-sb-zu/wikis/home (accessed on 26 October 2022).

27. Bourdeauducq, S.; Whitequark,; Jördens, R.; Nadlinger, D.; Sionneau, Y.; Kermarrec, F. ARTIQ—A Leading-Edge Control System for Quantum Information Experiments. 2021; Available online: https://zenodo.org/record/6619071 (accessed on 10 March 2023). [DOI: https://dx.doi.org/10.5281/zenodo.6619071]

28. Electronics for High Energy and Quantum Physics Research Group. MCORD DAQ Firmware. Available online: https://github.com/elhep/MCORD_DAQ_Firmware (accessed on 26 October 2022).

29. M-Labs. Migen (Milkymist Generator). Available online: https://github.com/m-labs/migen (accessed on 26 October 2022).