Merge branch 'main' of forge.bexus.org:D3D/d3direna

etc/inittab

@@ -54,6 +54,11 @@ po::powerokwait:/etc/init.d/powerfail stop
 
 # Start the irena DAQ in a screen session
 # The watchdog on tty2 takes care that this is running properly
+
+# Runlevel 2: maintainance without irena
+# Runlevel 3: daq running with watchdog
+# Runlevel 4: daq running without watchdog
+
 gpio:234:wait:/home/etd3d1/d3d/d3direna/gpio.sh
 d3d:34:respawn:sudo -inu d3d screen -U -S IRENA -D -m /home/etd3d1/d3d/d3direna/d3d.py
 1:2345:respawn:/sbin/getty -J -a root 38400 tty1

etc/watchdog.md

@@ -0,0 +1,120 @@
+# Dostel 3D startup and watchdog
+
+The upboard single board computer indide the D3D unit runs Debian `sid`
+with `sysv-init` and a bare minimum of system services.
+
+## Kernel
+
+The Linux kernel is build with with required hardware biultin and boots
+without `initrd`.  The bootloader is `grub2`.  The kernel is selected
+by a script `/etc/grub.d/08-upboard` as first boot option that boots
+automatically.
+
+
+## Network Configuration
+
+The network is configured with `ifupdown` in the file
+`/etc/networl/interfaces`, interface `eth0`.  The bootloader must pass
+the option `net.ifnames=0` to the kernel commandline, both in
+`/etc/default/grub` as well as `/etc/grub.d/08-upboard`.
+
+The interface definition starts the firewall
+`/etc/network/firewall.nft` when the network comes up.
+
+The firewall blocks broadcast packets via a blacklist.  A whitelist
+allows all traffic from select hosts at CAU Kiel.  The whilelist
+includes `asterix`, `asterix3`, `falbala`, and `etbs`.  The last,
+`etbs` performs daily backups of the whole filesystem.
+
+`icmp`, `ssh`, and `ntp` traffic is allowed from anywhere.  Everything
+else is blocked.  The filters are applied in both incomiung and
+outgoing direction.
+
+## System Boot
+
+The `sysv-init` is configured in `/etc/inittab`.  The Debian
+configuration is modified to 
+- start a `root` shell on `tty1`
+- initialise the `gpio` in runlevels 2, 3, and 4.
+- start (respawn) the watchdog in runlevel 3,
+- start (respown) the data acquisition in runlevels 3 and 4.
+
+The default boot entry is configured to boot into runlevel 3
+(`/etc/grub.d/08-upboard`).
+
+Booting or switching (`$ sudo init 2`) to runlevel 2, disables the
+watchdog and stops the data acquisition.  Runlevel 4 disables the
+watchdog but runs the data acquisition.
+
+In the files `/etc/init.d/checkroot.sh` and `/etc/init.d/checkfs.sh`
+add a timeout to `sulogin --force`, so we at least try to continue
+unattended even when the root filesystem is in trouble.
+
+## Watchdog
+
+The watchdog hardware of the upboard is used with the Debian package
+[watchdog](https://sourceforge.net/projects/watchdog/) to reset and
+reboot the computer after 60 seconds when nobody kick the dog.  The
+watchdog program configured via `/etc/watchdog.conf` kicks the dog as
+long as all of these tests pass:
+- the file `…/irena-status` updates at least every 10  minutes,
+- the file `…/irena-live` updates at least every 10  minutes,
+- the process that id is given in `…/irena.pid` is alive,
+- the CPU core temperatures are below 70°C.
+
+## Data Acquisition
+
+The Irena is controlled and read out with the program `irena.c`,
+configured via `d3d.rc`.  When it starts up it writes its process id
+into the file `d3d/irena.pid`.  The status of the data acquisition
+process is written every minute into the file `d3d/irena-status`.
+Both of these are used by the watchdog to verify that the program is
+running.  The `irena-status` file is further analysed to verify that
+the data acquisition operates properly, see below.
+
+Raw binary data files and histogram files are written into
+hourly files with rate limits applied based an data priorities.
+
+The program listens on a unit socket `d3d/irena-control`.  Connecting
+to that socket via `netcat` allows to command the program.  Connecting
+to the screen session where the program is running allows
+to command directly via the console.
+
+
+
+## Start, Monitor and Failure Recovery
+
+The script `d3d.py` is started inside a screen session directly from
+`init` in runlevels 3 and 4.  This screen session is killed when
+switching to a different runlevel.
+
+The script shall
+- start `d3d.rc` in a new screen window,
+- monitor is operation,
+- update the file `d3d/irena-live` every minute when all looks well.
+- try to fix any irregularities:
+   * reboot and reconfigure the FPGA,
+   * reboot the µC,
+   * restart `d3d/rc`.
+- drive the LEDs:
+   * the green LED shall pulse depending on the trigger rate, and
+   * the yellow LED shall indicate warning or error (blinking)
+     conditions.
+- read and report the primary supply current.
+
+The `irena.c` program is configured to reboot the µC when certain
+error conditions persist for a certain time.
+
+## GPIO
+
+The GPIOs of the upboard are used directly without Raspberry Pi
+emulation mapping, straight from the `/sys/` filesystem:
+- the UART is enabled as `/dev/ttyS1` in the bias,
+- two pins are are configured as GPIO outputs driving the `Reset` and
+  `EINT1` pin of the Irena µC,
+- two PWM circuits are enabled and drive the LEDs, and
+- the ADC reads the primary supply current.
+
+The script `gpio.sh` run by `init` as root, initialized these IOs,
+assigns permissions to access them as group `gpio`, and creates
+symlinks in `/run/gpio/`.