For my recently installed PXACB I was looking for a way to remotely power it on and off. I found the Wi-Fi Smart Plug “HS100” and a blog post that it can be controlled from the command-line.

The referenced script uses captured results from wireshark and just re-transmits these messages from a shell script. In one of the comments someone points out that this is XOR’d JSON and how it can be decoded. Instead of a shell script I re-implemented it in Python and I am now always using XOR to encode and decode the JSON messages without needing to include the encoded commands in my script. This makes it easier to read the script and to extend the script.

The protocol used is JSON which is XOR’d and then transmitted to the device. Same goes for the answers. The JSON string is XOR’d with the previous character of the JSON string and the value of the first XOR operation is 0xAB. Additionally each message is prefixed with ‘x00x00x00x23’.

The message to turn on the power looks like this:

{
 "system": {
  "set_relay_state": {
   "state": 1
  }
 }
}

To find more about which commands the device understands I used the information I got from: Why not root your Christmas gift?

I downloaded the firmware for the US model of my smart plug and used binwalk to analyze the content of the firmware. The firmware contains busybox based ramdisk which includes the smart plug relevant programs /usr/bin/shd and /usr/bin/shdTester and it seems at least following commands exist:

  • system
  • reset
  • get_sysinfo
  • set_test_mode
  • set_dev_alias
  • set_relay_state
  • check_new_config
  • download_firmware
  • get_download_state
  • flash_firmware
  • set_mac_addr
  • set_device_id
  • set_hw_id
  • test_check_uboot
  • get_dev_icon
  • set_dev_icon
  • set_led_off
  • set_dev_location

With the knowledge from the original shell script implementation and the results from binwalk I wrote the following script: https://lisas.de/~adrian/hs100.py

Using this script I can power the device behind the smart plug easily on and off:

$ ./hs100.py -H p-pxcab.example.com off
$ ./hs100.py -H p-pxcab.example.com state
Power OFF
$ ./hs100.py -H p-pxcab.example.com on
$ ./hs100.py -H p-pxcab.example.com state
Power ON

The only annoying thing about the smart plug is, that it tries to communicate with some cloud systems so that it could be controlled from anywhere. After starting the smart plug it makes a name lookup for devs.tplinkcloud.com and connects to port 50443. I can connect to that system with openssl s_client -connect devs.tplinkcloud.com:50443 but what the smart plug actually sends to that system I do not know. If I do not block the smart plug in the firewall I see a NTP request after that and then the communication seems to stop. Right now the smart plug is blocked and does no NTP requests but it still tries to reach devs.tplinkcloud.com:50443 once a minute.

Quick bugfix release to address some issues with the audio backends: The user interface allowed selecting the PulseAudio backend, even when terminatorX was built without PulseAudio support. In addition the error message was not really helpful and PulseAudio was not set as default as it was intended.

These issues have been fixed with release 4.0.1, as usual you can find the tarball on the download page, the PPA builds are currently in progress – the resulting .deb packages should be available shortly.

A long time ago (2007 or 2008) I was developing firmware for Cell processor based systems. Most of the Slimline Open Firmware (SLOF) has been released and is also available in Fedora as firmware for QEMU: SLOF.

One of the systems we have been developing firmware for was a PCI Express card called PXCAB. The processor on this PCI Express card was not the original Cell processor but the newer PowerXCell 8i which has a much better double precision floating point performance. A few weeks ago I was able to get one of those PCI Express cards in a 1U chassis:

PXCAB

This chassis was designed to hold two PXCABs: one running in root complex mode and the other in endpoint mode. That way one card was the host system and the other the PCI express connected device. This single card is now running in root complex mode.

I can boot a kernel either via TFTP or from the flash. As writing the flash takes some time I am booting it right now via TFTP. Compiling the latest kernel from git for PPC64 is thanks to the available cross compiler (gcc-powerpc64-linux-gnu.x86_64) no problem: make CROSS_COMPILE=powerpc64-linux-gnu- ARCH=powerpc.

The more difficult part was to compile user space tools but fortunately I was able to compile it natively on a PPC64 system. With this minimal busybox based system I can boot the system and chroot into a Fedora 24 NFS mount.

I was trying to populate a directory with a minimal PPC64 based Fedora 24 system with following command:

dnf --setopt arch=ppc64 --installroot $PWD/ppc64 install dnf --releasever 24

Unfortunately that does not work as there currently seems to be no way to tell dnf to install the packages for another architecture. I was able to download a few RPMs and directly install them with rpm using the option --ignorearch. In the end I also installed the data for the chroot on my PPC64 system as that was faster and easier.

Now I can boot the PXCAB via TFTP into the busybox based ramdisk and from there I can chroot in to the NFS mounted Fedora 24 system.

The system has one CPU with two threads and 4GB of RAM. In addition to the actual RAM there is also 256MB of memory which can be accessed as a block device using the axonram driver. My busybox based ramdisk is copied to that ramdisk and thus freeing some more actual RAM:

# df -h
Filesystem         Size    Used Available Use% Mounted on
/dev/axonram0    247.9M   15.6M    219.5M   7% /

System information from the firmware:

SYSTEM INFORMATION
 Processor  = PowerXCell DD1.0 @ 2800 MHz
 I/O Bridge = Cell BE companion chip DD3.0
 Timebase   = 14318 kHz (external)
 Config     = SMP disabled
 SMP Size   = 1 (2 threads)
 Boot-Date  = 2016-07-21 19:37
 Memory     = 4096MB (CPU0: 4096MB)

RPM Fusion’s mirrorlist server which are returning a list of (probably, hopefully) up to date mirrors (e.g., http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-rawhide&arch=x86_64) still have been running on CentOS5 and the old MirrorManager code base. It was running on two systems (DNS load balancing) and was not the most stable setup. Connecting from a country which has been recently added to the GeoIP database let to 100% CPU usage of the httpd process. Which let to a DOS after a few requests. I added a cron entry to restart the httpd server every hour, which seemed to help a bit, but it was a rather clumsy workaround.

It was clear that the two systems need to be updated to something newer and as the new MirrorManager2 code base can luckily handle the data format from the old MirrorManager code base it was possible to update the RPM Fusion mirrorlist servers without updating the MirrorManager back-end (yet).

From now on there are four CentOS7 systems answering the requests for mirrors.rpmfusion.org. As the new RPM Fusion infrastructure is also ansible based I added the ansible files from Fedora to the RPM Fusion infrastructure repository. I had to remove some parts but most ansible content could be reused.

When yum or dnf are now connecting to http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-rawhide&arch=x86_64 the answer is created by one of four CentOS7 systems running the latest MirrorManager2 code.

RPM Fusion also has the same mirrorlist access statistics like Fedora: http://mirrors.rpmfusion.org/statistics/.

I still need to update the back-end system which is only one system instead of six different system like in the Fedora infrastructure.

This one has been in the making for quite a while, but after some struggling terminatorX has once again reached a release-worthy state. While regular users may not notice a lot of changes right away, this is probably one of the biggest change sets yet. Among a lot of smaller fixes release 4.0.0 brings:

  • Port to Gtk+3, which lead to some unexpected ramifications
  • New audio driver backend for PulseAudio
  • The old X11-DGA based mouse grab mode was incompatible with Gtk+3, so terminatorX now reads from /dev/input/mice directly (when run setuid-root) or falls back to the good old pointer-warp mode (potentially losing some precision compared to reading the events directly from Linux input). The good thing is that both methods should also work for upcoming display server technologies replacing X11.knob-4.0.0
  • The icons now adapt to the configured font size; the knob has been re-rendered to adapt to high-dpi displays (automatic size adjustment can be overridden via configuration)needle-4.0.0
  • The turntable cursor (or needle) now has a short trail (can be switched off) and the audio zoom level can be adjusted using the mouse wheel

terminatorX 4.0.0 is now available from the download page; pre-built packages for Ubuntu 16.04 are available in the terminatorX PPA.

There have been two protocol related issues with MirrorManager open for some time:

Both issues have been resolved. The first issue, to drop FTP URLs from the metalinks, has been resolved in multiple steps. The first step was to block FTP URLs from being added to Fedora’s MirrorManager (Optionally exclude certain protocols from MM, New MirrorManager2 features) and the second step, to remove all remaining FTP URLs from Fedora’s MirrorManager, was performed during the last few days and weeks. Using MirrorManager’s mirrorlist interface (which is not used very often) only returned FTP if the mirror had no HTTP(S) URLs. So it was already rather unusual to be redirected to a FTP mirror. Using MirrorManager’s metalink interface returned all possible URLs for a host. With the removal of all FTP URLs from MirrorManager’s database no user should see FTP URLs any more and the problems some clients encoutered (see Drop ftp:// urls from metalinks) should be ‘resolved’.

The other issue (Add a way to specify you want only https urls from metalink) has also been solved by adding a protocol option to the mirrorlist and metalink back-end. The new MirrorManager release (0.7.2) which includes these changes is already running on the staging instance and the result can be seen here:

To have more HTTPS based mirrors in our database we scanned all existing public mirrors to see if they also provide HTTPS. With this the number of HTTPS URLs was increased from 24 to over 120.

The option to select which protocol the mirrorlist/metalink mirrors should provide is not yet running on the production instance.

The brave effort to create a Debian package for aseqjoy lead to a discussion on whether all parts of aseqjoy (and yes, there are not that many parts) come under the same terms and conditions. To resolve this ambiguity and finally release some dusted modifications sitting the git repository for ages, I finally released aseqjoy-0.0.2 today. Aside of addressing these legal matters aseqjoy now also supports emitting fine MIDI control change events with higher resolution.

Process Migration

Using CRIU it is possible to checkpoint/save/dump the state of a process into a set of files which can then be used to restore/restart the process at a later point in time. If the files from the checkpoint operation are transferred from one system to another and then used to restore the process, this is probably the simplest form of process migration.

Source system:

  • criu dump -D /checkpoint/destination -t PID
  • rsync -a /checkpoint/destination destination.system:/checkpoint/destination

Destination system:

  • criu restore -D /checkpoint/destination

For large processes the migration duration can be rather long. For a process using 24GB this can lead to migration duration longer than 280 seconds. The limiting factor in most cases is the interconnect between the systems involved in the process migration.

Optimization: Pre-Copy

One existing solution to decrease process downtime during migration is pre-copy. In one or multiple runs the memory of the process is copied from the source to the destination system. With every run only memory pages which have change since the last run have to be transferred. This can lead to situations where the process downtime during migration can be dramatically decreased.

This depends on the type of application which is migrated and especially how often/fast the memory content is changed. In extreme cases it was possible to decrease process downtime during migration for a 24GB process from 280 seconds to 8 seconds with the help of pre-copy.

This approach is basically the same if migrating single processes (or process groups) or virtual machines.

It Always Depends On…

Unfortunately pre-copy optimization can also lead to situations where the so called optimized case with pre-copy can require more time than the unoptimized case:

In the example above a process has been migrated during three stages of its lifetime and there are situations (state: Calculation) where pre-copy has enormous advantages (14 seconds with pre-copy and 51 seconds without pre-copy) but there are also situations (state: Initialization) where the pre-copy optimization increases the process downtime during migration (40 seconds with pre-copy and 27 seconds without pre-copy). It depends on the memory change rate.

Optimization: Post-Copy

Another approach to reduce the process downtime during migration is post-copy. The required memory pages are not dumped and transferred before restoring the process but on demand. Each time a missing memory page is accessed the migrated process is halted until the required memory pages has been transferred from the source system to the destination system:

Thanks to userfaultfd this approach (or optimization) can be now integrated into CRIU. With the help of userfaultfd it is possible to mark memory pages to be handled by userfaultfd. If such a memory page is accessed, the process is halted until the requested page is provided. The listener for the userfaultfd requests is running in user-space and listening on a file descriptor. The same approach has already been implemented for QEMU.

Enough Theory

With all the background information on why and how the initial code to restore processes with userfaultfd support has been merged into the CRIU development branch: criu-dev. This initial implementation of lazy-pages support does not yet support lazy process migration between two hosts, but with the upstream merged patches it is at least possible to checkpoint a process and to restore the process using userfaultfd. A lazy restore consists of two parts. The usual ‘criu restore‘ part and an additional, what we call uffd daemon, ‘criu lazy-pages‘ part. To better demonstrate the advantages of a lazy restore there are patches to enhance crit (CRiu Image Tool) to remove pages which can be restored with userfaultfd from a checkpoint directory. Using a test case which allocates about 200MB of memory (and which writes one byte in each page over and over) requires after being dumped about 200MB. Using the mentioned crit enhancement make-lazy reduces the size of the checkpoint down to 116KB:

$ crit make-lazy /tmp/checkpoint/ /tmp/lazy-checkpoint
$ du -hs /tmp/checkpoint/ /tmp/lazy-checkpoint
     201M       /tmp/checkpoint
     116K       /tmp/lazy-checkpoint

With this the data which actually has to be transferred during process downtime is drastically reduced and the required memory pages are inserted in the restored process on demand using userfaultfd. Restoring the checkpointed process using lazy-restore would look something like this:

First the uffd daemon:

$ criu lazy-pages -D /tmp/checkpoint 
--address /tmp/userfault.socket

And then the actual restore:

$ criu restore -D /tmp/lazy-checkpoint 
--lazy-pages --address /tmp/userfault.socket

The socket specified with --address is used to exchange information about the restored process required by the uffd daemon. Once criu restore has done all its magic to restore the process except restoring the lazy memory pages, the process to be restored is actually started and runs until the first userfaultfd handled memory page is accessed. At that point the process hangs and the uffd daemon gets a message to provide the required memory pages. Once the uffd daemon provides the requested memory page, the restored process continues to run until the next page is requested. As potentially not all memory pages are requested, as they might not get accessed for some time, the uffd daemon starts to transfer unrequested memory pages into the restored process so that the uffd daemon can shut down after a certain time.

Having read about using syslinux as a boot-loader for virtual machines I tried to replace grub2 on one of the Fedora 24 virtual machines I am using with syslinux:

Not completely knowing what to do I did:

  • dnf install syslinux-extlinux.x86_64
  • /sbin/extlinux –install /boot/extlinux/

The I tried to create a configuration file using grubby:

  • grubby --extlinux --add-kernel=/boot/vmlinuz-4.4.6-300.fc23.x86_64 --title="4.4.6" --initrd=/boot/initramfs-4.4.6-300.fc23.x86_64.img --args="ro root=/dev/sda3"

Which resulted in:

# cat /etc/extlinux.conf 
label 4.4.6
 kernel /vmlinuz-4.4.6-300.fc23.x86_64
 initrd /initramfs-4.4.6-300.fc23.x86_64.img
 append ro root=/dev/sda3

I added following lines to the file manually:

default 4.4.6
ui menu.c32
timeout 50

After that I rebooted and the virtual machine was still using grub2 to load the kernel.

To write syslinux to the MBR following additional command was required:
dd if=/usr/share/syslinux/mbr.bin of=/dev/sda bs=440 count=1. I was a bit nervous rebooting the system after overwriting the MBR, but it rebooted successfully. The configuration file was also correctly updated after I installed a new kernel via dnf. I also removed grub2 (dnf remove grub2*) and was able to successfully reboot into the new kernel without grub2.

My son got a tiptoi. I was interested how it works and a little bit of googling lead me to this page. It provides a tool to create your own pages, books, adventures or puzzles. I gave it a try and this is the result.

a hand

result of 1st try with tttol

It does not look pretty and I could not print it in color, but the b/w version works. You can see the dotty area on each finger and on the i/o and play button. They contain the code that is read by the tiptoi pen. The example ha two modes. Mode one will just say the name of the finger when you touch it. Mode two can be activated by touching the play button on the lower right. If you touch the fingers in order starting with the thump it’ll tell the German poem “Das ist der Daumen …” or complain if the oder is not correct.

Find here the code:

product-id: 42
comment: das_ist_der_daumen
init: $spiel:=0
welcome: hallo
language: de
scripts:
 dau:
 - $spiel == 0? P(daumen)
 - $spiel == 1? $pos == 0? P(vdaumen) $pos := 1
 - $spiel == 1? $pos != 0? P(vnochmal,vanderer,vsicher,vhmmm)
 zei:
 - $spiel == 0? P(zeige)
 - $spiel == 1? $pos == 1? P(vzeige) $pos := 2
 - $spiel == 1? $pos != 1? P(vnochmal,vanderer,vsicher,vhmmm)
 mit:
 - $spiel == 0? P(mittel)
 - $spiel == 1? $pos == 2? P(vmittel) $pos := 3
 - $spiel == 1? $pos != 2? P(vnochmal,vanderer,vsicher,vhmmm)
 ring:
 - $spiel == 0? P(ring)
 - $spiel == 1? $pos == 3? P(vring) $pos := 4
 - $spiel == 1? $pos != 4? P(vnochmal,vanderer,vsicher,vhmmm)
 kle:
 - $spiel == 0? P(klein)
 - $spiel == 1? $pos == 4? P(vklein) $pos := 0
 - $spiel == 1? $pos != 4? P(vnochmal,vanderer,vsicher,vhmmm)
 spiel:
 - $spiel == 0? P(spiel_start) $spiel:=1 $pos := 0
 - $spiel == 1? P(spiel_end) $spiel:=0 $pos := 0
speak:
 hallo: "Hallo!"
 daumen: "Daumen" 
 zeige: "Zeigefinger" 
 mittel: "Mittelfinger" 
 ring: "Ringfinger" 
 klein: "kleiner Finger" 
 spiel_start: "Das Spiel wird jetzt gestartet. Beginne mit dem Daumen!"
 spiel_end: "Das Spiel wird jetzt beendet"
 vdaumen: "Das ist der Daumen!" 
 vzeige: "Der schüttelt die Pflaumen!" 
 vmittel: "der liest sie auf!" 
 vring: "der trägt sie nach Haus!" 
 vklein: "und der isst sie alle alle auf!" 
 vnochmal: "Versuchs nochmal!"
 vanderer: "Versuch einen anderen Finger!"
 vsicher: "Sicher?"
 vhmmm: "Hmmmm!"

As mentioned by Alex the link was down. Two things happened:

  1. The raspberry pi was not running anymore.
  2. The Internet connection was down.

For the second problem I don’t have a solution yet. For the not running raspberry pi there might be one:

The internal watchdog of the raspberry pi. It can be activated by loading the module, making sure it gets reloaded after a restart and installing the triggering software.

$ sudo modprobe bcm2708_wdog
$ echo "bcm2708_wdog" | sudo tee -a /etc/modules
$ sudo apt-get install watchdog

Configuration happens in the file

/etc/watchdog.conf

by uncommenting the following lines:

watchdog-device        = /dev/watchdog
max-load-1             = 24

This is a very basic configuration and it will restart the raspberry pi in case the load is above 24 for a 1 minute interval.

Activation of the demon can be done like this:

$ sudo service watchdog start

Specific in my case is the additional option to check whether the file, that was not working as mentioned above, is written to on a regular basis. This can be achieved by adding the following lines in the configuration:

file = /data/solar/solar.touch.start
change = 300
file = /data/solar/solar.touch.end
change = 600

Each “file” entry specifies a file that will be checked by the watchdog whether it’s been touched and the “change” entry specifies the time that the file can stay untouched before the watchdog will not be triggered any more and by that lead to a system reset. The first file is touched at the start of the script, the second one at the end. So in case the script for updating the yield data is not called any more the system will be reset after 5 minutes. If the script is started, but does not finish properly it’ll be reset after 10 minutes.

Time will tell how reliable the watchdog is.

After a long break I’ve started logging the PVIs in my father’s house again. The main reason for reactivating the scripts was that the two PVIs have shown different yield numbers at the end of the day. Further investigation has shown that the internal clock of one of the PVIs was wrong, so at around noon the yield counter was reset, which of course led to different results. Anyway the graphs are online now. Currently the graphs are generated using google charts. Hints for an alternative are welcome.

I’ve taken some pictures and short clip

wild animal

click on image for gallery

of the parrots living in the tree in front of my house.