Using Node-Red to graph scale data

Having used Node-Red briefly about two years ago, i decided to load it again onto the Raspberry Pi which listens for my BS440 scale. Just to dig into Node-Red and see where it got.

Installing Node-Red is a breeze although it takes a coffee and a beer or two before it is installed. But it’s more than worth the wait! A newly added set of nodes is the “dashboard” which can be added through the palette manager once Node-Red is up and running.

My idea was that i could graph the BS440 scale data which is stored into a CSV file (if the CSV plugin is enabled). I was shocked to find out i had made a fully operational webpage showing the data within one hour of coding (or drawing to be more precise)! And that included getting  familiar with Node-Red.

The result is a fancy looking page which shows nicely on my SmartPhone and is lightning fast. It offers a drop down selection for the type of data and two buttons to select the user. Ain’t that cool!

So how is this done? Without giving a lecture on Node-Red (i’ll leave it up to you to explore this piece of magic) this is what the flow looks like:

NR-BS440-flow

The core is reading the CSV file. The filename is set by selecting a user. Once the file is read it is converted to a node-js object. Type selects which data we want to graph. The selected type of data is read from the object and fed into the graph node.

The selecting thing might need some explanation as it uses JSONata (just watch the video). It is included in Node-Red and enables you to make a query from a bunch of JSON formatted data in just a single statement. As an example. If you want to craft an array of “vet” values from all data and change the unix timestamp to show milliseconds on the fly the JSONata for that is just:

[ { “key”: “vet”, “values”: msg.payload.[tijd*1000, vet] } ]

and the node “Select vet”  then looks like (note the J: selection)

NR-BS440-4

The rest of the nodes just show the dropdown list and store and show the current user.

The project is made available on my Github

On to the next project. Lets have fun!

Advertisements
Posted in BLE | Tagged , , , , , , , | Leave a comment

Connecting the Medisana BS440 Bluetooth scale (epilogue)

Having used the BS440 bathroom scale script for over a year i got time to update the whole thing. I decided to do a full reinstall and log all the individual steps it takes to get things running. So here is the recipe.

Hardware:

  • Raspberry Pi B+ (note: i have no Pi3 so no experience in using the built in BLE adapter)
  • USB BLE adapter (I use brand ADJ model 100-00006 BT 4.0 adapter which uses driver broadcom BCM20702A0)
  • WiFI adapter (unless you can position a wired RPi within 1 m from the scale) like this one

 

  1. Download latest Raspbian Jessie Lite build for your RPi from here.
  2. Format and image the SD card. Instructions here
  3. Enable SSH access by placing a file named ‘ssh’, without any extension, onto the boot partition of the SD card.
  4. Use a temporarely wired connection to connect to the RPi using Putty
  5. Change pwd and run sudo raspi-config and do
    expand filesystem
    set timezone
    set hostname like bs440rpi (or any other nice name)
  6. run sudo apt-get update and sudo apt-get dist-upgrade
  7. Enable WiFi using this tutorial.
  8. Follow Tony DiCola tutorial to set up Bluez on the RPi. I selected the latest and greatest Bluez 5.44. Follow this tutorial step by step. No need to enable “experimental”. Building Bluez will take half an hour or so…
  9. Copy btmgmt to the local bin folder with:  sudo cp /home/pi/bluez-5.44/tools/btmgmt  /usr/local/bin so it can be used from anywhere.
  10. Enable BT low enegry with btmgmt le on
  11. run hcitool lescan  and step on the scale with a registered user selected so all your properties are measured and stored. The scale should report with its MAC address and name like F1:37:57:6C:DE:64 [0202B664DE6C5737F1] write down both.
  12. Test connectivity with sudo gatttool -t random -b F1:37:57:6C:DE:64 -I
  13. at the prompt type connect which should respond connected. Error 111 indicates that the adapter is not in LE mode (retry the btmgmt le on).
  14. Avoid having to use super user privileges (sudo) on hcitool by sudo setcap 'cap_net_raw,cap_net_admin+eip' 'which hcitool'
  15. Avoid having to use super user privileges (sudo) on btmgmt by sudo setcap 'cap_net_raw,cap_net_admin+eip' which btmgmt
  16. Install Pygatt 3.0.0 by sudo apt-get install python-pip and sudo pip install "pygatt[GATTTOOL]" This will throw a number of warnings and even errors but should end with Successfully installed pygatt pyserial enum34 pexpect ptyprocess
  17. Get the BS440 files by downloading the zip file from https://github.com/keptenkurk/BS440  and put them in a separate folder on the RPi like /home/pi/BS440.
  18. Enable the plugins of your choice by removing the underscore in the plugins folder for the .py and .ini files.
  19. Edit BS440.ini (fill in your scale’s MAC address and name from step 11) and all the enabled BS440<plugin>.ini in the plugins folder
  20. Start BS440.py with  python BS440.py. Start a second shell and monitor the logfile with tail -f BS440.log
  21. Enjoy
Posted in BLE | 2 Comments

Connecting the Medisana BS440 Bluetooth scale (part 5)

Triggered by a comment on episode 1 of this blog i decided to use Domoticz to visualize the measurements in a graph. I use Domoticz to control my home since july 2013 and it has proven to be very reliable.

To graph the weight first a “virtual sensor” must be created. First find a unique index and paste the url below in your browser:

http://domoticzip:8080/json.htm?type=command&param=udevice&hid=2&did=60&dunit=4&
dtype=93&dsubtype=1&nvalue=0&svalue=75.0
* did=60 represents the device ID number for this device 
  (could be any unique number). Make a note of it. 
  Upon creation the device will be given an unique index.
* dtype=93 refers to the BWR102 device, a Oregon Scientific connected scale we mimic.
* svalue=75.0 refers to the weight. Every call to this URL will update the value.

After creation of the virtual sensor, push the green button to have the sensor showing up on the “Other devices” tab:

Domoticz1

Other Devices tab:

Domoticz2

Now the sensors are ready the BS440 program can call the sensor URL with the appropriate Device number and weight (see BS440domoticz.py). In the .ini file the Domoticz connection parameters (IP-adres and username/password) can be entered.

Now every time a user steps on the scale the weight will be logged and graphed.

Domoticz3

To have consistent results it is advised to do your weighings at the same time of day. In my case this will be between 07:00 and 08:00 in the morning (8:00 – 11:00 in the weekends). I use an IP controlled switch based on a Sonoff ESP8266 based 220V switch flashed with ESP Easy firmware. Now Domoticz can switch on/off the Raspberry through WiFi by calling the ip address of the Sonoff like

http://ip-address/control?cmd=gpio,12,1 (=on) 
http://ip-address/control?cmd=gpio,12,0 (=off)

Domoticz4

Before switching off at 8:15 (11:15 in the weekeinds) be sure that a cron job did a shutdown earlier to not corrupt the flash card. Also when powering up, the “python BS440.py” needs to be started automatically through BS440.sh and a @reboot cron. These are the cron definitions to accomplish this:

@reboot sh /home/pi/BS440.sh > /home/pi/cronlog 2&1
0 8 * * 1,2,3,4,5 sudo shutdown -h +0
0 11 * * 0,6 sudo shutdown -h +0 

A 1 minute sleep is inserted in BS440.sh to have a stable booted Raspberry before starting the bluetooth communications in background.


#!/bin/sh
cd /
cd home/pi
sleep 1m
sudo python BS440.py &
cd /
Posted in FF32 | Tagged , , , , , , , | 40 Comments

Connecting the Medisana BS440 Bluetooth scale (part 4)

To start with: i am not a professional programmer, never was, never will be. Started off with some assembly on a 8 bit (yes!) SC/MP processor in my early days and ended with Turbo Pascal on MS-DOS6.2 as highest level of sophistication. Maybe that’s why i enjoy poking around on the credit-card size Raspberry in Python. The wealth of internet resources surely helps. This journey learned me about C-structs, callback routines, running shell commands, logging, parsing ini files, exception handling, pep8 and of course a little bluetooth. The complete code can be found  on my Github pages

The sequence of commands we want to issue through gatttool after being connected to the scale are basically these (when issued from the shell, pygatt will use the interactive mode of gatttool):

gatttool -t random -b F1:37:57:XX:XX:XX --char-write --handle 0x001f -n 0200
gatttool -t random -b F1:37:57:XX:XX:XX --char-write --handle 0x001c -n 0200
gatttool -t random -b F1:37:57:XX:XX:XX --char-write --handle 0x0026 -n 0200
gatttool -t random -b F1:37:57:XX:XX:XX --char-write-req --handle 0x0023 -n 0000000000 --listen

But the problem is we need to connect first and connecting is only possible for about one minute after the person has stepped onto the scale and we don’t know when that will be. The first attempt was just endless keep on trying to connect with 5 second timeouts. This is what it looked like:

adapter = pygatt.backends.GATTToolBackend()
adapter.start()
# wait for someone to step on the scale
while True:  
    try:
        device = adapter.connect('f1:37:57:xx:xx:xx', 5, 'random')
        break
    except pygatt.exceptions.NotConnectedError:
        print('Waiting...')

after which the program would continue with regestering to the indications and reading data… However every connection attempt/timeout would take more and more time every go, eventually slowing down to once a minute. Also CPU utilization would rise to 100% for some obscure reason.

The current approach uses the “hcitool lescan” command to scan for active bluetooth devices (filtered on our device of interest) before attempting to connect. This works better although the bluetooth driver needs a reset after every “hcitool lescan”/timeout by issuing the pygatt “adapter.reset()” command. Probably because “hcitool lescan” is not nicely terminated. There is some discussion about this behaviour of Pygatt at the Pygatt repo of Peplin. For now i just reset the interface after every timeout. Not nice but it works!

Awaiting the code to put data in a database and adding a nice web interface to it, last three measurements will be sent to the user by e-mail. The first step in tracking your weight!

 

21-03 07:4723-03 07:2323-03 23:30
76.375.976.9
17.717.417.7
36.736.936.7
2.92.92.9
64.564.764.5
258725792600

Datum
Gewicht (kg) Vet (%) Spieren (%) Botten (kg) Water (%) Verbruik (kCal)

Feel free to ask for more clarification and meanwhile: enjoy coding!

Posted in BLE | Tagged , , , , , , , | 2 Comments

Connecting the Medisana BS440 Bluetooth scale (part 3)

Time to dive into the (bluetooth) specifics of the BS440 scale. When starting this project i was lucky to bump into this post on StackOverflow where user Edmundo was actually trying to acomplish the same. He snooped the bluetooth traffic on his Android phone and thus discovered the commands to send to the scale to make it send the data we need.

Data is presented to us by use of Indications. Indications are messages conveying the data for certain characteristics the device supports. Characteristics have a long (16 byte globally unique UUID) identifier but also a two-byte shortcut for it: the handle. Writing the command 0200 to a handle tells the device you register to the indications of this characteristic. Writing 0000 will stop it. A data packet of this characteristic will report with a handle 1 less than the handle of the characteristic. The BS440 has 3 kinds of interesting characteristics:

Handle UUID                                  Data
0x26   00008a82-0000-1000-8000-00805f9b34fb  person, gender, age, size, activity
0x1c   00008a21-0000-1000-8000-00805f9b34fb  weight, time, person
0x1f   00008a22-0000-1000-8000-00805f9b34fb  time, person, kcal, fat, tbw, muscle, bone

Some notes: person ranges from 1~8 and corresponds with the user selected before stepping on the scale. Or: being a “smart scale” : if the weight roughly corresponds to a known user weight that user is automatically selected. And when multiple users with the same weight are found it even lets you select the appropriate user.

So what does the data look like?

The last 30 measurements per person will be stored in the scale and upon communication the history for this user will be dumped. So you will receive values like this

handle=0x25, value=0x845302800134b6e0000000000000000000000000
handle=0x1b, value=0x1d8c1e00fe6e0aa056451100ff020900000000
handle=0x1e, value=0x6f6e0aa05602440ab8f07ff26bf11ef0000000
handle=0x1b, value=0x1d961e00fe72fe9f56291100ff020900000000
handle=0x1e, value=0x6f72fe9f5602460ab8f07ff26bf11ef0000000
handle=0x1b, value=0x1d961e00fe2dfe9f569e1000ff020900000000
handle=0x1e, value=0x6f2dfe9f5602460ab8f080f26cf11ef0000000
handle=0x1b, value=0x1d8c1e00feb3fd9f56c21000ff020900000000
handle=0x1e, value=0x6fb3fd9f5602440ab8f080f26cf11ef0000000
handle=0x1b, value=0x1d8c1e00fe39fc9f56641000ff020900000000
handle=0x1e, value=0x6f39fc9f5602440ab8f080f26cf11ef0000000
handle=0x1b, value=0x1d8c1e00fe02fb9f56991000ff020900000000
handle=0x1e, value=0x6f02fb9f5602440ab8f080f26cf11ef0000000
handle=0x1b, value=0x1d8c1e00fe80fa9f561d1000ff020900000000
handle=0x1e, value=0x6f80fa9f5602440ab7f080f26cf11ef0000000
handle=0x1b, value=0x1d8c1e00fef7f89f56021100ff020900000000
handle=0x1e, value=0x6ff7f89f5602440ab8f080f26bf11ef0000000
handle=0x1b, value=0x1d8c1e00fe10f39f566a1000ff020900000000
handle=0x1e, value=0x6f10f39f5602440ab8f080f26cf11ef0000000
etc.

Nice, but what does it actually mean?

Now the puzzle starts where to find the data we need, but luckily Edmundo helps us out again:

All multibyte values represented in unsigned int, little endian (multi byte values 
start with least significant byte).
First byte of string is byte 0.
0x1b:                              value                     
fixed: byte: 0                     [0x1d]
weight: byte: 1 & 2                [kg*100]
timestamp: byte 5-8                Unix timestamp
person: byte 13                    [1..8]

0x1e:                              value                    
fixed: byte 0                      [0x6f]
timestamp: byte 1-4                Unix timestamp
person: byte 5                     [1..8]
kcal: byte 6 & 7                   first nibble = 0xf, [kcal]
fat: byte 8 & 9                    first nibble = 0xf, [fat*10]
tbw: byte 10 & 11                  first nibble = 0xf, [tbw*10] 
muscle: byte 12 & 13               first nibble = 0xf, [muscle*10]
bone: byte 14 & 15                 first nibble = 0xf, [bone*10]

0x25:                               value                   
fixed: byte 0                       [0x84]
person: byte 2                      [1..8]
gender: byte 4 (1=male, 2=female)   [1|2] 
age: byte 5                         [year]
size: byte 6                        [cm]
activity: byte 8 (0=normal, 3=high) [0|3]

In Python the datastrings can be interpreted simply using struct.unpack using the pattern ‘BxBxBBBxB’ for the 0x25 data, ‘<BHxxIxxxxB’ for the 0x1b data and ‘<BIBHHHHH’ for the 0x1e data. In my program BS440decode.py takes care of the decoding job.

These insights should get us going. Just one more thing. When does the scale start sending data? Data transfer starts after sending 0x02 followed by the unix timestamp to handle 0x23. This is also the way the scale resyncs it’s RTC. Note that this last write must be a write with response (i.e. “write-req” instead of a “write-cmd”).

By now you might start to understand why this project dragged over 3 months to complete. Part 4 will discuss the Python code.

 

Posted in BLE | Tagged , , , , , , , | 2 Comments

Connecting the Medisana BS440 Bluetooth scale (part 2)

It’s been a while after part one but not because of me abandoning the subject. On the contrary: i must have been stepping on an off that darn scale like a million times trying to debug my way around the BLE mysteries.

Continuing the previous post we now should first install Pygatt. It installs through the pip installer which needs to be installed first.

sudo apt-get install python-pip

after which pygatt and all dependencies may be installed

sudo pip install 'pygatt[GATTTOOL]'

Apart from throwing all kinds of errors and warnings it ends with “Successfully installed…”. To be able to use Pygatt for my purpose i needed to fix another few things.

Fix 1: Support for receiving multibyte indications

Receiving data from the scale is performed by subscribing to an indication. The indications contain 40 hex characters and the installed Pygatt (as of march 2016) does not support that. Ilya Sukhanov fixed that in function “_handle_notification_string” but his changes have not been merged (yet).

But Ilya made some more improvements so just grab his gatttool.py entirely and replace the one found at /usr/local/lib/python2.7/dist-packages/pygatt/backends/gatttool

EDIT: As of 6/9/16 the above changes have been merged into Peplin’s master

Fix 2: Restarting bluetooth

At adapter.start() Pygatt will first try to restart the bluetooth service by “sudo systemctl restart bluetooth” however on debian Jessie this should be “sudo systemctl restart bluetooth.target“. This is changed manually at the bottom of gatttool.py by changing “subprocess.Popen([sudo, systemctl, restart, bluetooth]).wait()” into “subprocess.Popen([sudo, systemctl, restart, bluetooth.target]).wait()”

Fix 3: Enable le (low energy) mode

Using the gatttool in interactive mode, like (replace address with yours):

gatttool -i hci0 -b 20:C3:8F:E9:56:F1 -I

would respond with the “[20:C3:8F:E9:56:F1][LE]>” prompt. But the connect command would fail:

[20:C3:8F:E9:56:F1][LE]> connect
Attempting to connect to 20:C3:8F:E9:56:F1
Error: connect: Connection refused (111)

In my case this could be solved by

sudo btmgmt le on

So for this to use from Python it is usefull to copy the “btmgmt” command to the /usr/local/bin directory too.

To be frank: Even after fiddling around with the Bluetooth utils it is still not quite clear to me what tool to choose from: hcitool, hciconfig, btmgmt, gatttool…

Anyway: With this installation you now should be able to run the python code i wrote which can be found at my Github pages.

Next post will zoom in the Python code and de BS440 connected scale specifics.

 

Posted in BLE | Tagged , , , , , , , | 6 Comments

Connecting the Medisana BS440 Bluetooth scale (part 1)

About a year ago I bought the Medisana BS440 bathroom scale. It measures not only weight but also the percentage of muscle, fat and water in your body. And it has Bluetooth! Although it comes with an Android/iOS app to upload data to Medisana’s VitaDock cloud (tough luck with my Windows Phone) I couldn’t resist the challenge to get the data off the scale  to a Raspberry Pi and handle the data the way I like (like sending an email to my wife when… No! Better don’t!)

ScaleProject

The goal for this project

Continue reading

Posted in BLE | Tagged , , , , , , , | 6 Comments