Plotting TTN gateways on a map

Introduction

While using the TTN tracker application for some time now on my home-made LoPy tracker it shows that signals were recorded by at least 246 different gateways in the Dutch area. This raised a question: How many more did not yet receive my tracker, where are they located and which would be the easiest to drive by?
In my job as an IT consultant for Healthcare applications I drive all across The Netherlands so lots of opportunities to “take the long way home”.

The problem described has two parts: Gathering the gateways which received the tracker (A) and gathering the entire list of gateways (B). Plotting B minus A would be the answer.

Prequisits

This solution is built on NodeRed (0.19.4) with node-red-contrib-ttn (2.0.4) and node-red-contrib-web-worldmap (1.4.3) installed.

Which gateways received my tracker

I built this some time ago when the tracker was brought live and uses the TTN integration for Node-Red.SolutionA

Note: in contrib-ttn 2.0.4 the TTN message node is now called TTN uplink //post & picture needs update 🙂

The TTN message receiver is registered to the tracker application on The Things Network. On each received packet it sends an array of gateways (with tracker data, gateway ID, location, signal strengths etc) which received the particular packet. This data can be found in the msg.metadata part (not the ordinairy msg.payload). This array is decomposed and written to a CSV logfile. In the end this file contains all gateways, but not yet the unique list…

Contents of “Process received TTN messages:

//process new message which may contain the response
//of multiple gateways which received the packet
var gatewaysarray = msg.metadata.gateways;
var logMsgs = [];
for (i = 0; i < gatewaysarray.length; i++){
   logMsgs.push({payload: {time: gatewaysarray[i].time,
                           my_lat: lat_m,
                           my_lon: lon_m,
                           gtw_id: gatewaysarray[i].gtw_id,
                           channel: gatewaysarray[i].channel,
                           rssi: gatewaysarray[i].rssi,
                           snr: gatewaysarray[i].snr,
                           rf_chain: gatewaysarray[i].rf_chain,
                           gwy_lat: gatewaysarray[i].latitude,
                           gwy_lon: gatewaysarray[i].longitude,
                           dist: distancesarray[i],
                           }
                 });
   }
return logMsgs;

How to get the entire list of gateways

My first guess was asking JP Meijers of TTNMapper to build this list from his TTNMapper database. Although he was quite willing, a better suggestion was posed: This data is available through an API at TheThingsNetwork. The JSON reply looks like (shown for a single random gateway)

{“statuses”:
  {"eui-b827ebfffef23042":
    {"timestamp":"2018-04-21T10:44:03.020602294Z",
     "uplink":"52790",
     "downlink":"87",
     "location":
        {"latitude":50.06118,
         "longitude":9.459908,
         "altitude":260,
         "source":"REGISTRY"},
    "platform":"IMST + Rpi",
    "gps":
        {"latitude":50.06118,
        "longitude":9.459908,
        "altitude":260,
        "source":"REGISTRY"},
    "time":"1524307443020602294",
    "rx_ok":52790,
    "tx_in":87}

This JSON contains all we need to know.
One word of advice: This gateways JSON is about 5MB large. So we won’t do the TTN operators a favor when requesting these data every minute.

SolutioB

Upon receiving the (manual) trigger the second part of the flow first reads the entire CSV file with received packets created above and filters it to a single array of unique gateways which once received my tracker. The list is stored in the flow context for later use. Note: Of course if this application is the only purpose for this CSV logfile we could have done filtering right away i.e. only add to and save the unique list.

After retrieving the entire gateway list the results are filtered. I’m only interested in a rectangular area containing The Netherlands and gateways which have been online today.
From this data an array is formed containing the markers with for each marker the properties:

"name":key, // the gateway ID
"lon":gateways[key].location.longitude, //the gateway longitude
"lat":gateways[key].location.latitude, //the gateway lattitude
"layer": gwlayer, //the layer to put the marker on (either visited or new)
"icon": "dot-circle", //the icon used
"iconColor": gwcolor, //the color used: green or various reds depending on gwy activity
"weblink": weblink, //clickable url to TTNMapper radar for this gateway
"rx packets": gateways[key].uplink, //nr of received packets this gateway ever received
"tx packets": gateways[key].downlink, //nr of transmitted packets this gateway ever sent
"altitude": gateways[key].location.altitude, //gateway antenna height
"plaform": gateways[key].platform // type of hardware used.

Feeding this sequentially to the Node-Red Worldmap not only shows the gateways but lets us select the particular layer (seen or unseen gateways). Unseen gateways are colored according to their chance of being able to pick up the tracker. An indoor gateway with 100 packets received probably won’t hear the tracker at 2km distance.
Clicking a gateway shows its details and a “TTN Radar link” to the TTNMapper radar page as well.

2018-09-21 10_41_44-Node-RED map all the things

2018-09-21 13_25_26-

Contents of Filter & Color:

var gwcolor = "";
var gwlayer = "";
var gwlist = [];

//only list gateways actively seen today
var d = new Date();
var todaystr = d.getFullYear() + "-" + ("0"+(d.getMonth()+1)).slice(-2) + "-" + ("0" + d.getDate()).slice(-2);
var gateways_seen = flow.get("gateways_seen");
var gateways = msg.payload.statuses;
node.status({fill:"red",shape:"dot",text:"Busy building list"});

//traverse through all existing gateways
//filter for The Netherlands area and active gateways only
//color seen gateways green, the others red intensity depending on nr of uplink messages
for (var key in gateways){
    if (gateways.hasOwnProperty(key)){
        if ((gateways[key].location.longitude > 2.8) &&
           (gateways[key].location.longitude < 7.6) &&            (gateways[key].location.latitude > 50) &&
           (gateways[key].location.latitude < 54) &&            (gateways[key].timestamp.substring(0,10) == todaystr)){                if (gateways_seen.indexOf(key) > -1 ){
                   gwcolor = "#66ff33";  //green
                   gwlayer = "visited gateways";
               }
                else {
                   gwlayer = "new gateways";
                   gwcolor = "#FFE6E6"; //very light red
                   if (gateways[key].uplink > 1000){
                      gwcolor = "#FF9999"; // bit brighter red
                      if (gateways[key].uplink > 10000){
                         gwcolor = "#FF4D4D"; // /another bit brighter red
                         if (gateways[key].uplink > 100000){
                            gwcolor = "#FF0000"; // absolutely red
                         }
                      }
                   }
                }
                //create URL to TTN Mapper radar page of this gateway
                //"eui-" type gateways need the eui part removed
                var newkey = key;
                if (key.substring(0, 4) == "eui-"){
                    newkey = key.substring(4).toUpperCase()
                }
                var weblink = {"name":"TTN radar", "url":"https://ttnmapper.org/colour-radar/?gateway=" + newkey + "&type=radar&hideothers=on"};
                //put marker in list to show
                gwlist.push({"name":key,  
                             "lon":gateways[key].location.longitude,
                             "lat":gateways[key].location.latitude,
                             "layer": gwlayer,
                             "icon": "", 
                             "iconColor": gwcolor, 
                             "weblink": weblink,
                             "rx packets": gateways[key].uplink,
                             "tx packets": gateways[key].downlink,
                             "altitude": gateways[key].location.altitude,
                             "plaform": gateways[key].platform
                });
           }
    }
}
node.status({});   // clear the status
flow.set("gwlist", gwlist);
msg = {payload: gwlist};
return msg;

Future developments

It would be great if the map would be realtime: As soon as the tracker is received by a gateway the marker could be updated realtime.

Download

A copy of the flows can be downloaded from my github pages

Enjoy!

Advertisements
This entry was posted in The Things Network and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s