Introduction
While using the TTN tracker application for some time now on my home-made LoPy tracker it shows that signals were recorded by at least 246 different gateways in the Dutch area. This raised a question: How many more did not yet receive my tracker, where are they located and which would be the easiest to drive by?
In my job as an IT consultant for Healthcare applications I drive all across The Netherlands so lots of opportunities to “take the long way home”.
The problem described has two parts: Gathering the gateways which received the tracker (A) and gathering the entire list of gateways (B). Plotting B minus A would be the answer.
Prequisits
This solution is built on NodeRed (0.19.4) with node-red-contrib-ttn (2.0.4) and node-red-contrib-web-worldmap (1.4.3) installed.
Which gateways received my tracker
I built this some time ago when the tracker was brought live and uses the TTN integration for Node-Red.
Note: in contrib-ttn 2.0.4 the TTN message node is now called TTN uplink //post & picture needs update 🙂
The TTN message receiver is registered to the tracker application on The Things Network. On each received packet it sends an array of gateways (with tracker data, gateway ID, location, signal strengths etc) which received the particular packet. This data can be found in the msg.metadata part (not the ordinairy msg.payload). This array is decomposed and written to a CSV logfile. In the end this file contains all gateways, but not yet the unique list…
Contents of “Process received TTN messages:
//process new message which may contain the response
//of multiple gateways which received the packet
var gatewaysarray = msg.metadata.gateways;
var logMsgs = [];
for (i = 0; i < gatewaysarray.length; i++){
logMsgs.push({payload: {time: gatewaysarray[i].time,
my_lat: lat_m,
my_lon: lon_m,
gtw_id: gatewaysarray[i].gtw_id,
channel: gatewaysarray[i].channel,
rssi: gatewaysarray[i].rssi,
snr: gatewaysarray[i].snr,
rf_chain: gatewaysarray[i].rf_chain,
gwy_lat: gatewaysarray[i].latitude,
gwy_lon: gatewaysarray[i].longitude,
dist: distancesarray[i],
}
});
}
return logMsgs;
How to get the entire list of gateways
My first guess was asking JP Meijers of TTNMapper to build this list from his TTNMapper database. Although he was quite willing, a better suggestion was posed: This data is available through an API at TheThingsNetwork. The JSON reply looks like (shown for a single random gateway)
{“statuses”:
{"eui-b827ebfffef23042":
{"timestamp":"2018-04-21T10:44:03.020602294Z",
"uplink":"52790",
"downlink":"87",
"location":
{"latitude":50.06118,
"longitude":9.459908,
"altitude":260,
"source":"REGISTRY"},
"platform":"IMST + Rpi",
"gps":
{"latitude":50.06118,
"longitude":9.459908,
"altitude":260,
"source":"REGISTRY"},
"time":"1524307443020602294",
"rx_ok":52790,
"tx_in":87}
This JSON contains all we need to know.
One word of advice: This gateways JSON is about 5MB large. So we won’t do the TTN operators a favor when requesting these data every minute.
Upon receiving the (manual) trigger the second part of the flow first reads the entire CSV file with received packets created above and filters it to a single array of unique gateways which once received my tracker. The list is stored in the flow context for later use. Note: Of course if this application is the only purpose for this CSV logfile we could have done filtering right away i.e. only add to and save the unique list.
After retrieving the entire gateway list the results are filtered. I’m only interested in a rectangular area containing The Netherlands and gateways which have been online today.
From this data an array is formed containing the markers with for each marker the properties:
"name":key, // the gateway ID "lon":gateways[key].location.longitude, //the gateway longitude "lat":gateways[key].location.latitude, //the gateway lattitude "layer": gwlayer, //the layer to put the marker on (either visited or new) "icon": "dot-circle", //the icon used "iconColor": gwcolor, //the color used: green or various reds depending on gwy activity "weblink": weblink, //clickable url to TTNMapper radar for this gateway "rx packets": gateways[key].uplink, //nr of received packets this gateway ever received "tx packets": gateways[key].downlink, //nr of transmitted packets this gateway ever sent "altitude": gateways[key].location.altitude, //gateway antenna height "plaform": gateways[key].platform // type of hardware used.
Feeding this sequentially to the Node-Red Worldmap not only shows the gateways but lets us select the particular layer (seen or unseen gateways). Unseen gateways are colored according to their chance of being able to pick up the tracker. An indoor gateway with 100 packets received probably won’t hear the tracker at 2km distance.
Clicking a gateway shows its details and a “TTN Radar link” to the TTNMapper radar page as well.
Contents of Filter & Color:
var gwcolor = "";
var gwlayer = "";
var gwlist = [];
//only list gateways actively seen today
var d = new Date();
var todaystr = d.getFullYear() + "-" + ("0"+(d.getMonth()+1)).slice(-2) + "-" + ("0" + d.getDate()).slice(-2);
var gateways_seen = flow.get("gateways_seen");
var gateways = msg.payload.statuses;
node.status({fill:"red",shape:"dot",text:"Busy building list"});
//traverse through all existing gateways
//filter for The Netherlands area and active gateways only
//color seen gateways green, the others red intensity depending on nr of uplink messages
for (var key in gateways){
if (gateways.hasOwnProperty(key)){
if ((gateways[key].location.longitude > 2.8) &&
(gateways[key].location.longitude < 7.6) && (gateways[key].location.latitude > 50) &&
(gateways[key].location.latitude < 54) && (gateways[key].timestamp.substring(0,10) == todaystr)){ if (gateways_seen.indexOf(key) > -1 ){
gwcolor = "#66ff33"; //green
gwlayer = "visited gateways";
}
else {
gwlayer = "new gateways";
gwcolor = "#FFE6E6"; //very light red
if (gateways[key].uplink > 1000){
gwcolor = "#FF9999"; // bit brighter red
if (gateways[key].uplink > 10000){
gwcolor = "#FF4D4D"; // /another bit brighter red
if (gateways[key].uplink > 100000){
gwcolor = "#FF0000"; // absolutely red
}
}
}
}
//create URL to TTN Mapper radar page of this gateway
//"eui-" type gateways need the eui part removed
var newkey = key;
if (key.substring(0, 4) == "eui-"){
newkey = key.substring(4).toUpperCase()
}
var weblink = {"name":"TTN radar", "url":"https://ttnmapper.org/colour-radar/?gateway=" + newkey + "&type=radar&hideothers=on"};
//put marker in list to show
gwlist.push({"name":key,
"lon":gateways[key].location.longitude,
"lat":gateways[key].location.latitude,
"layer": gwlayer,
"icon": "",
"iconColor": gwcolor,
"weblink": weblink,
"rx packets": gateways[key].uplink,
"tx packets": gateways[key].downlink,
"altitude": gateways[key].location.altitude,
"plaform": gateways[key].platform
});
}
}
}
node.status({}); // clear the status
flow.set("gwlist", gwlist);
msg = {payload: gwlist};
return msg;
Future developments
It would be great if the map would be realtime: As soon as the tracker is received by a gateway the marker could be updated realtime.
Download
A copy of the flows can be downloaded from my github pages
Enjoy!
Projects by Keptenkurk 