SPC Ltd: A proposed interview process for technical people

SPC Ltd doesn’t exist yet – but I’ve been thinking for a few years now about what the simplest, least effort and most discuss-able interview process might be for technical people. ‘Technical people’ means anyone whose job it is to write code and build systems. What about this?

  • Explain how the internet works. Go into the level of detail you’re comfortable with
  • Write a useful program (that runs on a computer) in less than 50 lines. Use any language of your choice, use any resource you like (e.g. Google, ChatGPT)

Tao Helsuit

It’s cold in London tonight: 4c/39.2f [feels like 1c/33.8f]. Imagine if that info was available on-suit?

Since I’m (almost begrudgingly) awake, let’s have a stab at ‘a way’ – as they might say in Sunderland.

Consider the following:

  1. What would the world look like if everyone had a top-o-the-line M2 MacBook Pro strapped not to their faces or heads, but to a non-intrusive part of the human body?
  2. What if screens, cameras and other peripherals were decoupled from compute and were used as an extension of an arm?
  3. What if eight billion computers were networked together in one global mesh, what would the world look like? Would we still need data centres?
  4. What if sitting with friends and loved ones, in close proximity, led to an increase in computation? What problems could we solve together?
  5. What is required to make the above a reality?
  6. What hardware and software is required to get us as close to that reality, as close to today as possible?
  7. What is the simplest ‘no-interface’ to such a system?

That’s Helsuit.

Another tribute to Ray Davies

Whilst searching for respected figures in my life to help me decide if SPC is an idea worth validating ‘in the wild’ or not, I remembered an exchange on LinkedIn, some years ago, I had with Ray Davies.

Ray was the leader of the Photonics Academy at OpTIC during the summer of 2009 where I attended that year’s summer school. He had been leading it for a good while and he led it for a good while after. The exchange we had is now lost to the LinkedIn archives because his profile no longer exists. Standing outside vaping (of all things) a few possible root causes crossed my mind as to why his profile had become invisible. LinkedIn is a large and complex beast with many a user definable setting – perhaps he’d changed his privacy settings or even boycotted the platform in some way? Perhaps it was a migration problem that Ray, being so focused on teaching and learning, hadn’t seen fit to chase up?

Then the other possibility hit me like a ten ton truck:
https://www.bangor.ac.uk/computer-science-and-electronic-engineering/news/ray-davies-tribute-45033

Except through otherworldly means that would be ineffective at best, I can’t share my wild ramblings with Ray. So I’ll share who Ray was to me.

During my time at OpTIC, close to the mists of the past as it is for me, the consistent memory I have of Ray is a warm, fierce and challenging teacher. He taught me that ‘everything is a computer’ (in the sense that every object on planet earth has the ability to change information – in most cases electromagnetic waves in the visible spectrum – that is, light). He taught me that even though you don’t know how to build it yet, the idea is all that matters. The details can be figured out by trusted experts (I’m looking at you, Boris). He taught me that naming things, even with janky backronyms, is an important part of the Tao of the Electronic (and indeed Photonic) Engineer. Without Ray, it is likely I wouldn’t have found my way into my chosen subject at university. Without Ray, I would have never believed that a group of teenagers in an open plan classroom could make a group of entirely novel, and socially useful, inventions in five weeks. It was Ray’s insistence that our inventions be of social utility that likely grounded my engineering practice from then on. Whatever SPC turns out to be, we owe a large part of it to Ray.

Without Ray, I would be living a dimmer version of my life. In the software engineering game we often talk about (10x) Nx engineers. We don’t often talk about teachers – what is their N? To describe Ray as a 10x teacher doesn’t cut any mustard. Imagine a googol (yep – neither can I). Imagine a googolplex. Imagine a googolplex + 1. Ray is a ‘that’x teacher.

So let this be a lesson. Stay in touch with your mentors and teachers – or one day you might read a webpage authored by a Professor and that page might make you feel regret for conversations never continued.

Thank you for teaching me that too, Ray, even after you’ve gone.

Photonics for life!

Helsuit (base) v0

Call sign: “sleeper-service”. Another first stab:

Software (and some hardware):

OS evaluation – perhaps Android 14 on Rpi 4 or 5?

https://konstakang.com/devices/rpi5/AOSP14/

Cluster management for heterogeneous computing: k8s / docker swarm or similar

Messaging core: MQTT / mosquitto

Initial centralised inter-suit data sharing / message passing: something like UDX (https://www.urbandatacollective.com/)

Certificate generation: ‘self’ certification, X509, ‘everyone is a trusted authority’ (links to call sign generation)

Message format: Smart Data Models / w3c web thing model or similar

Power connectors: stretchy, flexible cords, native USB etc for now – moving towards ‘something that clicks in’ / mag safe? Moving further towards a suit with wireless charging pads in strategic places?

Wireless: WiFi / Bluetooth on RPi 4. Bluetooth / WiFi on samsung device. Bluetooth on Bangle.js. Native wiring for temperature / flex sensors (bluetooth receiver units?)

Subtle Physical Computing

And so, after absorbing a few different sorts of literature, here’s my first stab at a) ‘developing in the open’ and b) attempting to reconcile my software engineering practice with elements of philosophy.

I give you draft 1 of the ethos of ‘Subtle Physical Computing’ aka SubPhys Computing or, in short, SC. Special Circumstances may become the norm, who can say? (SPC is a more valid acronym but I had to get a Culture reference in there somewhere)

It is an offshoot of the idea of ‘pervasive computing’ from the world of IoT. With any luck, the ethos will grow. It is not designed to turn a profit. It is designed to change. Olamina’s ‘Earthseed’ aside, God is Change. AI is not God. For all the hells on earth, there is Helsuit. For the rest of us, there’s Earthsuit.

TL;DR:

“””

The most useful things in the world were invented by people who have no names. No one ever made ‘serious’ money from the fundamentals: Maxwell did not patent his equations. Truly useful things are developed by civilizations across centuries. Some of those centuries are already ending. Some begin today. Take your names off your products. Share your ideas and your implementations. Make them – and set them free.

Draft 1:

We are Helsuit
Wear Helsuit
We_ar_He_ls_ui_t
We are Helsuit

Helsuit: compatible with birthday suit.

Helsuit: you’re already wearing it.

We raise Helsuit

  • we believe the first draft of anything is shit
  • we read books like Doughnut Economics, The Deficit Myth, Everything you need (the A L Kennedy one), Jonathan Strange & Mr Norrell, The Metamorphosis, The Old man and the Sea, Harry Potter, Thus Spake Zarathustra and, with healthy pinches of salt and sugar, Maps of Meaning, Bleak House, Jane Eyre, Three Body Problem, Zima Blue, Surface Detail, Excession, Grief is the Thing with Feathers, A Man called Ove, Zen and the Art of Motorcycle maintenance, Lord of the Rings, Robert Webb’s autobiography, (Memories, reflections, dreams), The Parables of the Sower and Talents, The Wasp Factory
  • we believe in giving things away
  • we believe not in paying for things, but working for things. Although money can still change hands and lives
  • we believe that ideas are worthless
  • we believe the best ideas should be allowed to percolate without pressure from profit or management
  • we believe that an idea, however good, should not be all encompassing. We First!
  • we believe that moonshots are worth taking – with calculated guesstimates. We know there are many moons out there!
  • we believe that if you want to go fast, go alone. If you want to go far, go together.
    • it: Chi va piano va sano e va lontano
    • phonetically: ki vah peeano vah saano ey vah lontaanoh
    • Google translate says: ‘Who goes slow and steady wins’
  • we believe that theory without application is like bread without butter: you need damn good bread to get most people excited. For good bread contact Dusty Knuckle bakery
  • we believe that things don’t always go to plan: the goal you expect might not arrive. What arrives might not be the goal
  • we believe in ‘subtle (and frequently wearable!) computing’
  • we believe that distrubuted systems running locally are easier to reason about than those running remotely
  • we believe that computing is art and should be free and open to all (as it already is) (like the Tate Modern)
  • we believe that art should be made with discipline (Thanks Barbara!)
  • we believe that great artists share
  • we believe in taking what we need and giving back what we can afford
  • we believe in the full context of the mantra: “optimization is the root of all evil”
  • we believe that implementations should tend towards freedom the more people they touch (or, the more people that wear them…)
  • we believe that your Mum and Grandma should be able to use the implementation, unguided
  • We believe that ‘apps’ only lead to more apps. ‘Domains’ and ‘use cases’ lead to more solutions
  • we believe not in reinventing the wheel, but making the wheel fit your car (or suit)
  • we believe the best interface is no interface
  • we believe the best interface is your interface
  • we believe that the domain of in-body monitoring should be part of the platform (to enable preventative medicine)
  • we believe that the domain of out-body monitoring and maintenance should be outside, and perhaps connected to, the platform (to treat acute matters, e.g. brushing teeth – we are not inspector gadget)
  • we believe that not all out-body systems need to be connected: your toothbrush does not need to be connected to the Internet, nor does your toaster
  • we believe that technology can be used to connect us to our planet (not in an Avatar way, dw, just by allowing us to be outside for longer)
  • we believe that a broken escalator is still a staircase
  • we believe in a ‘personal platform for platforms’ (PPfP)
  • we believe in playing the hands we are dealt
  • we believe in an ‘always on’ architecture
  • we believe that ‘openness’ leads to fewer vulnerabilities in most contexts
  • we believe in the largest attack surface on earth: all of us
  • we believe no one is too small to make a difference (Thanks Greta!)
  • we believe that wobbly bridges can be fixed by the same people that made them wobble – without loss of reputation. There is usually a precedent!
  • we believe good technology lasts across centuries and across politics. Technology is not always ‘computers’ and ‘coding’ and ‘dashboards’ and ‘apps’ and ‘machine learning’
  • we do not believe in Earthseed or E/Acc. But, if that’s how it goes, Helsuit can help. This is our only wager. Pascal has something to do with it.
  • we believe in the stories we tell ourselves
  • we are conscious of hubris
  • we are kind to our nemesis
  • we wear hylozoic (hardware / software / wetware) architectures
  • we do not believe in altering carbon, we believe in shielding, absorbing and protecting it
  • we believe in incentivising healthy bodies and consentful physical touch
  • we believe that while thoughts and ideas are important, we are not them. Dawkinsian memes might be as important as food, shelter, recreation and companionship
  • we believe a digital twin should live on your body (personal data store on suit)
  • we believe in forwards compatibility at all costs
  • Memento mori
  • our corollary: “Be more Woz”

We want to avoid a Wall-E future. We also want to avoid a Matrix Revolutions future.

For more information and development progress, see https://helsuit.com

The obvious question: what in the bloody hell (Harry) is a Helsuit? To which I’d hazard a reply: “What is WordPress? Anything you want it to be”

Another, more mysterious and antagonistic reply: “We’ll show you”

Finally, we believe in the following, wrong attributed to Mahatma Ghandi:

First they ignore you, then they laugh at you, then they fight you, then you win.

The full, correct quote according to AP, is from Nicolas Klein:

First they ignore you. Then they ridicule you. And then they attack you and want to burn you. And then they build monuments to you.

https://apnews.com/article/archive-fact-checking-2315880316

This is how we fight. However, SPC does not expect monuments. We support subtle wins and quiet protest.

Wear Helsuit. We are Helsuit.


Pokemon Cobol -> day 0

Sitting in Bridge Street Coffee in Chester, UK as a family bereavement takes place, I met a chef who was also a software developer. He mentioned, off hand, Pokemon and then mentioned, off hand, COBOL, that most business oriented of programming languages.

It is not necessarily my business to create a Pokemon Go clone using COBOL as an ASCII art graphics driver, HTTP backend and hopefully a whole bunch of other things that the committee never intended it to be used for.

But also, why not?

This will hopefully be the first in a series of sporadic posts about my journeys into learning COBOL and harnessing its power to create a janky version of the most popular game of 2016.

https://pokemoncobol.com

[at the moment there’s a 302 set up to redirect back here. That will change!]

It shall be FOSS, of course:

https://github.com/InBrewJ/pokemonco-bol

I choose you!

Jason’s Applied Spectrum of Ned

There’s a thing about recursive acronyms in the world of tech / software / general nerd-ery. Think GNU (GNU’s Not Unix) and PHP (PHP: Hypertext Processor).

Or should I say general Ned-ery?

At my current place of work, feedback is very important to the team. We distribute feedback forms each half year in order to grow, collaborate and also gather useful data on our own personal pros and cons.

Since it’s also a super fun team, the feedback forms are entirely free(form); we’re at liberty to inject any form of nonsense that we choose.

I give you “Jason’s Applied Spectrum of Ned,” or “JASON”. I’ve been waiting for a good long while until my first recursive acronym presented itself. Maybe the next one will be the name of a library or something. Maybe not. Maybe I should just call it a day:

It’s a comical way to gauge how effective you are as a software engineer / developer / code monkey / software person in general. It might be applicable elsewhere, too. I’m a particular fan of the ambiguity of ‘WTF NED’.

Any copyrights belong to whoever makes The Simpsons, which might still be Fox and Matt Groening. Ned Flanders is absolutely not of my own creation. Any money that you make from JASON should probably to donated directly to the First Church of Springfield.

DitTICK: A conglomerate end-to-end stack for extensible Open IoT

Fork at will!
https://github.com/arupiot/DitTICK
https://github.com/arupiot/dummy-iot-gateway
https://github.com/arupiot/udmiduino
https://github.com/arupiot/UDMI-dummy

What is this about?

Ditto + TICK = DitTICK. This may or may not roll off the tongue. Would have also accepted: TICKto

This tutorial outlines a rough-and-not-ready end to end IoT solution that includes:

  • Creating a simple (and not production ready!) Arduino based device to spit out ‘telemetry’ data using an incomplete implementation of UDMI
  • Sending the ‘telemetry’ data to a dummy ‘gateway’. This can either be installed on something like a Raspberry Pi or the same laptop running DitTICK. The dummy ‘gateway’, at its core, is a 2 way MQTT client that translates the data from the Arduino into MQTT messages on an arbitrary topic
  • Connecting the ‘gateway’ to a Mosquitto broker
  • Connecting that broker to the TICK stack and Grafana for logging and monitoring what the Arduino is doing
  • Connecting Ditto to the broker to create near real-time digital twin of the Arduino
  • Using a basic Angular ‘web app’ to toggle the LED connected to the Arduino via the Ditto REST API
  • [optional] Deploying parts of the system to GCP (potentially for free)
  • [optional] Deploying the web app to AWS (also potentially for free)

Some terms above are in single quotes, e.g. ‘telemetry’ and ‘gateway’. They are so labelled because they intend to simulate production grade and MUCH more fully featured devices made by companies like this. All we’re doing is this:

The only purpose of slotting together these technologies to toggle an LED on and off via the internet is to provide the reader with an appreciation for the basic elements that are common to most IoT systems. There is very little concern for security in the system I’m going to cover. There is also no concern for any sort of automated commisioning of additional devices that could be added to the system. It’s entirely possible to add these features: this software is extensible! Predictably, the only limits are those of will, imagination, time and cold hard cash. On we go!

What I won’t cover

Git. You’ll need to pull the 3 repos listed at the top of this page to make things work. git clone ... is all you need right now but you’ll need git installed. You could also download the ZIPs of the repos, whatever works for you.

Various parts of the system are written in (basic) C++, Python and JavaScript. There’s a bit of HTML in there too, of course. We’ll be testing things as we go using a UNIX-like command line and eventually clicking around web consoles of two public cloud providers. I’ll assume you have an appreciation for this stuff. Most of the code is quick specific to this tutorial but if you really don’t know where to start and you’re completely out of your depth, perhaps start with this fun HTML tutorial. Come back after when your journey leads you back 🙂

https://www.websiteplanet.com/blog/html-guide-beginners/

I’m not using any fancy features of these languages, so if you have some experience with programming you should be okay. If you have a lot of experience with programming, you’ll notice how thrown together everything seems/is 🙂

I won’t cover what an Arduino is or how to debug electronic circuits. I also won’t cover what a Raspberry Pi is or how to troubleshoot if things go awry. The communities for both of these wondrous things are more than ample. I also won’t go into great detail about what Telegraf, InfluxDB and Chronograf are – I’ll be covering what to do with them, though.

Most of the software runs using Docker and docker-compose. If you’re not familiar with the containerised style of software development, never fear. The docker run... and docker-compose... commands should work out of the box – but I won’t be covering what Docker is. Long story short, it makes the code work the same on both my machine and your machine, as well as in the fabled cloud. It also means you don’t have to install hundreds of dependencies to get started.

We’ll also be creating a simple local network that will communicate with another network in the cloud. I’ll assume you have at least a passing knowledge of basic TCP/IP stuff including the concept of ‘link local’ IP addresses and *NIX commands like ping and arp.

The code has been tested on local machines running Ubuntu 18.04+ or MacOS Catalina+. The dummy ‘gateway’ code has been tested on HypriotOS (a fantastic bit of kit) and once again on Ubuntu 18.04+ or MacOS Catalina+. I won’t be covering how to install Ubuntu/HypriotOS or how to use them in any great detail. For the most part, I will assume you’re running commands on Ubuntu/Debian. It won’t be much trouble to make them work on MacOS.

As a general rule, I’ll add resources to links that might fill gaps in knowledge but I won’t reference or explain them, lest their anchors become broken.

It can certainly be done, but I won’t be covering how to make this work on Windows. Something, something, something, embrace open source, Linux is your IDE, something something something, complete.

The Joy of End-to-End

There’s something immensely satisfying about creating a complete system that (sort of) works. This tutorial came out of a vague set of requirements and with little to no ideas on how to put all the parts together.

Admittedly, all of this seems convoluted when one could simply buy an off the shelf IoT solution. However, making each part of the solution from close to ground up gives a glimpse into how to critique off the shelf offerings in their entirety. You might also realise that existing offerings don’t offer something that you and your team could offer.

Plus, it’s free and you can do things in your own house with it with an extra effort. So, without further ado, let us start at the first end.

Dev Environment

You should have the following installed on your machine:

– A web browser, preferably the latest version of Chrome or Firefox
– Git
– Docker 18.09+
– Python 3 + pip
– The Arduino IDE
– Within the Arduino IDE, the ArduinoJson library
– [optional but strongly suggested] Postman
– [optional] Node.js 9+

I’d recommend downloading the three repositories to a directory called ‘DitTICK-dev’ or something. So:

$ cd /your/path/to/DiTTCK-dev
$ git clone https://github.com/arupiot/DitTICK.git
$ git clone https://github.com/arupiot/udmiduino.git
$ git clone https://github.com/arupiot/dummy-iot-gateway.git

The UDMIduino

UDMI + Arduino = UDMIduino. A naming theme is emerging

https://github.com/arupiot/udmiduino

Points for neatness: nul. The wires on the right side of the breadboard do absolutely nothing except make it seem like I need this much breadboard.

The UDMIduino is an Arduino with a button to toggle an LED, one LED acting as a sensor and other LED to act as an output to that sensor. The sketch is written in a mangled way. That’s it.

Its only purpose is to spit out packets of information on the state of each of the LEDs over the serial line and accept one command to toggle the LED. I used an Arduino Uno, some spare components that I had lying around and a breadboard. In theory, you can use any Arduino device and any components you want – but you’ll have to make some changes to the sketch. Speaking of the sketch, here it is:

#include <ArduinoJson.h>
int count = 0;
int x = 2000;
int y = 500;
int bub = 0;
int incoming_byte = 0;
boolean remote_override = false;
StaticJsonDocument<200> udmi_out;

void setup() {
  pinMode(3, OUTPUT);
  pinMode(A0, INPUT);
  pinMode(5, OUTPUT);
  pinMode(2, INPUT_PULLUP);
  Serial.begin(9600);
}

void toggleLED() {
  digitalWrite(3, !digitalRead(3));  
}

void loop() {

  // Monitor inputs

  int bib = digitalRead(2);

  if (bib == HIGH) {
    remote_override = false; 
  }
  
  if (bib == HIGH && bub == LOW && remote_override == false) {
    toggleLED();  
  }
  
  int bob = analogRead(0);
  analogWrite(5, 255-bob);

  bub = bib;

  if (Serial.available() > 0) {
    remote_override = true;
    // read the incoming byte:
    incoming_byte = Serial.read();

    // say what you got:
    Serial.print("I received: ");
    Serial.println(incoming_byte, DEC);
    if (incoming_byte == 't') {
      toggleLED() ; 
    }

  }

  // Generate output
  
  
  //  build udmi

  udmi_out["version"] = 1;
  udmi_out["timestamp"] = "0";
  udmi_out["points"]["lux_level"]["present_value"] = bob;
  udmi_out["points"]["lum_value"]["present_value"] = digitalRead(3)*100;
  udmi_out["points"]["dimmer_value"]["present_value"] = 255-bob;

  //  write udmi
  
  serializeJson(udmi_out, Serial);
  Serial.println();
  
}

It’s not immediately important to understand the sketch – just appreciate that the LED that toggles simulates digital actuation (on/off), the other LEDs simulate analogue actuation and a sensor. That is:

– Toggling LED = ‘the luminaire’ = lum_value
– The reversed polarity LED = ‘the sensor’ = lux_level
– the LED that reacts to ‘the sensor’ = ‘the dimmer’ = dimmer_value

So, let’s build us a UDMIduino.

1. Construct the circuit:

Made with Fritzing. FZZ file and full size png are provided in the udmiduino repository. I’m aware there aren’t any current limiting resistors on the LEDs – add them if you like. The USB port I’m using doesn’t have enough juice to make this circuit dangerous in any way

2. Upload the sketch to your Arduino in the usual way. Then it will become the UDMIduino!
3. Check that the button toggles the ‘luminaire’ on and off
4. Cover the ‘sensor’ and make sure that the ‘dimmer’ changes. It will flash weirdly or do something else based on the stability of your power supply/USB power line.
5. In the Arduino IDE, monitor the serial line (baud rate 9600) and ensure that data is coming through. It should look something like the print out below after the screen command. If you’re having trouble (as I did) getting the Arduino IDE serial monitor to work on Ubuntu, please see here.

On Ubuntu/Debian/Linux Mint/Pop OS/etc, you can also monitor the serial line with:

$ sudo screen /dev/ttyACM0

To find the ID of the serial device, since it may vary for a number of reasons, you can monitor device status on your machine with:

$ dmesg -wH

If, while watching the output of dmesg, you unplug and replug the USB cable from your Arduino you’ll be able to figure out what device (/dev/xxx) it’s paired with.

The output will look something like this:

The raw data coming from the UDMIduino’s serial cable

That’s it for the UDMIduino! We now have an ‘IoT device’ that does something marginally useful.

The Dummy Gateway

https://github.com/arupiot/dummy-iot-gateway

Why is it such a dummy? It’s a dummy because all it can do is talk to a single UDMIduino. In the real world of IoT, a gateway is a single point of contact for many devices to talk to at once. A real gateway allows many things pass through. The dummy gateway has no such agenda.

The core of the dummy gateway is this Python script. A stripped down version of this is shown here:

import serial
from time import sleep
from serial import Serial
import paho.mqtt.client as mqtt

SERIAL_CONN = '/dev/ttyACM0'
SERIAL_BAUD = 9600
ser = Serial(SERIAL_CONN, baudrate=SERIAL_BAUD, timeout=None)

def mqtt2serial(message, ser):
    ser.write(b't')

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        client.connected_flag = True  # set flag
        print("connected OK")
    else:
        print("Bad connection Returned code=", rc)

def on_message(client, userdata, message):
    payload = str(message.payload.decode("utf-8"))
    print("message received " ,payload)
    print("message topic=",message.topic)
    print("message qos=",message.qos)
    print("message retain flag=",message.retain)
    mqtt2serial(payload, ser)


mqtt.Client.connected_flag = False  # create flag in class
broker = "YOUR_MOSQUITTO_HOST"
client = mqtt.Client("DitTICK Dummy Gateway")  # create new instance
client.on_connect = on_connect  # bind call back function
client.on_message = on_message #attach function to callback
client.loop_start()

print("Connecting to broker ", broker)
client.connect(broker)  # connect to broker

# Subscribe to the topic from ditto (or elsewhere)

while not client.connected_flag:  # wait in loop
    print("In wait loop")
    sleep(1)

print("Subscribing to LED toggle topic... ")
client.subscribe("dittick/UDMIduino-000/events")

print("in Main Loop")
print("Publishing...")

while(True):
    line = ser.readline()   # read a '\n' terminated line
    try:
        decodedLine = line.decode('utf-8').rstrip()
        ret = client.publish("dittick/UDMIduino-000/events", decodedLine)
    except:
        print('Serial is a bit janky, retrying...')
        sleep(0.5)

client.loop_stop()  # Stop loop
client.disconnect()  # disconnect

The mqtt2serial function is innocuous enough but highly important. We’ll come back to that later.

Local gateway

First things first, let’s start a Mosquitto broker on your local machine. Assuming…

– You’re working on an Intel/AMD x86_64 computer
– You have Docker 18.09+ installed

…you can start an entirely insecure and unconfigured mosquitto broker with:

$ docker run -it -p 3389:1883 -p 9001:9001 eclipse-mosquitto

This will make it accessible via localhost:3389. Now, with the UDMIduino still plugged into your computer and streaming data through the serial line, start the serial_mon.py script. The ouput should look something like below. First install the dependencies with:

$ pip3 install -r requirements.txt

Then start with:

$ sudo python3 serial_mon.py 
Connecting to broker  localhost
In wait loop
connected OK
in Main Loop
Publishing...

If you see ‘Publishing…’ it means that data is successfully being channeled from the UDMIduino through the broker. Based on the topic set in serial_mon.py, you can monitor the broker using mosquitto_sub or MQTT.fx. There are more sanity checking details in the README. Generally, successful connection with the broker will look like this:

# mosquitto_sub -t dittick/UDMIduino-000/events
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":165},"lum_value":{"present_value":100},"dimmer_value":{"present_value":90}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":48},"lum_value":{"present_value":100},"dimmer_value":{"present_value":207}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":107},"lum_value":{"present_value":100},"dimmer_value":{"present_value":148}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":128},"lum_value":{"present_value":100},"dimmer_value":{"present_value":127}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":48},"lum_value":{"present_value":100},"dimmer_value":{"present_value":207}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":145},"lum_value":{"present_value":100},"dimmer_value":{"present_value":110}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":84},"lum_value":{"present_value":100},"dimmer_value":{"present_value":171}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":51},"lum_value":{"present_value":100},"dimmer_value":{"present_value":204}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":166},"lum_value":{"present_value":100},"dimmer_value":{"present_value":89}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":48},"lum_value":{"present_value":100},"dimmer_value":{"present_value":207}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":111},"lum_value":{"present_value":100},"dimmer_value":{"present_value":144}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":130},"lum_value":{"present_value":100},"dimmer_value":{"present_value":125}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":47},"lum_value":{"present_value":100},"dimmer_value":{"present_value":208}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":146},"lum_value":{"present_value":100},"dimmer_value":{"present_value":109}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":88},"lum_value":{"present_value":100},"dimmer_value":{"present_value":167}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":51},"lum_value":{"present_value":100},"dimmer_value":{"present_value":204}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":165},"lum_value":{"present_value":100},"dimmer_value":{"present_value":90}}}\r\n'
b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":47},"lum_value":{"present_value":100},"dimmer_value":{"present_value":208}}}\r\n'

There we have it, an Arduino connected to an MQTT broker with some Pythonic glue. Since the broker is the point where all other useful things can make contact, there are now essentially unlimited things we can do with these data parcels that show device state. We can subscribe to the UDMIduino topic with any number of programming languages and systems. We can store the data somewhere, we can do real time analytics, we can set up alerts, trigger emails, control other devices, kitchen sinks, anything!

We’re going to slim down those possibilities and start simple though – next we’ll store the data in a database and display it on a nice set of graphs on a website. Enter the TICK stack and Grafana.

A Raspberry Pi Gateway

I realised the “Raspberry Pi Gateway” deserves a step-by-step tutorial of its own. A link will be posted here soonish!

TL;WR – the “Raspberry Pi Gateway” is simply the serial_mon.py script running on a Raspberry Pi (the UDMIduino is plugged into the R’Pi via USB too).

No Arduino? No Raspberry Pi? No Problem!

Option 1: Unplug the Arduino

All being well, if you:

– shut down the serial_mon.py script
– unplug the USB cable from your machine
– restart serial_mon.py

…the serial_mon.py script should go into failsafe mode and start sending random messages with the same JSON format as the physical UDMIduino:

$ python3 serial_mon.py 
Could not connect to the serial line, activating DUMMY_MODE
Connecting to broker  localhost
In wait loop
In wait loop
connected OK
Subscribing to LED toggle topic... 
in Main Loop
Publishing...

Option 2: Use the UDumMI

Code: https://github.com/arupiot/UDMI-dummy

I wrote another little thing for those who don’t have Arduinos, Raspberry Pis or other miscellaneous electronic paraphernalia, find it here.

By default, the UDMI-dummy (or ‘UDumMI’) connects to the services defined in this tutorial. Run it with:

$ cd /your/path/to/UDMI-dummy
$ python3 run.py -i

A retro curses interface should pop up. Press the spacebar to start autosending messages:

Using the UDumMI is entirely unecessary but it has the advantage of simulating a button press, in this case using the ‘w’ key.

TICK and Grafana

I could have gone with TICKraf or GraTICK but, at some point, the madness has to stop

Telegraf conf: https://github.com/arupiot/DitTICK/tree/master/docker/telegraf
Grafana conf: https://github.com/arupiot/DitTICK/tree/master/docker/grafana

TICK stands for:

Telegraf
InfluxDB
Chronograf
Kapacitor

For our purposes here, we won’t be using Kapacitor so we’re really only using it to 75% of its potential. Right?

We’ll use Telegraf to connect to the our dummy gateway/broker and a database. We’ll store our UDMIduino data in InfluxDB and we’ll test that everything works with Chronograf before hooking up Grafana, all on our local machine (for now).

Config

The first step is to properly configure Telegraf. We need to configure an input that talks to our dummy ‘gateway’ broker (which should still be running on localhost:3389) and an output that funnels the MQTT payloads into InfluxDB. Telegraf uses .conf files to set everything up. The most important parts of our config looks something like this:

...

# Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  servers = ["tcp://your.mosquitto.host.com:3389"] # EXAMPLE ONLY!
  name_suffix = "_udmiduino"
## MQTT QoS, must be 0, 1, or 2
  qos = 0

## Topics to subscribe to
  topics = [
    "dittick/#",
  ]
  data_format = "json"

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
  urls = ["http://localhost:8086"] # required
  ## The target database for metrics (telegraf will create it if doesn't exist)
  database = "dittick" # required

  ## Retention policy to write to. Empty string writes to the default rp.
  retention_policy = ""
  ## Write consistency (clusters only), can be: "any", "one", "quorum", "all"
  write_consistency = "any"

  ## Write timeout (for the InfluxDB client), formatted as a string.
  ## If not provided, will default to 5s. 0s means no timeout (not recommended).
  timeout = "5s"
  username = "root"
  password = "root"

...

The Grafana config is more straightforward, we’re just setting up usernames, passwords and Grafana URLs here. One of Grafana’s shining features is pre-deployment dashboard provisioning. We’ll use this to create a dashboard for our UDMIduino so that when we run TICK and Grafana we’ll start seeing data straight away.

Grafana dashboards are represented by a fairly large and baffling JSON structure. For the UDMIduino, it looks like this. All this is doing is creating a dashboard called ‘UDMIduino’, connecting it to the Influx datasource and creating 3 graphs to show what the LEDs are doing. That is, lum_value, lux_level and dimmer_value.

Running and testing TICK

This is where everything starts to come together. Navigate to where you cloned the DitTICK repository. In there, there is directory called ‘docker’:

$ cd /your/path/to/DiTTCK/docker

Shut down anything else you may have running on TCP port 80. Then, run:

$ docker-compose -f docker-compose.tick.grafana.yaml up -d
Creating network "docker_default" with the default driver
Creating tick ... done
Creating nginx ... done

This will start Telegraf, Influx, Chronograf and Grafana with the right configuration to connect to the Mosquitto broker that is, hopefully, still running on your machine and still accepting data from the UDMIduino. Also included in this docker-compose config is an Nginx instance that routes the various TICK containers to sensible endpoints. Don’t worry if:

a) You don’t fully know what Nginx is for
b) you don’t care about Nginx right now
c) you don’t understand how the nginx.conf works

It’s just there to avoid having to navigate to the TICK services without connecting to random TCP ports. It’ll also make it easier to deploy this stack elsewhere later on. If you are familiar with Nginx, this is one of many ways to do this and I’m no expert!

All being well, you should be able to navigate to http://localhost/chronograf and see something that looks like this:

Chronograf is described as the user interface to the TICK stack. We’ll be using it to ensure everything is set up correctly in a graphical way to test some queries to Influx.

First, check that Chronograf is successfully connected to InfluxDB (click the spanner/wrench icon):

Next, head over to the ‘Explorer’ window (the icon that looks like a line graph/a lightning bolt that’s fallen over):

Your screen will be empty initially!

The Explorer allows you to interactively query InfluxDB data. On the far left panel at the bottom of the window, there should be a database called dittick.autogen – this is the database we told Telegraf to create.

The middle panel are all the datapoints Chronograf has found inside InfluxDB. The far right window are the fields within each datapoint. You should see one called dittick/UDMIduino-000/events. Select all of the fields within and see the data from the last five minutes:

Verily, a 1983 colour pallette

Seeing the UDMIduino on Grafana

Up until now, we’ve seen glimpses of what our UDMIduino is doing, be it line by line in text format on a terminal or value by value on Chronograf. I mentioned that it’s possible to provision dashboards in Grafana earlier. Well, the good news is they’re already provisioned and working! Navigate to http://localhost/dashboardsand you should see a log in screen to Grafana:

If you’ve ONLY JUST STARTED Grafana you might see a 502 bad gateway error @ localhost/dashboards. Wait around 15 seconds and everything should work fine

– username: admin
– password: dittick

Click on the ‘UDMIduino’ dashboard and there it is!



By default, the panels will update every 5 seconds. So press the button on the UDMIduino circuit to toggle the LED on and off. After a maximum delta of 5 seconds, you’ll see the lum_value change. If you completely cover up the ‘sensor’ LED, you’ll see dimmer_value and lux_level go to one extreme or the other. Lovely:

Ditto: an LED API?

So, we have an Arduino with some LEDs and a button. We have it sending its state to a time series database through an MQTT broker. We can see some nice web based graphs of what’s happening, pretty much in real time. But what about if we want to control the LED remotely from anywhere in the world?

Well, we’d have to create some sort of software system that connects to the MQTT broker via the internet. Easy enough, we could just write another Python script. But it would also be nice if it exposed a REST API so we could also get the last state of the device too. While we’re at it, we could also expose a websocket API so we could get real time updates. Our REST API would need to work both ways though. We need to send something to the API to change things. And, while we’re at it, it would be nice if we could store the state of other devices in this system too. So it would need a database. And it would need to have security features too. Sounds like too a deep rabbit hole.

Thankfully, Eclipse Ditto does all of that already and it’s maintained by people who are far better than I am. So let’s use that.

Go back to the docker directory in the DitTICK repo:

$ cd /your/path/to/DitTICK-dev/DitTICK/docker

Presuming you’ve still got the docker-compose.tick.grafana.yaml stack running, shut it down with:

$ docker-compose -f docker-compose.tick.grafana.yaml down -v

The -v is very important: this removes the volumes associated with the running containers. When that’s all happy and deleted, run the start.sh script:

$ ./start.sh

The start script first pulls down a version of Ditto that I know to work (v0.9.0) and then starts up the TICK stack, Grafana, Nginx, a Mosquitto broker and Ditto all together using docker-compose. Depending when you’re reading this, v0.9.0 may be very old indeed!

Depending on your local machine, starting this entire stack could take some time. If your machine starts lagging or even crashing, skip to the section below where we deploy DitTICK to Google Cloud Platform.

I’m using Postman to interface with Ditto’s REST API. Included is a pre-baked collection that acts as a script, of sorts, for what we need to do. You can use any other tool you wish to interface with Ditto; if you’re a purist no one is stopping you using curl. The raw requests can be found in the root of the DitTICK repo: they’re the ones ending in .txt.

NB: you may need to tweak the raw requests to make them work. There are odd things that happen with line endings on different command line clients. I’d recommend using Postman – everyone will be happier that way. Import the collection and follow the rest of this text with ease.

Now, as I said, the Ditto stack can take some time to boot up. If you’re using the included Postman collection, you should see something like this:

Our friend, the 45 degree postie

The ‘Ditto health’ request should be set up to send a GET to http://localhost/health. If you send this request when Ditto is ready, you’ll get a { "status" : "UP" }` response:

If Ditto is not ready, you might get a 502 Bad Gateway response or similar. As we speak, I’m running Ditto on an Ubuntu host with a dual core i7 something-or-other and 16GB RAM, it normally takes around a minute to become stable.

When Ditto is ready, run the ‘Get things’ GET request:

This returns all of the so-called ‘Things’ that are currently registered in Ditto’s database. At the moment, we have no Thing. Which is as it should be since this is a fresh install.

Things, in Ditto speak, are devices or units that you want Ditto to talk to. In this context, a UDMIduino is a ‘Thing’. Ditto uses a MongoDB instance internally to store records of ‘Things’ (devices), connections to those devices and security policies to protect those devices from bad actors. Also, the T in IoT stands for ‘Things’. The naming is good.

To connect our UDMIduino to Ditto so we can take advantage of its device REST API, we first need to create a policy. Fire off the ‘UDMIduino policy’ request in the Postman collection, the response should be something like the following:

We’ve just created a policy called dittick:policy. We’ll use this to ‘secure’ a tunnel between our UDMIduino and Ditto’s REST API.

Next, we create a ‘connection’ (use the ‘Create connection’ Postman request):

Annoyingly, the response if this request is successful is meant to be null. If there’s an error, there’s something more informative, don’t worry. We’re POSTing the following payload to Ditto ‘Connectivity‘ service. Let’s unpack this a little:

{
    "targetActorSelection": "/system/sharding/connection",
    "headers": {
        "aggregate": false
    },
    "piggybackCommand": {
        "type": "connectivity.commands:createConnection",
        "connection": {
            "id": "mqtt-udmiduino-000",
            "connectionType": "mqtt",
            "connectionStatus": "open",
            "failoverEnabled": true,
            "uri": "tcp://mosquitto:1883",
           "sources": [
                {
                "addresses": [
                    "dittick/UDMIduino-000/events"
                ],
                "authorizationContext": ["nginx:ditto"],
                "qos": 0,
                "filters": []
                }
            ],
            "targets": [{
                "address": "dittick/UDMIduino-000/lum-value",
                "authorizationContext": ["nginx:ditto"],
                "topics": [
                "_/_/things/twin/events",
                "_/_/things/live/messages"
                ],
                "qos": 0
            }],
            "mappingContext": {
                "mappingEngine": "JavaScript",
                "options": {
                    "incomingScript": "function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {     const thingId = 'UDMIduino-000';     const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));     const jsonData = JSON.parse(jsonString);     const value = {         udmi: {             properties: {                 version: 1,                 timestamp: 0,                 points: {                     lux_level: {                         present_value: jsonData.points.lux_level.present_value                     },                     lum_value: {                         present_value: jsonData.points.lum_value.present_value                     },                     dimmer_value: {                         present_value: jsonData.points.dimmer_value.present_value                     }                 }             }         }     };      return Ditto.buildDittoProtocolMsg(         'open.iot',         thingId,         'things',         'twin',         'commands',         'modify',         '/features',         headers,         value     ); }"
                }
            }
        }
    }
}

There are a few important parts:

  • targetActorSelection, this is saying that we want to use the Ditto Connectivity API
  • piggybackCommand.type is what we want to do there
  • piggybackCommand.connection.id is how we’ll link this connection to our UDMIduino
  • piggybackCommand.connection.uri is what we’re connecting to (in this case the mosquitto broker started with our start.sh script). The broker is running via Docker with a hostname alias of mosquitto. In this configuration, tcp://mosquitto is only accessible from within other containers running in this stack
  • piggybackCommand.connection.sources is (are) the MQTT topics we want to associate with this connection
  • piggybackCommand.connection.targets is (are) where we’re sending messages back from Ditto’s REST API (there is also a WebSocket API but that’s out of scope for now). We’ll use this to send the toggle command back to the UDMIduino
  • *.*.authorizationContext relates to the policy we set up earlier
  • piggybackCommand.connection.mappingContext is how to programmatically map messages coming into Ditto from ANY source into a structure Ditto will understand. Here, we’re using the JavaScript mapping engine. There’s also a Java mapping engine but I have less than no idea about that. Docs are here and are not for the faint of heart!

Of these, piggybackCommand.connection.mappingContext is by far the most crucial. Here it is in full:

function mapToDittoProtocolMsg(headers, textPayload, bytePayload, contentType) {
    const thingId = 'UDMIduino-000';
    const jsonString = String.fromCharCode.apply(null, new Uint8Array(bytePayload));
    const jsonData = JSON.parse(jsonString);
    const value = {
        udmi: {
            properties: {
                version: 1,
                timestamp: 0,
                points: {
                    lux_level: {
                        present_value: jsonData.points.lux_level.present_value
                    },
                    lum_value: {
                        present_value: jsonData.points.lum_value.present_value
                    },
                    dimmer_value: {
                        present_value: jsonData.points.dimmer_value.present_value
                    }
                }
            }
        }
    };

    return Ditto.buildDittoProtocolMsg(
        'open.iot',
        thingId,
        'things',
        'twin',
        'commands',
        'modify',
        '/features',
        headers,
        value
    );
}

If you recall, our incoming message from the UDMIduino looks like this:

b'{"version":1,"timestamp":"0","points":{"lux_level":{"present_value":84},"lum_value":{"present_value":100},"dimmer_value":{"present_value":171}}}\r\n'

The above is being printed out by a Python interpreter, so here the b' means ‘bytes’. \r is the ASCII carriage return and \n is the ASCII linefeed character. This is low level stuff – this is what the UDMIduino is giving us. Ditto doesn’t care too much about lower level stuff, it wants strings of easy to understand text. That’s what the script is doing to create the jsonString variable.

value is the JSON structure Ditto wants and needs to function. All Ditto cares about is attributes and features. Every ‘Thing’ must have attributes (that don’t change that often) and features (which change more often). Both features and attributes can be arbitrary JSON structures.

In Ditto, we’ve set our UDMIduino Thing to have a few dummy attributes and one feature called udmi. The JavaScript mapping script simply rips out the important values from any incoming MQTT messages and forwards them on to Ditto. As it stands, this mapping script will only work for a Thing with a name of UDMIduino-000. That works for this tutorial, but in real life you’d want this kind of script to work for a subset of incoming messages with a certain structure and an arbitrary device ID.

So, after reading the word ‘Thing’ enough times, let’s finally add our UDMIduino to Ditto’s database. Use the ‘Create UDMIduino’ request in the Postman collection:

Now, if you resend the ‘Get things’ request in Postman, you should see that the values of features.udmi.properties.points.lux_level and ...dimmer_value change if you keep on resending ‘Get things’. If you push the button the UDMIduino to toggle the LED, lum_value will to 100 or 0 if it is toggled on or off.

What we’ve just created is a Digital Twin of the UDMIduino. Congratulations! But on its own, a Digital Twin is lonely and useless. Its real life partner, its soulmate, is so far away – perhaps they’ll never even meet in real life. Tragic to the point of paradoxical.

As is sometimes the case, however, adding some extra technology into the mix can make the Digital Twin enslaved forever an unwilling customer useful! It’s already useful enough being able to monitor the last known state of our UDMIduino. If we unplug the Arduino then the last state is saved in Ditto’s database. In a more complex situation, this might give us some insight into why something important started to malfunction.

What we’re about to do is ‘complete the loop’. The ‘connection’ we made earlier coupled with the mqtt2serial method in the serial_mon.py Python script in our dummy ‘gateway’ form the path from Ditto back to the UDMIduino. Let’s remind ourselves of that method:

import serial
from time import sleep
from serial import Serial
import paho.mqtt.client as mqtt

SERIAL_CONN = '/dev/ttyACM0'
SERIAL_BAUD = 9600
BROKER = "localhost"
BROKER_PORT = 3389
ser = Serial(SERIAL_CONN, baudrate=SERIAL_BAUD, timeout=None)

# To become a generic mapping function
# mqtt2bacnet could feature BAC0?

def mqtt2serial(message, ser):
    ser.write(b't')

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        client.connected_flag = True  # set flag
        print("connected OK")
    else:
        print("Bad connection Returned code=", rc)

def on_message(client, userdata, message):
    payload = str(message.payload.decode("utf-8"))
    print("message received " ,payload)
    print("message topic=",message.topic)
    print("message qos=",message.qos)
    print("message retain flag=",message.retain)
    mqtt2serial(payload, ser)


mqtt.Client.connected_flag = False  # create flag in class
client = mqtt.Client("DitTICK Dummy Gateway")  # create new instance
client.on_connect = on_connect  # bind call back function
client.on_message = on_message #attach function to callback
client.loop_start()

print("Connecting to broker ", BROKER)
client.connect(BROKER, BROKER_PORT, 60)  # connect to broker

# Subscribe to the topic from ditto (or elsewhere)

while not client.connected_flag:  # wait in loop
    print("In wait loop")
    sleep(1)

print("Subscribing to LED toggle topic... ")
client.subscribe("dittick/UDMIduino-000/lum-value")

print("in Main Loop")
print("Publishing...")

while(True):
    line = ser.readline()   # read a '\n' terminated line
    try:
        decodedLine = line.decode('utf-8').rstrip()
        ret = client.publish("dittick/UDMIduino-000/events", decodedLine)
    except:
        print('Serial is a bit janky, retrying...')
        sleep(0.5)

client.loop_stop()  # Stop loop
client.disconnect()  # disconnect

The most important bits are:

def mqtt2serial(message, ser):
    ser.write(b't')
...
def on_message(client, userdata, message):
    payload = str(message.payload.decode("utf-8"))
    print("message received " ,payload)
    print("message topic=",message.topic)
    print("message qos=",message.qos)
    print("message retain flag=",message.retain)
    mqtt2serial(payload, ser)
...
client.on_message = on_message
...
client.subscribe("dittick/UDMIduino-000/lum-value")
...

When we tell the client to subscribe to the topic: dittick/UDMIduino-000/lum-value, we’re telling it to listen to the topic we set up in the piggybackCommand.connection.targetsbit of our Ditto ‘connection’. When we receive a message, we’ve set up a callback (on_message) to print out the message and then call mqtt2serial. This sends the character ‘t’ back to the UDMIduino. In the UDMIduino sketch we have this block:

...
if (Serial.available() > 0) {
    remote_override = true;
    // read the incoming byte:
    incoming_byte = Serial.read();

    // say what you got:
    Serial.print("I received: ");
    Serial.println(incoming_byte, DEC);
    if (incoming_byte == 't') {
      toggleLED() ; 
    }

  }
...

When the UDMIduino unit receives a ‘t’, toggle that LED. This will happen if we get any message on the topic dittick/UDMIduino-000/lum-value – there’s no sort of validation in serial_mon.py. Why not add it yourself? However, onwards…

You might have done this already, but if not, try the ‘Test toggle’ request on Postman. If the lum_value LED was off, it should now be on and vice versa. Loop completed! You can still turn the LED off with the button in the UDMIduino circuit too. Hooray!

And with that, we have our UDMIduino API. We can get the last state of our device and control it using RESTful calls. That, coupled with TICK and Grafana gives us a complete history of what the device has done in the past and a way to control it remotely now.

As it stands, the most important parts of the system are now in place. You can connect to Ditto’s REST API and do things with the UDMIduino with any number of systems. With a bit of tweaking, you can also connect to influx directly, say using a Python script, to query the data and do other useful things. You could build a mobile app to control the LED. You could explore Ditto’s websocket functionality and ingest the data into an entirely different pipeline. You could even extend the serial_mon.py Python script to automatically add the UDMIduino to Ditto whenever it’s plugged in to a new system. Et cetera. Worlds/oysters.

The following sections deal with making a basic web app that does the same thing as Postman to toggle the LED before finally deploying everything (except the dummy ‘gateway’) to the cloud so that we can monitor/control the UDMIduino from anywhere in the world. The point of this tutorial is to explore the stack so these things aren’t strictly necessary, but I find that nothing feels real until it leaves one’s local machine and until one can do something on a mobile device.

The following sections also skim through a lot fairly quickly – extra dependencies will be needed in your dev environment!

[optional] The Big Red Button

Every important machine has a Big Red Button

https://github.com/arupiot/udmiduino/tree/master/web-ui

Now, I’m a big fan of big red buttons. So let’s use a website with a big red button to toggle that LED. Find the code at the link above. The app is written in ~Angular 5.2, which at this point is a fairly old version. No matter – as ever I’m not using any fancy features of Angular and, if you’re well versed with Angular, this code should work with later versions of the framework too.

If you know what you’re doing, I’ve provided a static build of the app that talks to the Ditto instance started above with:

$ ./start.sh

If not, you’ll need Node.js installed on your machine. Anything above version 8 should work fine, versions above 10 are recommended. Navigate to the directory with the big red button app in the udmiduino repo:

$ cd /your/path/to/udmiduino/web-ui

We need to install the dependencies of the app with:

$ npm install

If all has gone well (you may need to remove the package-lock.json file if Node complains) then you can start the web app with:

$ npm start

This should automagically launch the app in a new tab on your operating system’s default browser on http://localhost:5000, as dictated by the run script in package.json. You should see:

okay, it’s not strictly a big red button, more of a hello world/ng new Angular app with a grey background and a blue halo

Hit the button! Is the LED toggling on and off? Can you see the value changing on Grafana? Can you still see the value changing in the ‘Get things’ Postman request to Ditto? Good! If not, there’s something wrong with your configuration.

Let’s take a look at what the app is doing when the big red A button is clicked:

import { Injectable } from "@angular/core";
import { HttpClient } from "@angular/common/http";
import { HttpHeaders } from "@angular/common/http";

const httpOptions = {
  headers: new HttpHeaders({
    "Content-Type": "application/json",
    Authorization: "Basic ZGl0dG86ZGl0dG8="
  })
};

@Injectable()
export class DittoServiceService {
  dittoURL: string = "http://localhost/api/2/";
  ledToggleEndPoint: string = "things/open.iot%3AUDMIduino-000";
  toggleBody: any = {
    version: 1,
    timestamp: "0",
    points: {
      lux_level: {
        present_value: 0
      },
      lum_value: {
        present_value: 0
      },
      dimmer_value: {
        present_value: 0
      }
    }
  };

  constructor(private http: HttpClient) {}

  toggleLED() {
    return this.http.put(
      this.dittoURL + this.ledToggleEndPoint,
      this.toggleBody,
      httpOptions
    );
  }
}

When the A button is clicked, we’re simply sending the structure Ditto wants to the Ditto REST API, exactly the same way Postman was earlier. That’s it!

If you’re on a home network (e.g. a regular domestic WiFi router) the Big Red Button app will also be served over that network. Find your IP address with ifconfig on *NIX/Debian/Mac systems (or ipconfig on Windows):

$ ifconfig
...
wlp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.24.3.197  netmask 255.255.252.0  broadcast 172.24.3.255
        inet6 fe80::6e59:f85c:ef3d:1b80  prefixlen 64  scopeid 0x20<link>
        ether b8:08:cf:3e:8f:d4  txqueuelen 1000  (Ethernet)
        RX packets 638191  bytes 619106510 (619.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 351308  bytes 79868724 (79.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

My IP address will be different to yours, but if I go to http://172.24.3.197:5000, I will see the Big Red A Button. If you don’t see it, your firewall may be blocking TCP port 5000. DitTICK is also being served over your home network. This means that, if your mobile phone/tablet/whatever are also connected to your home WiFi, you can control the LED with them too! We’ll need to change the following line in the Angular app to make it work though:

dittoURL: string = "http://localhost/api/2/";

will become:

dittoURL: string = "http://{{THE IP OF YOUR MACHINE ON YOUR NETWORK}}/api/2/";

e.g.

dittoURL: string = "http://172.24.3.197/api/2/";

The Big Red A Button app might take a while to load, but when it does you should be able to toggle the LED as usual.

Naturally, because Ditto works, Grafana works too, just go to http://your_machine's_ip/dashboards and log in again with:
– username: admin
– password: dittick


Now, imagine if we could change the IP in the Angular app/DiTTICK to a public IP address so we could control the LED from anywhere. Enter GCP and, additionally, the most valuable virtues of Docker and docker-compose.

[optional] The Big Red Button: Worldwide

DiTTICK on GCP

What we’re going to do now is use docker-compose to run our DitTICK stack on a virtual machine (VM) in the cloud. I’ll be using Google Cloud Platform (GCP) because:

a) If you don’t have a GCP account already, (and at the time of writing this) you get $300 free credit for a year. DitTICK is fairly resource intensive so AWS’s ‘free tier’ won’t cut it here
b) GCP has a fantastic ‘cloud shell’ – we can access a terminal of a VM in the browser. This is handy!
c) To get stuff like DitTICK up and running, GCP documentation is by far the most helpful
d) GCP runs on 100% renewable energy, so our flagrant disgregard for server efficiency won’t harm the planet

If you don’t like/can’t use GCP for some reason then any VM that runs docker-compose and has a configurable firewall will be just fine. You could even plausibly run DiTTICK in something like AWS ECS as a set of ‘tasks’. Kubernetes is also an option but let’s not get ahead of ourselves.

For starters, sign up to GCP, create a project called ‘DitTICK’ and head to the ‘Compute Engine’ tab. If you’ve just created the project, GCP needs a bit of a breather before it’ll let you spin VMs up:

Take your time, sweet prince

When it’s ready, hit ‘Create’:

Set up a VM in whatever region you like with at least 15GB of RAM (Ditto is hungry) and a Container Optimized OS:

I tried to run this on a VM with 4GB and 8GB of RAM and it wasn’t at all happy
Bash commands still work on the ‘Container Optimized OS’, don’t worry

A WORD OF CAUTION:

You can see in the screenshot above that, if this VM is left running, it’ll cost you an estimated $97.69 a month. That means, being pessimistic, it’ll probably cost you more that this! If you’ve got that kind of money to burn, more power to you. If you’d do literally anything else with that money, do remember to shut down this VM once you’re done with this tutorial! (I’ll remind you again later)

Leave everything else on default and check the boxes to allow HTTP and HTTPS traffic:

Once the VM is up and running, hit the ‘SSH’ button in the GCP console:

If your broswer doesn’t block it, a pop up will open that serves as a remote terminal to your VM:

Your username will be different of course

The ‘Container Optimized OS’ has docker installed by default, so check everything is working as expected with a quick docker ps – you should see that no containers are running.

Once you’re in, clone the DitTICK repository onto this machine, as before, with:

$ git clone https://github.com/arupiot/DitTICK.git

Then go into the docker directory:

$ cd ./DitTICK/docker

Unfortunately, docker-compose isn’t installed by default, so our start.sh script won’t work here. We still need to clone the ditto repo though, so:

$ git clone --branch 0.9.0 --single-branch --depth 1 https://github.com/eclipse/ditto.git ditto

We can then run our docker-compose stack with a clever workaround that’s outlined here:

$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD:$PWD" -w="$PWD" docker/compose:1.24.0 -f docker-compose.all.yaml up -d

NB: the command above should all be on one line!

You should see the usual feedback from docker-compose that all of the containers have started successfully. Now, back in the GCP Compute Engine window you can find the public IP of your server, the ‘External IP’:

Your external IP will be different!

Click it, and you should see a helpful homepage to navigate our TICK and Ditto services:

You’ll need to edit the URL in the address bar to http:// rather than https:// – the link on the Compute Engine page assumes we’re doing things securely… (we’re not)

You can access Grafana with the same credentials as before, admin:dittick.

Of course, there will be no log of the UDMIduino on Grafana because we’re now looking at an InfluxDB instance running in the cloud. So let’s get some data in there! All we need to do is point our dummy ‘gateway’ script to our new VM. Change the BROKER variable to the External IP and restart the script:

...

SERIAL_CONN = '/dev/ttyACM0'
SERIAL_BAUD = 9600
BROKER = "35.239.139.61" # <--- your 'External IP' goes here!
BROKER_PORT = 3389 # weird looking port, right?
ser = Serial(SERIAL_CONN, baudrate=SERIAL_BAUD, timeout=None)

# To become a generic mapping function
# mqtt2bacnet could feature BAC0?

def mqtt2serial(message, ser):
    ser.write(b't')

...

A quick note about the mysterious use of port 3389. The ‘standard’ port for MQTT without TLS certificates is 1883. The only reason I’ve used port 3389 is because this port is open by default on fresh GCP VMs. Security! Port 3389 is normally used for RDP. If you’ve been having trouble with any of the scripts above, this could be why.

Presuming everything is set up correctly, the ‘gateway’ script, with the UDMIduino still connected to it, should say:

$ sudo python3 serial_mon.py 
[sudo] password for guestsudo: 
Connecting to broker 35.239.139.61
In wait loop
In wait loop
connected OK
Subscribing to LED toggle topic... 
in Main Loop
Publishing...

And now, after around 10 seconds, we should see data feeding into our Grafana in the cloud:

And there we are! We have a vague mock up of a generic IoT device sitting right next to us and we can now monitor it from anywhere in the world.

Ditto is also running here, however much like our until-recently-fresh install of Grafana, the UDMIduino ‘Thing’ won’t exist on our server either. You can use the same Postman collection we used earlier to populate Ditto’s database with a Policy, Connection and Thing. Just change localhost in the request URLs to your External IP, e.g.

It will always pay to do a health-check at the start of the new day

You’ll also need to change the value of the piggybackCommand.connection.uri key in the ‘Create connection’ request body:

I’m no Ditto expert, so for reasons I’m not entirely clear about you must use the External IP of your VM for the connection string. If you try to use ‘localhost’ or ‘mosquitto’ hostnames Ditto will reply saying that they’re blacklisted. If you have any insight, let me know!

All being well, you should also be able to monitor and toggle the LED, as before, using the ‘Get things’ and ‘Test Toggle’ requests:

Monitor a Thing
Change a Thing

That means the API to the UDMIduino is now on the public internet! Hooray for convenience! Blood curdling screams for security concerns!

Because everything is on the public internet, however, it means we can change one line in our Big Red A Button web app and it will talk to our newly deployed Ditto instance:

...

@Injectable()
export class DittoServiceService {
  dittoURL: string = "http://35.239.139.61/api/2/"; // CHANGEME!
  ledToggleEndPoint: string = "things/open.iot%3AUDMIduino-000";
  toggleBody: any = {
    version: 1,
    timestamp: "0",
    points: {
      lux_level: {
        present_value: 0
      },
      lum_value: {
        present_value: 0
      },
      dimmer_value: {
        present_value: 0
      }
    }
  };

...

Restart the Angular app and Hit That Button. Eventually, the LED will toggle. You may notice that the LED will now take a good deal more time to change state, compared to when DitTICK was deployed on your local machine. Depending on where you deployed the VM on GCP, we’re now potentially talking across the world. These things take time!

The Big Red Button on AWS

For no reason* in particular, let’s deploy our slightly updated Angular app to AWS via S3. This is one of many ways to deploy a web site, so any other way you might know:

  • Via some generic hosting service (like GoDaddy/OVH/Ionos/whatever)
  • Running your own LAMP stack
  • Serving the site from an internal server
  • etc

Will all work just fine. The DitTICK instance we’ve just deployed to GCP is on the public internet. If you have a server in mind, make sure it has outbound access. I’m using S3 because it has the least set up I’ve found to deploy a simple static HTML/JS/CSS site. Angular is a tool used for developing web applications – what you get at the end is still HTML/JS/CSS.

Presuming you’ve changed the dittoURL above:

dittoURL: string = "http://35.239.139.61/api/2/"; // CHANGEME!

Make sure you’re in the root folder of the Angular app and run:

$ cd /your/path/to/udmiduino/web-ui
$ npm run build

Your output should be something like this:

$ npm run build

> big-red-button@0.0.1 build /home/guestsudo/Projects/DitTICK/udmiduino/web-ui
> ng build --prod

 10% building modules 7/10 modules 3 active ...-ui/node_modules/zone.js/dist/zone.j 
...[potentially more messages]
 27% building modules 148/149 modules 1 active ...core-js/modules/_to-absolute-inde 27% building modules 149/150 modules 1 active ...tTICK/udmiduino/web-ui/src/stylesDate: 2020-01-24T14:37:16.132Z                
Hash: 482bf2729909f9ee9807
Time: 4468ms
chunk {0} polyfills.f0eca7a3efead95b6d35.bundle.js (polyfills) 63.5 kB [initial] [rendered]
chunk {1} main.931146859e0d26884cce.bundle.js (main) 173 kB [initial] [rendered]
chunk {2} styles.abbabe9089be11cce627.bundle.css (styles) 27 bytes [initial] [rendered]
chunk {3} inline.318b50c57b4eba3d437b.bundle.js (inline) 796 bytes [entry] [rendered]

This will create a new directory, /dist, with files that look like this inside it:

$ ls -la
total 272
drwxr-xr-x 2 guestsudo guestsudo   4096 Jan 24 14:37 .
drwxr-xr-x 7 guestsudo guestsudo   4096 Jan 24 14:37 ..
-rw-r--r-- 1 guestsudo guestsudo   5636 Jan 24 14:37 3rdpartylicenses.txt
-rw-r--r-- 1 guestsudo guestsudo   5430 Jan 24 14:37 favicon.ico
-rw-r--r-- 1 guestsudo guestsudo    605 Jan 24 14:37 index.html
-rw-r--r-- 1 guestsudo guestsudo    796 Jan 24 14:37 inline.318b50c57b4eba3d437b.bundle.js
-rw-r--r-- 1 guestsudo guestsudo 172629 Jan 24 14:37 main.931146859e0d26884cce.bundle.js
-rw-r--r-- 1 guestsudo guestsudo  63550 Jan 24 14:37 polyfills.f0eca7a3efead95b6d35.bundle.js
-rw-r--r-- 1 guestsudo guestsudo     27 Jan 24 14:37 styles.abbabe9089be11cce627.bundle.css

The hashes before each .bundle, e.g. 318b50c57b4eba3d437b will be different for you.

If you’re unfamiliar with AWS and S3, there are now a few prerequisites:

  • You’ll need an AWS account. If you’re signing up today it won’t cost you a penny, the method we’re using for deployment here is, for all intents and purposes, free forever given how AWS billing works in early 2020
  • You’ll need to set up an S3 bucket for static hosting
  • I could cover how to set up S3 here but the link above will point to the current and recommended way to do it. Main caveats;
    • Remember to give public read access (just for the S3 bucket that’s hosting your site)
    • Remember to also add the right bucket policy like the one below:
{
   "Version":"2012-10-17",
   "Statement":[{
 	"Sid":"PublicReadForGetBucketObjects",
         "Effect":"Allow",
 	  "Principal": "*",
       "Action":["s3:GetObject"],
       "Resource":["arn:aws:s3:::example-bucket/*"
       ]
     }
   ]
 }

Once all that’s done, click and drag the contents of the /dist folder into your S3 bucket:

From left to right

Click through the prompts and everything should be okay. In the picture above, the name of my bucket is ‘urawizard-red-b-worldwide’ so my bucket policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadForGetBucketObjects",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::urawizard-red-b-worldwide/*"
        }
    ]
}

Note the different in the Resource value. The URL to my site is: http://urawizard-red-b-worldwide.s3-website-us-east-1.amazonaws.com/. Yours should look similar to that.

I’ll keep the link up for reference and you can press the button as many times as you like, it’s not connected to anything so don’t worry! In Chrome/Firefox/maybe Edge by now you can see it’s not connected to anything by opening the ‘network’ tab in the Inspector console and seeing a failed GET request every time the button is pressed, e.g.

Hopefully, by checking your own Big Red A Button deployment, you can use the same method to check if your Angular app is talking to DitTICK happily. And, by now, it should go without saying that every time you hit the Red Button, the LED on the UDMIduino should toggle.

*the reason is so I can tell people this is a ‘hybrid cloud’ tutorial

Going Remote

It’s taken a while to get here, I get that. But what we’ve just done is connected a Thing to the Internet through our dummy ‘gateway’. It should become apparent how useful (and extensible) this is if:

  • You note down the URLs of
    • your GCP deployed Grafana instance
    • your Big Red A Button app in S3
  • Walk outside
  • Put the links into your smartphone’s web browse
  • Toggle the LED with the web app
  • Watch in change on Grafana
  • Share the links with someone on the other side of the room / office / town / city / hamlet / world
  • Get them to do the same thing

This system works anywhere! And you’ve done it all using Open Source components and very well thought out standards like HTTP and MQTT. Good show!

Cleaning Up

If you are so inclined to, PLEASE REMEMBER TO TURN OFF THE VIRTUAL MACHINE ON GOOGLE CLOUD PLATFORM – IT MIGHT COST YOU $100 PER CALENDAR MONTH

If you’ve made it this far…

This is nothing more than a whistle stop tour of one approach to doing this sort of thing. There are a lot on concepts to get one’s head around so, especially if you’re inexperienced in this area and you got to the end with everything working, well done!

If any of the code doesn’t work or any part of this tutorial is confusing, feel free to post an issue on the relevant GitHub repository or leave a comment below.

Below are some brief pointers on what would need to be done to make this system ready for a ‘real world’ scenario.

Security

In the real world, we don’t serve REST resources, endpoints or web applications over HTTP. Unless we have a very, very solid reason, we use HTTPS. We try and use a better method than Basic Authentication.

We don’t open an MQTT socket with no security whatsoever. We use TLS certificates or secure passwords or we use both. Certificates need to be rotated from time to time. When dealing with thousands of devices, this should be automated.

Everything deployed in this tutorial is either entirely unsecured or uses default credentials. The credentials used in this tutorial are all publicly accessible via the GitHub repos we’ve been using. Do not reuse them!

If you’re intent on generating your own TLS certficates using openssl, see here for a proposed method:

https://github.com/arupiot/self_signed_certs

If you’re intent on hosting your own web servers, see here for a simple development environment for a simple static site with free SSL certificates forever using Let’s Encrypt:

https://github.com/arupiot/auto_ssl_nginx

Stability and Data Provenance

In the real world we do not store databases in Docker volumes. If this were a real system, there should at least be regular backups of the database server. Ideally, if you want to use InfluxDB, use a managed database service. If the service provides rollbacks of your data, jump on it!

The docker-compose files we’ve discussed all describe a number of containerised services. These services could be deployed to different servers or some sort of managed ‘auto-scaling’ system like AWS ECS or K8S. There will always be an overhead of time and cost to operate a service like this!

Robustness

Imagine you want to put a temperature sensor in your garage and send the data to something like DitTICK. It is manageable to use Postman to add your temperature sensor to Ditto and extend the dummy ‘gateway’ code to forward your data from the sensor to Influx.

Now imagine you have 4000 new temperature sensors that are about to be installed in delivery trucks that drive for 12 hours a day. Each temperature sensor will be powered down when the engine is off. The failure rate of your sensors is 40 per year.

Ditto can act as your device and connection registry, no problem. But you’ll need a robust process to add every new sensor to Ditto and, if something goes wrong, replace or update device in the registry.

While the truck is driving, it goes into an underground tunnel and stops pushing data to the cloud. What happens then?

These things need consideration!

Realism

Real ‘IoT’ devices are a lot more complex than the UDMIduino. They will have their own quirks. They may have their own management interfaces and systems – they will certainly have their own security. These will need to be integrated carefully with DitTICK. Real IoT gateways are even more complex.

Conclusion

All in all, it sounds like there’s plenty of scope to extend the functionality of DitTICK, fork it at will!

Tales from LushRooms (coming soon…)

Team LRPI

In the absence of any other account of how LushRooms came to be, and if you’ll allow me, I’ll go through how we managed to design, build, test, deploy and monitor an open source, highly scalable microcomputer infrastructure for Lush Spa using only open source tools.

More to follow. But for now, find (most of) the code here:
Main repo with CAD files
Player server
Player control web application