hp-mark Exists Now

15-2-2021

I climbed down a rabbit hole recently. Nearly half a year working on hp-mark. The past 84 days even skipping blog post and newsletter writing.

Progress has been steady. It works now. hp-mark measures positions, like:

First ever non-zero result from hp-mark — Result 1: [-1.27, -217.65, 2.68]. Hand measured position is [0, -217, 0].

Accuracy is not fantastic. The image above is a lucky one, with only 3 mm of error. A more commonly seen error is around 35 mm.

A lot of hp-mark work remains. I ticked 2 boxes, and added 6 new ones in the roadmap today:

Aquire camera 6D pose (this includes defining our world coordinate system)
Aquire effector 6D pose
Detect all markers on 95% of training images
Take image ourselves upon request, don't rely on other programs to take image first
Create a continous stream of position measurements (video?)
Get a statistical idea about size of error
Respond to RepRapFirmware/Duet request for position measurement
Integrate a second camera, to reduce error

I'm a bit overwhelmed still. I need to look backwards a bit.

What Have I Done?

I've been surprised.

Video stitched together from todays test images.

I chose C++, OpenCV, and a very traditional approach for this project. I just wanted the most basic code that would mecanically find circles among pixels, and then apply static geometric equations to transform circles into a nozzle position. No AI stuff, no GPU stuff, no fancy hardware control, just $$ f(x) \rightarrow y, $$ where $x$ is an image, and $y$ is a nozzle position.

Such an old and explored problem! I felt confident.

Some things that took long for me to program, in descending order:

Good enough ellipse detection. OpenCV's SimpleBlobDetector was ok, but ultimately not robust enough for our purposes. I was lucky to find ED_Lib, and spent a little over a month integrating it into hp-mark. For example, I had to fight a little to get sub-pixel accuracy in the center and size data. More about that here.
Geometry. A photon bounces off a sphere, through a pinhole, and onto an image sensor. Many more follow. They create a pattern, a projection, on the image sensor. Given the position of the sphere, what's the equation of its projection? Or vice versa; given a projection, what's the position of the sphere? Which projected point correspond to the sphere's center? I derived all relevant equations by hand and wrote tests that wouldn't go green before I was exactly right. To bootstrap the chain of testable geometric functions, I derived the most basic equation in two different ways, and tested that the two derived functions gave exactly the same results. The system would never be good if the theory it rested upon wasn't sound, so I spent the time to make this perfect.
Filtering among ellipses. The improved ellipse detector found lots of ellipses. Getting hp-mark to determine which ellipses represent real markers, was hard. To be real useful, hp-mark must be able to identify all its markers almost all the time, so I spent extra time here as well. I will return to this point in the future, but we're at least finding all the markers ca 80% of the time now.
Color recognition. The human brain is full of heuristics that helps us determine color, in a wide variety of situations. Very fine details of camera settings, light/shadow, and way of definition, throws a simplistic computer program off very easily. The strongest heuristic I was able to program, in which all colors are relative, still can not identify a non-marker in a set of six real and one false marker. Still, I was able to keep color as a marker categorizer that lets us avoid having to put QR-codes on our markers. I fought a bit for the aesthetics.

I feel like I'm almost too tired of it to even write about this stuff now. But before I climb out of this rabbit hole, I can show you something about how the ellipse detector works internally:

Input to ellipse detector — A zoomed in version of the original image from the PiCam v2.

edge image inside ellipse detector — Color contrast between pixels were used to find these edges in the image.

segment image inside ellipse detector — The edge pixels are the grouped into segments. Each segment has a different color in this visualization. A single segment that forms a closed circle or ellipse will be detected already at this stage. Most detectors can find complete closed-curve ellipses quickly and easily. What sets ED_Lib apart is how hard it tries to piece together the chopped-up circles, and how quick and cheap it manages to do this.

lines image inside ellipse detector — The segments are split up further into lines and arcs. These are the lines.

arcs image inside ellipse detector — ... and these are the arcs. (If you compare very closely with the segments image, you find that not all arcs match exactly. They should match exactly. I mixed up images from two different runs when preparing this, so this arc visualization is ever so slightly off. It's still representative for the result below though, in that the top blue marker could not be found amog these arcs either, but all other markers were found.)

output of ellipse detector — The arcs are combined into circles and ellipses according to many different heuristics.

In the image above, we see that one marker was missed, despise ED_Lib's many different heuristics. I tried to tune the heuristics for the hp-mark use case. See how that went, here.

Ultimately failing to tune the heuristics convinced me to put background plates below my markers. That is what I'm currently working on. They will probably end up looking something like this:

marker with bent octagonal background — Test of marker background with bent octagonal background.

The final background design will probably be flat and circular, and hp-mark will use the edge of the background, as well as the marker itself, to determine its position, further improving both robustness and accuracy of the system.

But not today. I'm so tired of that kind of programming. Let's celebrate the restuls at the top of this post for a while first. And let's rest.

- tobben

Links

Hangprinter Campaign: Bountysource Salt

Hangprinter Merchandise USA: Spreadshirt.com

Hangprinter Merchandise Sweden: Spreadshirt.se

Hangprinter Project: [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], [68], [69], [70]

Hangprinter Project Homepage: hangprinter.org

Print Issue Solution Filter Project: [1], [2], [3], [4]

Sourcing RepRappro Mendel in Larvik: [1], [2], [3], [4], [5], [6], [7]

Archive: 2014, 2015, 2016, 2017, 2018, 2020, 2021

Github Profile: link

Gitlab Profile: link

Hangprinter project on Gitlab: link

Vimeo User: link

Youtube User: link

Twitter User: link

Master's Thesis: link

Linkedin Profile: link

Appropedia User: link

RepRap Forums User: link

Forums threads: Hangprinter version 1, Hangprinter version 2, Hangprinter version 3, List of Hangprinter threads

Source for this blog: Gitlab repo

Everything on this homepage, except those videos who are published via Vimeo or Youtube, is licenced under the Gnu Free Documentation Licence. The videos published via Vimeo or Youtube are also licenced via Vimeo or Youtube.