Posts on SyxAxis

How-To : Build a Docker Host Using Ubuntu Server

Mon, 04 Jan 2021 07:47:22 +0000

The easiest way to get an installation of Docker up and running is to run it on an virtualiased Ubuntu Server build in VirtualBox

VirtualBox

For this demo I will be using Oracle's VirtualBox system to host the Ubuntu server as a Docker host. It's free and very easy to use.

Simply install the latest version of Ubuntu server with bog standard defaults, but be sure to install two network interfaces into the VirtualBox host.

Interface 1 - NAT routable to the outside world - So you can download updates and packages plus Docker images.
Interface 2 - Host-Only - While not mandatory it can be very useful for internal testing to have an interface back to your desktop/laptop box. Class C usually something like 192.168.56.0 network.

Ubuntu Server

Installing the Ubuntu Server just install with defaults but you may wish to hardcode the host-only Class C address to 192.168.56.111 for example.

Use /etc/hosts has the hostname matched to the host-only IP addresss
Example netplan config YAML file is shown below, note the fixed IP address.

 1# This is the network config written by 'subiquity'
 2network:
 3  ethernets:
 4    enp0s3:
 5      dhcp4: true
 6    enp0s8:
 7        addresses:
 8        - 192.168.56.111/24
 9        nameservers: {}
10  version: 2

Ensure you do a full update from the source repos and reboot.

1apt-get update -y
2apt-get upgrade -y
3apt autoremove -y
4reboot

Install the Docker CE from the official Docker Repo

First we remove the Docker that might already be installed, update the local packages and then install the pre-requisite packages we'll need.

1sudo apt-get remove docker docker-engine docker.io
2sudo apt-get update
3sudo apt-get install apt-transport-https ca-certificates curl software-properties-common

Next we get the GPG key used to verify te Docker repo package can be trusted.

1curl -fsSLk https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Next we need to register the Docker repo locally so we can get the packages down, there's two ways to do it. Either do it with a fixed reference but note this is version specific, I would only do this if you have to.

1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

The preferred option is to let the O/S find the right version.

1sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

If you find that you get certificate errors like "Certificate verification failed: The certificate is NOT trusted.", especially if you're behind a proxy, then extract the proxy certificates ( do this using a browser like Firefox or Chrome ) and save them to "/usr/local/share/ca-certificates" then run "sudo update-ca-certificates".

Now we're ready to install Docker from the official repo. This is a good time to add your user to the docker group so you can use docker commands without having to keep using sudo. You use sudo for all your root commands right, you'd never login with root?!

1sudo apt-get update
2sudo apt-get install docker-ce
3sudo groupadd docker
4sudo usermod -aG docker $USER

Assuming it all went OK, just check the service came up:

1sudo systemctl status docker

Finally test Docker is working, log out and log back in to make sure the privs take and then issue this:

1docker run --rm hello-world

Using Docker with a Proxy to get Images

If you're behind a proxy then do the following to get docker to work with a proxy.

1sudo mkdir /etc/systemd/system/docker.service.d
2sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

Add this to the open file:

1[Service]
2Environment="HTTP_PROXY=http://172.26.2.10:9480"
3Environment="HTTPS_PROXY=http://172.26.2.10:9480"

Then restart Docker.

1sudo systemctl daemon-reload
2sudo sysyemctl stop docker
3sudo sysyemctl start docker

Powershell - Find All File and Folder Names with Non-ASCII Characters

Sun, 03 Jan 2021 13:18:28 +0000

Despite us being a UTF-8 world file and folder names with non-latin chars can still trip processes up.

I was digging through my MP3 collection, around 420,000 tracks ripped from a spareroom that's just filled to the brim with CDs, and I found that auto-naming had set the file and folder names with some odd characters that made it hard to rename and locate files. Names like Händel are fine, no issue with those as I expect those but when you get something like this [ \The Mission\Godʼs Own Medicine ] it doesn't look wrong but note the apostrophe is not a standard ASCII one but one from another part of the wonderful UTF-8 selection.

So how to trace them down quickly, Powershell is always my go-to scripting language on Windows, it is simply the Swiss Army Knife of Windows admin work tools, or Schweizermesser as they say in German speaking countries.

Scan up the folder structure and pull out the names that have chars that fall beyond the standard simple, original ASCII set.

Long version for scripting:

1Get-ChildItem -Path "M:\GWJ\MUSIC\" -Recurse | ForEach-Object {
2    $currFile = $_
3
4    if ( $currFile.Name -cmatch "[^\u0000-\u007F]" ) {
5        $currFile.
6        Write-Host "FOUND:[$($currFile.FullName)]"
7    }
8}

This can be shortened to this if you need a single command verison:

1Get-ChildItem -Path "M:\GWJ\MUSIC\" -Recurse | Where-Object { $_.Name -cmatch "[^\u0000-\u007F]" }

Alternative:

1Get-ChildItem -Path "M:\GWJ\MUSIC\" -Recurse | Where-Object { $_.Name -cmatch "[^\x20-\x7F]" }

Go - EXIF Scanner and Extractor

Sat, 02 Jan 2021 16:05:26 +0000

Using Go to extract EXIF data from Image files

Extracting EXIF from a single file is easy but suppose you have a huge folder tree with hundreds or thousands of images, what then? Then suppose you have a publishing deadline for a book. What you need is a super fast, efficient language to do the job for you. Golang is what you need!

Have you ever wondered how when you take photos using your mobilephone and then upload them to your favourite website, how does it know so much about the phote image file? The secret is the EXIF data stored within the JPG images you capture and then upload. The EXIF data contains the most critical info needed about an image, the camera, the lens, the focal length, aperture size, time and date of the shot and even the GPS info. This is the most basic info, on top of that there are extensions for the IPTC info which has tons more info like keyword trees, descriptions, copyright, all valuable stuff you store and maintain using software like Adobe Lightroom.

When I worked on my first photo book for FotoVue I'd shot and edited around 3,000 images files, from those we selected around 1,000 images to go into the book. The books FotoVue produce are for photographers and that means you need to have the key informaiton like location, camera, lens, focal length, aperture, ISO, all that good stuff and one place that's stored is in the EXIF of every image. When you edit all the info is kept and stored so it can be used to correctly catalogue your images.

Getting the EXIF from a single image is dead easy, right-click on it select Propeties ( Windows ) or "Get Info" ( Mac ), there it is. How the heck do you do that for 1,000+ images that are scattered through sub-folders? The London book I worked on had 12 chapters with 14 locations per chapter, that's around 200+ sub-folders. We needed the key critical EXIF data into something we could pass to the page designers, so we needed the filename plus the EXIF data, they would then copy and paste it into the 1,000 images in the book's layout.

We thought Lightroom could be used to get the info but despite it begin one of the best photo image organisers available it's useless at allowing you access to it's internal database, it demands you jump through some severley archais hoops using LUA, I've no wish to learn a dead language from scratch just to get a CSV.

Enter the Gopher!

Luckily Go is a best damned language on the planet and I needed a simple Go project to get me started as I was learning Go.

Go-Exif Libary

DSOprea go-exif library

We need a library that can extract EXIF data from any JPG file and hand it over in a simple text based format.

The examples supplied with the library are superb and ready to be built, so I stripped out the requied code and then built myself a file-folder scanner using the filewalk function from the standard library. From that I was able to put together a suitable struct, read the info from file in a tree and dump the whole damned thing back out to a CSV file that could be loaded into any suitable spreadsheet app.

You can find the code for my EXIF scanner extractor here: GOExifExtractor on Github

The GOExifExtractor Utility

The reason for choosing Go was that I use Windows and Linux but others may use Mac, most togs will use Macs to manage photos and while it's a command line only util I needed a way to use it on any of the major platforms, Go can cross compile, so that's sorted.

The utility allows you to choose the fields you like to extract or simply all of them, filling in blanks where it can. The most common EXIF fields are shown below:

 1constEXIFFields = []string{
 2		"ImageDescription",
 3		"Make",
 4		"Model",
 5		"Software",
 6		"DateTime",
 7		"Artist",
 8		"Copyright",
 9		"ExposureTime",
10		"FNumber",
11		"ISOSpeedRatings",
12		"DateTimeOriginal",
13		"DateTimeDigitized",
14		"FocalLength",
15		"CameraOwnerName",
16		"BodySerialNumber",
17		"LensModel",
18		"GPSLatitudeRef",
19		"GPSLatitude",
20		"GPSLongitudeRef",
21		"GPSLongitude",
22	}

The folder scanned it really simple, nothing more than a simple JPG scanner. ( yeah hardcoded JPG only, it was my first Go project! )

1err := filepath.Walk(filepathArg, func(path string, info os.FileInfo, err error) error {
2		if filepath.Ext(path) == ".jpg" || filepath.Ext(path) == ".JPG" {
3			files = append(files, path)
4		}
5		return nil
6	})

One thing that I did have problems with was how the raw EXIF data is stored from some attributes, the two problematic ones were Exposuretime and Aperture setting sometimes called the F-number, often expressed as f/16 for example.

Exposure

Exposure is stored in EXIF as an expression of a fraction of a second. Sub-second that is something like 1/128, which is 1/128th of a second. That's easy enough. However anything longer than a second, long exposure images can be minutes, is stored as fraction 60/1, which is obviously 60 seconds. So that needed to be split and converted to decimal then have "sec" appended. Simple enough with a couple of floats.

Aperature

Aperature is expressed as a ratio of the the focal length and width of the aperture, so f/8 or f/16. There is no actual value on a lens of f/8 or f/16, they are simply ratios of the relationship between the settings of where the lens has been set and the aperture. Once again, we take the value from the EXIF and convert it from fractional to decimal then append "f/" to the result so that it makes sense when shown in the CSV.

Go-Exif Library

The core function of the Go-Exif library is to be presented with an image file, it then extracts all the EXIF data a JSON based struct. We then cycle the info brought back from the image file in the struct and pull the EXIF field we actually asked for, converting those mentioned where required for the CSV output. The real magic is that the Go-Exif library is able to extract the data into a really easy to use data format where the fields are just sets of tagged data.

Is it Worth The Effort?

All I can say is that when FotoVue books are put together gathering the EXIF data and extracting it can take one person around a week to get all the info if they spend every day copy and pasting the info from Lightroom into Excel! That's a week of click and image, copy 8-10 pieces of text per image from Lightroom into row in Excel, then repeat around 1,000 times!! No fecking way! I didn't spend 30 years of my life in IT for nothing!

I use Lightroom along with Jeffrey Friedl's superb "Collection Publisher" plugin for Lightroom to extract and dump all the images in the correct folder structure on disk to match the collections from within Lightroom. Then I run my scanner over it and I can have a 1,000 JPG file's worth of data into a CSV in less than 2 mins! 2 mins versus 60 hours of manual copy'n'paste, f**k yeah it's worth it!

If the EXIF data is wrong or not quite right, then I don't have to spend hours and hours fixing a spreadsheet, I simply fix the odd entries, re-dump the images I need and then re-run the util, all of which can be done in a few minutes. This means the data can be off to the book editor and layout artists same day, it's easier to fix if there's problems so the turnaround is infinitely faster and it means me, the editor and layout artist have much more time to do the creative stuff we enjoy. All thanks to Go and a will to automate.

Links:

Flashing Arduino Nano Bootloader from Arduino Uno

Sat, 02 Jan 2021 15:00:20 +0000

Arduino Bootloader - How-To (Re)Install the Aruduino Nano Bootloader

How to flash the Aruduino Nano bootloader and what to do if it doesn't work.

I recently obtained an Arduino Nano unit from my Dad ( he's 82 and loves playing with gadgets and electronics! ) and decided to have a play with it but I found it wouldn't take a sketch, it refused to accept the upload of sketch code. Turns out cheaper non-genunine Arduino boards are sometimes shipped without bootloaders. Sounds like a serious issue but not at all, the designers of the Arduino system thought about this.

What is the Arduino bootloader?

The Arduino bootloader is a small piece of code, as the name suggests, that allows the compiled sketch code to be installed and started. It's 512 bytes in size and sits in an area of "memory" within the 328P microprocessor chip.

Believe it or not, there's actually not a lot to an Arduino board and you can simply buy an ATmega328P microprocessor chip and flush a bootloader to it if you wish! The Arduino system is mainly about the supporting boards that hold the processor chips, the processor chips on the boards are, well, cheap as chips!

A special piece of code that is executed at every reset of the microcontroller and that looks for a sketch to be uploaded from the serial/USB port using a specific protocol and speed. If no connection is detected, the execution is passed to the code of your sketch. This little (usually 512 bytes) piece of code is called the "Bootloader" and it is in an area of the memory of the microcontroller - at the end of the address space - that can't be reprogrammed as a regular sketch and had been designed for such purpose.

Use an Arduino Uno to ReProgram An Arduino Nano

So you'll need the following:

Working Arduino Uno board
Arduino Nano board
Jumper wires with pins
10uF Capacitor ( must be 10uF )
Breadboard if you have one

First you need to do is to flash a specific sketch, supplied from within the standard Arduino IDE, to the Arduino Uno.

Plugin the Arduino Uno and from the Examples menu locate the "ArduinoISP" sketch, upload it to the Arduino Uno.

Disconnect the Uno from the USB and wire it up as shown below. Note the use of the capacitor slotted into the line to GND.

Most boot flashing tutorials and diagrams will simply show the GND being wired up between the two board but note that if you attempt to flash the bootloader with just the lines you may find it will fail. The line is hot and when you go to flash the target board needs its reset line hit and dropped, if the line is simply hot the reset not happen and the reprogram will fail. The capacitor is put in line to drop the voltage and pull the line back up when the flash is started this triggers the Uno to Nano reset and the Nano will be open to accept the boot flash. It needs to be 10uF anthing bigger or smaller will take too long to pull the reset.

Once you have this wired up as shown the next step is to flip the Arduino IDE into a programmer using the Uno as the hardware programmer passing through to the Nano.

Plug the USB into the Arduino UNO board NOT the Nano, the Uno is a programmer and will instruct the Nano.

You need the board set to the target, the Nano.
You need the chip set to ATMega328P. ( pre 2018 this would be "old bootloader" )
Port is set to the Uno's USB link.
Select "Arduino as ISP" from the Programmer options.

Finally hit Burn Bootloader.

What Happens if it Doesn't Work?

One issue I had with my Nano board was that it would not take the bootloader when I first tried and I kept getting failures. It took me 2 days to find the proble,.

If you see this:

1avrdude: stk500_recv(): programmer is not responding
2avrdude: stk500_getsync() attempt 1 of 10: not in sync: resp=0x03

You may see this:

1avrdude: Yikes!  Invalid device signature.
2         Double check connections and try again, or use -F to override
3         this check.
4
5Error while burning bootloader.

I was also seeing random signatures from the Nano board!

1Reading | ################################################## | 100% 0.05s
2
3     avrdude: Device signature = 0x000000
4     avrdude: Yikes!  Invalid device signature.
5     Double check connections and try again, or use -F to override
6     this check.

All these errors mean one thing, you have bad wiring! The lines between the boards must be good and you will need to ensure the cap in the GND line is good.

It turned out in my case that the VCC pin on my Nano board had a bad solder joint and it was joined to the pin next door to it. This meant the second it was flashed the VCC bled over and reset the board and kept producing random signatures. 5 secs with the soldering iron to clean up the bad solder joint and the flash worked perfect.

Moral of the Story

Genuine Arduino boards will have a higher quality check from the factory but clones may not be subject to quite such high standards. You can get 5 Nano boards for £10 from Amazon, well worth buying the clone boards as they're far cheaper than the genuine boards, but just be sure that your ready to check them over and you're ready to dig deeper into the Arduino guts when things don't work out of the box.