The Truth about AI models and tools in 2025

 

 

 Having just finished an intensive course in AI usage, I think it's time for some reality to come into the conversation on this topic and to drop a lot of the hyperbole that seems to constantly swirl around this subject.

Some Americans are prone to hyperbole at the best of times, you only have to listen to the drivel that comes out of the mouths of some of their CEO's to become frustrated. Elon Musk has been promising AI driven cars for almost a decade now - with zero delivery or expectation that it will be in the near future. Sam Altman talks everything his company does in such over ambitions terms that it is now almost funny when a small Chinese AI model blows his multi million dollar one into the weeds, with some extenuating conditions.

Now, to be up front, I use AI daily for a whole host of uses and for 85% of the time it saves time and energy and is accurate, but for the other 15% of time it produces programming code that is flawed and requires considerable amendment or an elongated conversation with the AI to get to the place you want to be. Also some of AI's halucinations(AI incorrect nonsense) can be not only wrong but quite dangerously wrong - I experienced this with a Linux server PAM security file - ChatGPT got the config completley wrong. What it proposed looked right, but further investigation showed if I had used it as is - it would have rendered the server unaccessable. Just imagine a junior without 30 years experience just implementing that - no time saved - but a massive amount of work just created to fix it.

This brings me to the nonsense I see in the media about what we have today with AI and where this will lead. I think most of the predictions are either grossly pessimistic or optimistic and lead people to come to all the wrong conclusions.

There is no doubt these tools will make a huge difference to an awful lot of different jobs, but today we are right at the start of this process for 99% of the population who struggle with differentiating between a browser and an application. The main operating systems are working to introduce the new technology in a user friendly way and this is having some success but is still not completely normal for most people. The PC hardware companies are also ramping up the spec's and ability of everyday machines to be able to run simple AI models - but they are mostly on their way and not delivered yet.

I think for anyone wanting to dip their toe into the waters, the best and most privacy aware internet web tool currently is one offered by DuckDuckGo called Duck.ai - which allows you to keep your task private and offers a selection of models to utilise. This really is a good place to start.

You can also go completely remote using a tool like Ollama, which allows you to load the models locally and keep all your conversations on your local machine - if it has the power and Memory to do this.

I think my conclusion is simply this, don't believe the hype, just get stuck into the new technology, like you would any other tool and see what it can and can't do for you. Be sceptical of the CEO's claiming they have just invited a model that achieves the singularity - and treat the people saying we have just invented the end of human kind with the same distain. Work will change, jobs will change and it will be more severe to some than others but with new technology that has always been the case. I'm sure there are still many weavers miffed with the spinning jenny 🤣

And no, I didn't get a AI to write this, but I probably could have.

 

 

 

 

Installing and Using ChromeOS Flex or FydeOS for a home media client on older hardware.

 

 

I had a few older mini PC's lying around in the office that I wasn't doing anything with, so decided to to try out this operating system from Google which has been open sourced for installing on older hardware. GoogleOS Flex

I have used Chromebooks in the past and for simple use cases they are very handy, especially if you spend most of time in a browser. It has some limitations depending on the hardware you have, like the unit I picked didn't support the Linux Virtual machine  sub-system as it only had an Intel J4105 Celeron processor - which is not exactly cutting edge.

It does however run the Plex Web app. very smoothly and at this unit was to go under the TV for streaming - it was fine.

The installation process is straight forward as you basically burn the downloaded ISO file from Google onto a USB stick using something like Balena Echer or the Linux dd CLI command following these instructions.

Once done, boot using the USB drive, and. select install. You will need a Google account, but you can create one on the fly if you don't have one.

Once installed and up and running my little mini performed wonderfully and was more than capable of running Netflix, Plex client, Disney etc. 

If you have a powerful enough machine you can install the Linux sub-system and install a whole host of Linux apps on the machine as well, which makes it a far more powerful machine.

Here are some links to YouTube videos on installing and running ChromeOS Flex and also another very interesting version called FydeOS which offers Android supported applications as well.

FydeOS - based on ChromiumOS Installation

ChromeOS Flex Installation

 

 

 

How to simply build and maintain your own VPN service for complete privacy

One of the best ways of staying safe on the web is to use a VPN(Virtual Private Network), which keeps all your traffic across the internet encrypted, and more importantly away from prying eyes.

Now the easiest way for most people to do this is via a paid for service, and for 95% of the population that's perfectly fine. However, the fact you are using someones else's service for computer professionals leaves a worry that logs(although they claim they don't keep them) could be kept on where you visited and when. This is unacceptable to me.

So I have built my own service using these basic tools.

1. Amazon EC2 service
2. Wireguard kernel VPN service docker container
   
3. Ubuntu 22.04(soon to be 24.04) operating system built on an Amazon AMI virtual instance
4. Terraform to build the whole service from scratch
5. Ansible to configure and constantly keep state of the virtual instance.
6. Docker and Docker Compose 
   

Now there is a lot of programming code that is tucked away in my private gitlab instance for this, but I will describe how the process works and how the system gets built from scratch.

Basically the Terraform code builds the infrastructure and Ubuntu Virtual Machine(VM) that the VPN will work on. Once this code has run, it will have created everything - including a static IPv4 and IPv6 address that the VPN can use.

Once the VM is up and running, and I can successfully log onto the machine with SSH, I then run Ansible code to build all the required parts of the system, and upload the docker compose code ready for building the docker container that will be the VPN server endpoint.

Once that has all been completed and the docker container is up and running, I test that the new container is accessible from my local laptop using netcat - via the AWS security groups that were built with Terraform.

The initialisation of the Wireguard docker container creates several user configuration files that are required by any client application that wishes to connect to this VPN. 

With the Wireguard client installed on any laptop or desktop, you can now connect to the VPN anywhere in the world that you created the VM in. This allows for great flexibility and availability.

Also as Docker containers are so efficient and small in size, this allows the VM to be used for many other task's using other docker containers, like OpenVpn, Transmission, Gitlab etc etc.

I have been using this method for many years now and it is rock solid and extremely reliable. 

This supports a small team of users, but could be easily scaled to support many more users, and multiple instances would provide resilience.

Another major advantage is this can all be run on an AWS micro instance, so is inexpensive, though slightly more expensive than just using a paid for service, you have 100% control and know that all logs can be deleted on a daily basis and the VM can be destroyed and re-built in minutes in another region, availability zone or any other reason. 

With the way the Internet is going and every company wishing to snoop on your work and activities, then the more you can keep things private, the better.

Things discovered while installing pi-hole service as a docker container on a Raspberry Pi 8GB Pi4 (Raspbian Buster OS)

Things discovered while installing pi-hole service as a docker container on a Raspberry Pi 8GB Pi4  (Raspbian Buster OS)

Using the docker compose file from the GitHub installation site it was obvious that this version of the OS was not compatible with the latest versions of Pihole - so the first task was to up grade the OS to Bullseye.

The pihole docker container install site is here.

https://github.com/pi-hole/docker-pi-hole

This website offers a great simple method. It’s not as seamless as Ubuntu - but also not rocket science either.

https://www.tomshardware.com/how-to/upgrade-raspberry-pi-os-to-bullseye-from-buster

The first problem was there was a service already sitting on port 53 - so this has to be disabled as Pihole uses this port for it's service. The eventual truth was there were actually two services fighting over this port - systemd-resolver and connman dnsproxy.

They both need separate solutions to stop them.  

Systemd-resolver needs to have it’s config file altered as follows,
Change DNS=1.1.1.1 and DNSStubListener=no - then restart the service.

With connman you need to follow the instructions in this blog post to turn off it’s attempt to grab that port



https://wiki.archlinux.org/title/ConnMan#Avoiding_conflicts_with_local_DNS_server

Once this was completed - the docker container would still not start - it was giving a spurious error message about IPv6 issues.

On further reading around - it appeared that the version of docker requires to be version 20 or above, so I updated that in Bullseye as follows using:

sudo apt install docker.io/bullseye

After all this the docker container started correctly and the Raspberry Pi is now function as our DNS proxy for the whole house network to try and stop all the crap the web throws at you.

Switching on DNS over HTTPS on various browsers

A new feature that is hitting all new modern web browsers is the ability to turn on DNS over HTTPS, which in my opinion is a very good idea - to keep the ISP snoopers off your traffic - so they have no idea where you are looking up or searching for. Not all Browsers have this facility at the moment - but I will cover those that do.

This by association is a recommendation for those that do.

Mozilla Firefox:

To turn this feature on in Firefox go to Preferences/Network/Settings and the just select the DNS over HTTPS as shown below.

Google Chrome Chromium:

To turn this feature on in Chrome or Chromium, open a new tab and type chrome://flags and in the search bar type dns and enter. The following will appear and just select enable for DNS over HTTPS

Opera:

As this browser is based on Chromium you have to type opera://flags and then follow the above.

Microsoft Edge:

Again, this browser is based on Chromium, only this time just type edge://flags and then follow the above.

Apple Safari and Microsoft Internet Explorer:

I'm afraid at the time of writing the above two don't support it. Safari may well in the future, but I very much doubt Internet Explorer ever will.

Switching to a small footprint Intel NUC computer to do everything - unexpectedly surprised at performance

I don't know about you, but I have a variety of computer systems in my house for a whole range of uses. Everything from a Mac mini to an Amazon Firestick, all doing their job for the task required. However the Mac minis I have serve as media servers and players but are getting a bit long in the tooth, so I decided to upgrade the media server with an Intel NUC.

I decided to spec it as fully as possible and gave it a 6 Core processor, 32 GB RAM, and a 1TB m.2 NVME SSD drive. This should make it future proof for a good few years, that was the thinking. It came in at around £700, which was way less than a new Mac Mini.

Its main purpose in life was to support 4 USB 3.1 Gen2 external hard drive boxes (Akitio) for all my media and backups. If it was capable of anything else that would be a bonus, but not expected. It is attached to a 32" Samsung curved monitor with 144Hz refresh at 2.5+K - crystal clear and super responsive.

I installed Xubuntu as the base operating system, though I only use openbox window manager on it to reduce the overhead of the host operating system even further. That has been a great learning curve to show how little GUI you actually need just to get stuff done.

However, the real surprise happened when I spun everything up, was just how fast this little box is. The memory wasn't being used by the media software (Plex) so I thought let's try running a few docker containers on here as well to do other network jobs for me. No issue at all. I currently have the following running on the box 24/7

1. Pi-hole DNS service
2. Portainer container manager service
3. Jenkins job management service
4. OpenVPN service

So, I thought, I wonder how it would perform if I stuck a few virtual machines on there as well whilst it's doing everything else.

No problem at all, I'm currently running a beta version of Ubuntu and a separate Arch Linux using KVM and QEMU, and it still is not scratching the sides of what this little box can do. It's currently using 8GB of RAM and the load on the server is never above 3, even while everything is running and I'm streaming HD content to other parts of the house.

Considering I used to have tower machines cluttering up my workspace to do this sort of thing, I now have one device to do it all.

Intel have just brought out a new edition of these, with even faster processors and RAM capacity, so I will be getting one of those to work alongside this one when my other Mac Mini dies.

The Mac mini's served me well, but I have now found a better device and with all the cost savings of not buying Apple kit again, I can literally have three for the price of one.

30/01/2022 Update

I have added several more docker containers to the machine to see just how far this can be pushed 

1. Gitlab for source code control
2. Plex for home media and music playback

As before with all these services running and pushing the device with streaming and carrying out all it's other functions it is still performing perfectly.  The replacement of the installed plex server to use a docker container is simply one of ease of maintenance. There is nothing wrong with the installed app version, but keeping plex up to date now is just a matter of a "Docker pull" and I've got the latest version - no Library issues etc.

Building a small footprint Ubuntu desktop or server for old,singleboard or virtual machines.

So the lockdown offers time to try out things I have put on the back burner for a while. This little project was to build as easily as possible a diminutive Linux install that can be used for many use cases, like single board computers, virtual machines and my older hardware that I use for various tasks. Also offering complete control over what you do and don't install.

I have tried all sorts of Linux distributions, but I think I have found the ideal solution with this one.

Starting with the Ubuntu mini iso this makes the starting point very easy. You can install as much or as little as you like as you go through the installation process. Burn the iso file onto a USB drive or use it directly for your virtual machines. I basically didn't install anything that I didn't need to - especially towards the end when it ask's about GUI desktops - select nothing.

One thing to look out for is when partitioning the disk, whether virtual, SD or SDD don't set up a swapfile - it allocates 500MB on a device with 4GB of ram - which is pretty common these days.

Once all is installed you are presented with a standard command line when you reboot - which can be enough for a lot of people if you are going to run this as a server for some purpose. That takes up around 1.5GB disk space. This could be pruned further if needed, but even with a 16GB SD card, that's not too shabby. Especially compared to a full Gnome Ubuntu install which will eat around 6.5GB.

Now to get a simple working desktop on top of that I recommend using openbox - the following command installs all you need to get going, and give you the desktop above - minus the wallpapers - more of that in a mo.

sudo apt install openbox obconf obmenu vim xterm lightdm lightdm-gtk-greeter tint2 nitrogen ncdu xfce4-terminal arandr

The above is one command on one line.

Reboot your machine and you will be greeted with a login screen - login with the user you set up and you will be presented with a blank screen and a cursor - that is Openbox's starting place - immediately right-click the mouse and select the terminal.

Then carry out the following:

1. Launch tint2 to give you a panel
2. Launch arandr to set your video resolution - and save it to a file name to be used later.
3. Copy any wallpaper from any machine or website using ssh to your users home directory
4. Launch nitrogen to set that wallpaper you just saved. You can install more later.
5. Make these changes permanent.

To make option 5 happen:

Create a folder in /home/your username/.config called openbox.
In that directory create a file called autostart.

Add these lines to that file

nitrogen --restore &
tint2 &
/home/your username/.screenlayout/name-you-saved-it-as &

Once you have done this - you are good to go. Logout and back in, and you will have similar to the above image.

Now with the Ubuntu eco-system, you can install anything you like. This can be a Bastion, NFS, Samba, DNS server - whatever.

If you want to make it into a full-function desktop, add Firefox, VLC, Spotify, etc, etc.

However, the base from which you now start is 2.4GB of disk space used, which is the key to this.

You now have complete control over whatever you want to install and make this device into something you have designed and like.

It also minimises your security attack vector - as you have a lot less installed, less to update and less to keep an eye on. This is a massive plus for the whole process.

Updated: 14/4/2020

Getting DNS working with an Ubuntu 20.04 virtual machine installed on an Ubuntu 19.10 host

While taking a look at the next LTS release of Ubuntu - I found after I had spun up the new image in KVM on an Ubuntu 19.10 host that the DNS would not resolve - which scuppered me taking a really good look at it.

Now, they have moved DNS resolution into systemd for a while now, and on the host machine this has not caused me an issue. I have to say though, it appears to me that using systemd to resolve DNS is not only overly complicated but a waste of everyone's time - but I'm sure someone must appreciate the value of it.

I tested that the network was working correctly and the virtual machine could access the DNS resolver if it had been configured correctly with the following command:

dig @192.168.1.1 www.ubuntu.com

This worked, so I knew that the virt. machine would work if the DNS resolver was working correctly.

So, how to fix the issue. I tried several methods - each trying not to disable the systemd service - but all met with failure with my testing , so in the end I decided to just turn it off and use the tried and trusted /etc/resolv.conf

The commands to achieve this are:

systemctl stop systemd.resolved
systemctl disable systemd.resolved

Then edit the following line in the file 

/etc/NetworkManager/NetworkManager.conf 

[main]  

dns=default

Then remove the link in /etc

rm resolv.conf

Now create a new resolv.conf in /etc with the name of the nameserver you wish to use i.e.

namesever 192.168.1.1 < or whatever yours is >

The you need to restart the NetworkManager 

sudo systemctl restart NetworkManager

This worked perfectly and the virtual machine is now happily resolving DNS correctly.

None of the above is destructive and can be reversed if the systemd could be made to work, but as this was only a test machine, I decided I had wasted enough time on it.

NB. While working on some more virtual machines I came across this blog post which offers a more elegant solution to this DNS problem for Ubuntu/Debian based distros.It allows you to keep DNS resoltion in systemd - so I can't have been the only person having issues with it.

Solve local DNS issues in Ubuntu and Debian

 

How to use pi-hole with a Docker container on your Mac Laptop to stop unwanted internet adverts.

If you are fed up with pointless internet advertising on sites you visit, here is a great additional service you can install on your local machine - or more importantly for your network to stop it dead. I shall not go through what this product is as here is a link - Pi-Hole.

Basically, you need to install the Docker application on you laptop or desktop so that running up the pi-hole docker container is straight forward. You can get docker for Mac here. For the network installation, a Linux server virtual machine or docker container on a machine continuously running would make sense.

Then you need to clone the pi-hole docker git repository to your local machine

https://github.com/pi-hole/docker-pi-hole.git

Change into that directory and run docker_run.sh 

Once the script has run - it will spit out an admin password that you will need to remember to log into the web-based admin screen.

You can look at that by pointing a browser tab at http://127.0.0.1

Once logged in you will see something like this.

 The last part of getting this working on your laptop is to point the DNS resolution of OSX to point to the localhost - as pi-hole is now listening on port 53. Again for network-based installation you would point this at the IP address of your server running the service. You can also then setup that IP address in your routers DHCP settings so any machine on your LAN will get the same protection as they will push all their traffic through the new DNS server.

You can then run a test from the command line to make sure all your DNS requests are going via your new DNS service like so:

dig www.ubuntu.com

;; Query time: 23 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 13 11:16:40 GMT 2019
;; MSG SIZE  rcvd: 139

You will notice the response comes from your machines localhost - so all is working. With the settings of pi-hole you can specify several upstream DNS resolvers which also keeps your DNS queries out of the clutches of Google. There are many options - but I tend to use OpenDNS and 1.1.1.1

Enjoy. 

 

Staying safe on the Web - what can I do to make my browsing more secure and leach less data?

Friends and family often ask me about technologies they can use to make their lives just that little bit safer on the Web. So to save me having to answer the same questions repeatedly - I thought I would write a blog post to just highlight the tools,apps and extensions I use to make it better than just connecting to the web and hoping for the best.

• Use a VPN whenever and wherever you are. There are so many good and inexpensive examples to use these days - there is really no excuse not too. I hear good things about these ones Mozilla VPN or Proton VPN. Make sure when choosing a VPN that the provider guarantees to not keep logs - very important. I use my own - but those should be a good ones. This will work with you PC, iPad, mobile phone. So you will be covered whenever you decide to get some free internet in a cafe  you have never been to before :-)
• Use Mozilla Firefox, Safari or Opera Browser as your main browser. I know Google Chrome offers many features, but can you honestly trust Google to not be constantly looking to take your data and use it? I certainly don't trust Chrome anymore.
• Install a good set of extensions to stop trackers and unwanted information leaching.
• Adblock Plus
• Ghostery
• uBlock Origin or uBlock Origin Lite (needed for Chrome now)
• DuckDuckGo privacy essentials
• Privacy Badger
• HTTPS Everywhere

• Don't use Google as your default search engine - switch to using DuckDuckGo or Qwant- they are an option on all modern browsers - just change the default. You will be amazed at how all the targeted ads suddenly stop appearing everywhere - because you will have stopped Google building a complete profile of you on the web.
• Never use Facebook or Twitter to log into another site - always select to create a new account with your email and a strong password.
• Use a password manager to ensure strong passwords on all sites you use. Three good examples are Bitwarden, Lastpass,1Password or KeepassXC. 
• Where sites allow it use 2FA - 2 Factor Authentication - on all sites. Not all sites do - but check and where you can implement it. There are many apps you can use,  

1. Bitwarden Authenticator,  
2. FreeOTP, 
3. YubiKey Authenticator, 
4. Protectimus Smart OTP, 
5. Lastpass Authenticator, 
6. Google Authenticator.

• Turn on Firefox browsers "DNS over HTTPS" to keep your ISP from snooping on your site visits - it's to be found in Settings/Privacy&Security/DNS of HTTPS - might have to scroll down a little to find it.
• If you must use Facebook - I recommend you don't - then install an extension that puts it in a sandbox container - Firefox FB Container. This will reduce the amount of data you will leach from that app. 
• Talking about Firefox containers - using these you can isolate all sorts of other apps as well , using this add-on.
  Firefox containers - Howto
  
• Also for facebook - look at this article and turn it off - How to delete Facebooks off page tracking of you web surfing. There are many other settings, turn as much off as you can.
• Always look to use an anti-virus products on your phones, tablets and computers - there are many to choose from - I use AVG.
•  Make regular backups - so if you machine does get hijacked you have always got access to you valuable files. This can be to a secure cloud storage area as well as local USB type storage disk. Do this weekly, and occasionally attempt a restore to some files to make sure your backups are good.
  
• Glasswire is a great tool for keeping an eye on your Windows and Android machines activity and is worth installing https://www.glasswire.com/. They promise a Mac version soon.
• Useful YouTube video on securing Google Chrome if you must use it https://youtu.be/9lIMSzrjUrU
• Switch on DuckDuckGo anti-tracking feature on your Android phone. duckduckgo-app-tracking-tool-beta-android-users
• With Spam a constant threat there are now useful email relays/proxies which hide your real email address and offer security and spam prevention for free. There are two good ones Mozilla Relay or use DuckDuckgo email service. Worth the time and effort to setup and use.
• Switch to using as many open source tools as possible to stop being held captured by the technology Oligarch's. A good example is use LibreOffice instead of Microsoft office. Europe is doing this and moving away from these tools, so now is a good time to take that step. 
• Use the DuckDuckGo Extension to Block FLoC, Google’s New Tracking Method in Chrome Stop FLOC 
   
  Specifics for AI
   
  With the rise of AI tools everywhere, it makes sense to know if what you are reading is AI generated or not, this add on for Firefox helps with that task.
  AI Tool checker
   
  Also with AI tools now being present in your browser of choice, you need to make decisions on how you use that additional functionality. Picking the right model to use is the first choice, I tend to use Mistral, as it is European based. You need to understand how it works on your system, what data gets sent where, and how you can control that. 
   
  It is also worth being aware that as AI models require more and more data, that your personal information and posts on the internet are going to be consumed. This has many problems, especially with images that can then be manipulated with AI video to produce all sorts of unsavoury output. The need for digital privacy has never been higher and the thought before one posts is now a necessity.
  Linked-In is about to start mining everyone's data on their site, so make sure you go into your settings and turn that ability off. Also check all of the apps you use , and turn off all that data mining where you can.
   
  If you want to use AI tools safely and local only to your machine, then follow the advice in this article I wrote earlier in 2025.
  Truth about AI in Spring 2025 
  
  
   
  

There are other add-ons to stop javascript - which can stop a lot of nasty attacks - however - it can make a huge difference to the way the web looks and feels and a lot of sites depend on it. So unless you know what you are doing , I would stay clear of that to start.

I  also use pi-hole, a network-wide tool to stop unwanted advertising in its tracks. I will be writing an article shortly on how to set this up on your local laptop or network server.

Here is the article on using a docker container to run pi-hole - but you can use a virtual machine or a dedicated little machine like a Raspbery-pi to achieve the same thing. It is incredible useful and effective to stop all sorts of rubbish coming into your machine and network.

Setting up Pi-hole locally using Docker

Updated: 15/7/2019 
Updated: 13/11/2019
Updated: 9/02/2020
Updated: 1/3/2020
Updated: 16/6/2020
Updated: 11/09/2020
Updated: 15/03/2021
Updated: 14/05/2021
Updated: 8/8/2022
Updated:4/12/2022

Updated:8/2/2025

Updated:20/5/2025 

Updated:24/8/2025

Update:22/10/2025 

Updated:7/6/2026 

Useful reading on the Topic :

• Electronic Frontier Foundation
• Security Crash course

DNS resolution in Docker containers with Ubuntu Artful on AWS

This post is solution to a problem I discovered - so I hope others will find it useful.

Spinning up AWS Ubuntu Zesty - 17.04 - images with Docker installed was straight forward with Ansible and Terraform , but then arrived Ubuntu Artful - 17.10 , and the containers spun up could not resolve DNS, regardless of which version of Docker I installed.

After a lot of testing , it appeared to me that the host computer was passing through the wrong DNS server entry into resolv.conf within the container - so it would never work.

The Solution:

With systemd and docker, the preferred way to change a daemon setting is to create a new file in /etc/docker called daemon.json.

In that file add the following to get it use the AWS VPC default DNS resolver - 10.0.0.2 - like so

{
   "dns": ["10.0.0.2"]
}

Restart the docker daemon , and the containers can now resolve DNS. There may be other ways to resolve this issue, but this works perfectly , and uses methods preferred by the docker community.

I hope this helps others who may run into this problem.

Other settings that can be made in that file can be found here. Dockerd settings documentation

Getting apps to work in cutting edge Ubuntu Docker containers when they grizzle about locales.

Whilst running up an Artful Aardvark Ubuntu docker container on a Zesty Ubuntu host server, I received a message that it couldn't load certain apps due to the locales for UTF-8 were not configured.

This was annoying, but I initially worked around it , by installing the locales-all package into the container - it worked - but it bloated the container considerable.

There is a better and simpler way, which I found after a lot of digging around in the Docker documentation, as others must have hit this issue before.

What you need to do is set the following env variable in your Dockerfile when you build your container, and the problem does get solved for most apps, like tmux and screen within the container.

ENV LANG C.UTF-8

If you find that this doesn't cure it for your application , you may need to move to the next step and include the following in your Dockerfile.

       
RUN apt-get update && apt-get install -y locales \
&& rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A \
/usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
       
 

Getting an elastic search shard to relocate after a cluster nodes disk had become full

I came into work today to find one of the test environments elastic search(ES) nodes had run out of disk space. A bad job had gone berserk overnight and filled the logs - which then filled ES nodes disks.

So what to do with a volume with 100% utilisation? After chatting to a few colleagues , it was decided to delete one of the earlier indices - as the data in the dev environment was not life or death.

I first tried this from the GUI - which didn't work at all , so I switched to the CLI and issued the following

curl -XDELETE curl localhost:9200/mydodgylogs-2017.05.08 

That did the trick and we now had 80% disk utilisation, so I was expecting the shards to sort themselves calmly out. Unfortunately no go - there was one shard that was still refusing to relocate and it was effectively marking the cluster as yellow, so I had another chat with my colleague and he informed of a recovery log file which could make this happen. Effectively the disk being full had left the shard in an unstable state, and needed some help to sort itself out.

So on the cli again, I found the offending shard directory in the correct index and removed the following file

 mydodgylogs-2017.05.08/4/translog/translog-1234567890.recovering

As soon as that was removed the shard relocated fine and the system went back to being happy and green.

As it took some time with a few colleagues to get to the bottom of the problem, I thought others may find it useful in the future.

Running the normal health check command then gave me the following healthy output.

curl localhost:9200/_cluster/health?pretty

{
  "cluster_name" : "myclustername",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 45,
  "active_shards" : 90,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0

}

Using S3cmd to list the ACL's of all the files in an S3 bucket

In the absence of any replies from the S3cmd forum, I managed to use a command line hack to get the ACL status of all files in a bucket with this:

 I tried parsing the whole bucket with an "astrix" as an option , but that didn't work with version 1.0.1 or 1.0.5 of S3cmd. 

s3cmd -c ~/.s3cfg_uk2 ls s3://test-hubs/ | awk '{print $4}' | sed 's/s3\:\/\/test-hubs\///g' | xargs -I file s3cmd -c ~/.s3cfg_uk2 info s3://test-hubs/file 

 This is all on one line - and the xargs option is a capitol "i" and not an "el" as it appears here ;-) If anyone can see how to refactor this to make it more efficient be my guest, but it works.

Or indeed answer my original question with a snappy command line option to s3cmd ;-)


If you want to see if there is "anyone" access to a file you will see "anon" as the ACL setting , so you can search on that if you want to look for globally available files - which is what I wanted to do.

VM created by boot2docker init does not have DNS set up correctly - heres how to fix it

I came across this problem while setting up Docker on my OSX environment , and after searching the web , I found a method that works well, and cures the problem. Basically once the Virtualbox VM is created you need to edit the network adapter drivers and set them to "PCNET-FAST III" instead of the paravirtualised drivers. You need to do both adapters, and then restart boot2docker. You will notice that you get the correct settings now in your /etc/resolv.conf file. This was annoying me, and meant I had to reset the settings in the above file everytime I start docker - which is not ideal. Hope this saves you some time.

Little trick I found helpful programming the DigitalOcean v2 API - create a new droplet

While looking through several examples I found a few good links on how to create a new Virtual machine(VM) using the Ruby programming language, and the digital ocean gem they provide.

They all however left off one important point out, that you have to specify the SSH key id number as part of the call to create a new VM.

Without this last extra bit of the puzzle , you end up with a VM that is up and running, but you can't use your ssh keys to log into.
 - not ideal ;-)

The ssh keys have to be specified as a ruby array, as you could want more than one ssh key associated with your VM's once they have been created - so here is an example - with the missing piece inserted. The token mentioned in this code segment - is your oauth token that you create when you set up your digital ocena account. You can set it up as an env variable - but I have not shown how to do this here.

To start you need to install this gem into your Ruby environment

gem install droplet_kit

#!/usr/bin/ruby

require 'droplet_kit'
token=ENV['Oauth_key']
client = DropletKit::Client.new(access_token: token)
droplet = DropletKit::Droplet.new(name: 'example.com', region: 'nyc3', size: '1gb', image: 'ubuntu-14-04-x64', ssh_keys: [1234567])
client.droplets.create(droplet)

The code is downloadable from Github gist link below.

You can find out the id number of your ssh keys with the following bash script , so that you know which id's to put into the array.

curl -X GET -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" "https://api.digitalocean.com/v2/account/keys" | jq "."

Where $TOKEN is you oAuth API v2 key.

Hope this saves you time getting this up and running.

Ref blog post and Github gist of the code:
How to use the DigitalOcean v2API
Github Gist of the code

Using Docker to run rtorrent to reduce your servers resource requirements

Let me state early and clearly , any one who thinks Docker is a new technology, knows nothing about IT and even less about virtualisation. So if some smartalec in your office starts spouting about this new technology, take them down a peg or two with a few links to Solaris containers, OpenVZ or LXC.

What the Docker team have done extremely well, is make these ancient technologies very accessible and very easy to use, with a well defined tool set and API. This they must be highly praised for.

In the words of Einstein however, they work on the shoulders of giants, who did a lot of the heavy lifting, and let us all not forget this.

I have been using containers for years, and we worked on a very successful cloud at Nokia using OpenVZ, where we built our own tools.

So I thought I would give Docker a spin on one of my cloud servers and just kick the tyres to start. I picked an application I use a lot for downloading Linux ISO's so it seemed a good choice. It was very straight forward indeed.

The host operating system was CentOS 6.5 - which I'm growing less fond of as each week passes, as in the fast moving cloud space, Ubuntu is simply better. It was installed however, and I couldn't be bothered to change it. You need to enable the EPEL repository and install docker with yum.

I decided to download and use the userland tools of a Docker Ubuntu image - but whatever image you choose - the host kernel is the one that will be used. This is to do with the historically well thought out ABI built into the Linux kernel that allows this all to work.

Once the image was retrieved from the Docker registry - I fired up a container with Ubuntu 14.04 tools and installed screen, rtorrent , vim and htop.

I always run rtorrent with screen so I can just leave it running and come back to it when required. Also importantly - when starting your container use /bin/bash so you can have an interactive session with it to be able to go back and check your screen/rtorrent session. A command like the following will do.

docker run -i -t my-ubuntu-image /bin/bash

Be care however when you want to exit this container , DON'T type exit but CTRL-p, CTRL-q , so everything keeps running and you can reattach to the container when you wish to check on progress.

This uses significantly less resources than spinning up a KVM virtual image to do the same job, as it uses the resources of the host system that are already running.

I have deliberately not put all the commands need to do this here , as the Docker documentation is good and very clear, so duplication is pointless

Industries illogical use of Agencies in IT contractor procurement - a humorous musing

I thought I would write a light hearted look at what I genuinely consider to be the most illogical waste of companies money, in paying the middle men. These middle men take anything from 10-30% off the top of other peoples work, for doing virtually nothing.

Hiring IT managers know the sort of people they need, but instead of putting an advert directly on their web site or via aggregated sites like Jobserve, they employ the services of an agency. Here in lies the problem, as most agents have a strange and various past employment history and almost exclusively know next to nothing about complex IT systems. They may have heard of Windows if you're lucky, but start talking about Linux, Java, Ruby and Cloud Services and they start to glaze over

So in an attempt to consider what might be going through an ex-RSPCA dog handlers mind when talking to an IT hiring manager - here is my stab.

1) Agile - hmm so he wants the new guy to do yoga - probably a small office

2) Puppet - ok that makes sense he can keep the kids happy with a show on "bring your kids to work day"

3) Chef - good point , saves on kitchen staff as he can cook lunch while programming

4) Java - well if he is making the lunch he might as well make the coffee as well.

5)Tomcat - lion taming is a great skill, and could work well while the kids are in

6) VMware - probably similar to Tupaware - but why he wants him to sell home goods is a bit odd

7) Solaris - well green energy is in vogue - so sola panel knowledge could be handy

8) Perl - Knowledge of jewellery is always handy - especially to please the bosses wife

9) Ruby - Boy he really is into his Jewellery, I will have to check what they do for a business.

10) Python - that is just odd, perhaps he keeps snakes in the office as pets. Hope they don't escape, could cause chaos with the Lion.

This is obviously not an exhaustive list , and I could have been a lot harsher, but I hope it makes you smile like it does me when I see job adverts for Pearl developers - do they want me to polish gems for a living.

Is using email still relevant? Hell yes

I have been working in IT long enough to remember many changes in technology usage. I can remember just missing out on having to use punch cards to get my programs loaded into the mainframe.

So I can remember a time when people seriously did not think that email would or could be useful. I have actually worked in companies where the senior managers would get their secretaries to print out their emails and give them to them on paper. I can also vividly recall the many hours spent convincing these people of the benefit of email , and how it could save them time and money.

If you look at the world today through your 21st Century eyes, you would probably find those sentiments cute, if not funny, however I'm beginning to hear statements in my technology reading about the post email era, and I really think this would be a bad thing for many busy people.

I know we all get fed up with the spam we get, and the unsolicited contacts etc. but with a well set up email configuration, and not clicking on all those "send me updates" boxes, this can be controlled fairly easily.

Now I'm not some techno Luddite arguing from the sidelines as your knowledge gets swept away with yesterdays chip papers, but from a very practical perspective. I have shifted to using the mobile technology paradigm like everyone else, and use Whatsapp, Facebook Chat, Twitter and Wechat as well as the next man, but they have a big drawback. They are all great tools and I can see a use for them going forward , no doubt. However on your mobile devices they demand your attention , and are constant interrupts to your busy day. They are like a nagging yappy dog , constantly demanding to get in your face. Yes you can turn them down or the notifications off , but you have just killed 9/10ths of the usefulness of them in the first place. They are either in your face , or why bother using them at all?

The absolute beauty of email is it's asynchronous nature, and you can choose the time and place that you wish to read or respond to what has arrived in your inbox. It has also got thirty years of tooling development surrounding it, so finding and dealing with issues that took place a year ago is a breeze. Try doing that with Whatsapp - good luck with that.

Skype does a better job than most for recording what you have been up to with conversations, but it is still time limited. You can literally search you email threads from the day you opened your account, which has saved my bacon on more than one occasion.

I think the tricks to using email wisely are to be ruthless on your inbox, make sure you have a sensible structure with the folders/tags of the email you wish to keep and constantly look to help your spam engine to get rid of rubbish.

I'm sure email will be replaced at some point with probably more human friendly methods of communication , like video or on-the-fly video conversation recording. The main issue there will be the extremely complex nature of search, not a trivial issue with video. Lets hope the worlds computer scientists are on the case as I type ;-)

Problem loading/starting a KVM virtual machine on a CentOS 6.5 host that has SELINUX disabled

If you have recently built a KVM host for all your virtual machines and left SELINUX enabled, then all will be well, but what happens when you turn that setting off. Well you will end up with VM's that won't load with the error message

error: Failed to start domain Ubuntu1404
error: unsupported configuration: Unable to find security driver for label selinux

It is easy to fix , use the following command and just remove the seclabel section on the virtual machine domain XML - virsh edit Ubuntu1404

# virsh edit Ubuntu1404
...
  <;seclabel type='dynamic' model='selinux' relabel='yes'>;
    <label>system_u:system_r:svirt_t:s0:c95,c664</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c95,c664</imagelabel>
  </seclabel>  

Once you have done that, the domains will load as normal.

Make sure if you have backup's of the domain XML, they are also changed, as I have found that they will over write the domain if your not careful

Check in /etc/libvirt/qemu to make sure the changes have happen, and the XML has been updated.