DNS resolution in Docker containers with Ubuntu Artful on AWS

This post is solution to a problem I discovered - so I hope others will find it useful.

Spinning up AWS Ubuntu Zesty - 17.04 - images with Docker installed was straight forward with Ansible and Terraform , but then arrived Ubuntu Artful - 17.10 , and the containers spun up could not resolve DNS, regardless of which version of Docker I installed.

After a lot of testing , it appeared to me that the host computer was passing through the wrong DNS server entry into resolv.conf within the container - so it would never work.

The Solution:

With systemd and docker, the preferred way to change a daemon setting is to create a new file in /etc/docker called daemon.json.

In that file add the following to get it use the AWS VPC default DNS resolver - 10.0.0.2 - like so

{
   "dns": ["10.0.0.2"]
}

Restart the docker daemon , and the containers can now resolve DNS. There may be other ways to resolve this issue, but this works perfectly , and uses methods preferred by the docker community.

I hope this helps others who may run into this problem.

Other settings that can be made in that file can be found here. Dockerd settings documentation

Getting apps to work in cutting edge Ubuntu Docker containers when they grizzle about locales.

Whilst running up an Artful Aardvark Ubuntu docker container on a Zesty Ubuntu host server, I received a message that it couldn't load certain apps due to the locales for UTF-8 were not configured.

This was annoying, but I initially worked around it , by installing the locales-all package into the container - it worked - but it bloated the container considerable.

There is a better and simpler way, which I found after a lot of digging around in the Docker documentation, as others must have hit this issue before.

What you need to do is set the following env variable in your Dockerfile when you build your container, and the problem does get solved for most apps, like tmux and screen within the container.

ENV LANG C.UTF-8

If you find that this doesn't cure it for your application , you may need to move to the next step and include the following in your Dockerfile.

       
RUN apt-get update && apt-get install -y locales \
&& rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A \
/usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
       
 

Getting an elastic search shard to relocate after a cluster nodes disk had become full

I came into work today to find one of the test environments elastic search(ES) nodes had run out of disk space. A bad job had gone berserk overnight and filled the logs - which then filled ES nodes disks.

So what to do with a volume with 100% utilisation? After chatting to a few colleagues , it was decided to delete one of the earlier indices - as the data in the dev environment was not life or death.

I first tried this from the GUI - which didn't work at all , so I switched to the CLI and issued the following

curl -XDELETE curl localhost:9200/mydodgylogs-2017.05.08 

That did the trick and we now had 80% disk utilisation, so I was expecting the shards to sort themselves calmly out. Unfortunately no go - there was one shard that was still refusing to relocate and it was effectively marking the cluster as yellow, so I had another chat with my colleague and he informed of a recovery log file which could make this happen. Effectively the disk being full had left the shard in an unstable state, and needed some help to sort itself out.

So on the cli again, I found the offending shard directory in the correct index and removed the following file

 mydodgylogs-2017.05.08/4/translog/translog-1234567890.recovering

As soon as that was removed the shard relocated fine and the system went back to being happy and green.

As it took some time with a few colleagues to get to the bottom of the problem, I thought others may find it useful in the future.

Running the normal health check command then gave me the following healthy output.

curl localhost:9200/_cluster/health?pretty

{
  "cluster_name" : "myclustername",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 45,
  "active_shards" : 90,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0

}