Why is my Docker volume not working in a remote build box?

I am attempting to add a volume to a Docker container that will be built and run in a Docker Compose system on a hosted build service (CircleCI). It works fine locally, but not remotely. CircleCI provide an SSH facility I can use to debug why a container is not behaving as expected.

The relevant portion of the Docker Compose file is thus:

  missive-mongo:
    image: missive-mongo
    command: mongod -v --logpath /var/log/mongodb/mongodb.log --logappend
    volumes:
    - ${MONGO_LOCAL}:/data/db
    - ${LOGS_LOCAL_PATH}/mongo:/var/log/mongodb
    networks:

    - storage_network

Locally, if I do docker inspect integration_missive-mongo_1 (i.e. the running container name, I will get the volumes as expected:

    ...
    "HostConfig": {
        "Binds": [
            "/tmp/missive-volumes/logs/mongo:/var/log/mongodb:rw",
            "/tmp/missive-volumes/mongo:/data/db:rw"

        ],
    ...

On the same container, I can shell in and see that the volume works fine:

docker exec -it integration_missive-mongo_1 sh
/ # tail /var/log/mongodb/mongodb.log 
2017-11-28T22:50:14.452+0000 D STORAGE  [initandlisten] admin.system.version: clearing plan cache - collection info cache reset
2017-11-28T22:50:14.452+0000 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" }

2017-11-28T22:50:14.452+0000 I INDEX    [initandlisten]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2017-11-28T22:50:14.452+0000 D INDEX    [initandlisten]      bulk commit starting for index: incompatible_with_version_32
2017-11-28T22:50:14.452+0000 D INDEX    [initandlisten]      done building bottom layer, going to commit
2017-11-28T22:50:14.454+0000 I INDEX    [initandlisten] build index done.  scanned 0 total records. 0 secs
2017-11-28T22:50:14.455+0000 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.4
2017-11-28T22:50:14.455+0000 I NETWORK  [thread1] waiting for connections on port 27017
2017-11-28T22:50:14.455+0000 D COMMAND  [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner
2017-11-28T22:50:14.455+0000 D COMMAND  [ClientCursorMonitor] BackgroundJob starting: ClientCursorMonitor

OK, now for the remote. I kick off a build, it fails because Mongo won't start, so I use the SSH facility that keeps a box alive after a failed build.

I first hack the DC file so that it does not try to launch Mongo, as it will fail. I just get it to sleep instead:

  missive-mongo:
    image: missive-mongo
    command: sleep 1000
    volumes:
    - ${MONGO_LOCAL}:/data/db
    - ${LOGS_LOCAL_PATH}/mongo:/var/log/mongodb

    networks:
    - storage_network

I then run the docker-compose up script to bring all containers up, and then examine the problematic box: docker inspect integration_missive-mongo_1:

    "HostConfig": {
        "Binds": [
            "/tmp/missive-volumes/logs/mongo:/var/log/mongodb:rw",
            "/tmp/missive-volumes/mongo:/data/db:rw"

        ],

That looks fine. So on the host I create a dummy log file, and list it to prove it is there:

bash-4.3# ls /tmp/missive-volumes/logs/mongo
mongodb.log

So I try shelling in, docker exec -it integration_missive-mongo_1 sh again. This time I find that the folder exists, but not the volume contents:

/ # ls /var/log
mongodb
/ # ls /var/log/mongodb/
/ #

This is very odd, because the reliability of volumes in the remote Docker/Compose config has been exemplary up until now.

Theories

My main one at present is that the differing versions of Docker and Docker Compose could have something to do with it. So I will list out what I have:

• Local
  
  
  
  
  • Host: Linux Mint
  
  
  • Docker version 1.13.1, build 092cba3
  
  
  • docker-compose version 1.8.0, build unknown
  
  
  


• Remote
  
  
  
  
  • Host: I suspect it is Alpine (it uses apk for installing)
  
  
  • I am using the docker:17.05.0-ce-git image supplied by CircleCI, the version shows as Docker version 17.05.0-ce, build 89658be
  
  
  • Docker Composer is installed via Pip, and getting the version produces docker-compose version 1.13.0, build 1719ceb.
  
  

So, there is some version discrepancy. As a shot in the dark, I could try bumping up Docker/Compose, though I am wary of breaking other things.

What would be ideal though, is some sort of advanced Docker commands I can use to debug why the volume appears to be registered but is not exposed inside the container. Any ideas?

Answer

CircleCI runs docker-compose remotely from the Docker daemon so local bind mounts don't work.

A named volume will default to the local driver and would work in CircleCI's Compose setup, the volume will exist where ever the container runs.

Logging should generally be left to stdout and stderr in a single process per container setup. Then you can make use of a logging driver plugin to ship to a central collector. MongoDB defaults to logging to stdout/stderr when run in the foreground.

Combining the volumes and logging:

version: "2.1"

services:

  syslog:
    image: deployable/rsyslog
    ports:
      - '1514:1514/udp'

      - '1514:1514/tcp'

  mongo:
    image: mongo
    command: mongod -v
    volumes:
      - 'mongo_data:/data/db'
    depends_on:
      - syslog
    logging:

      options:
        tag: '{{.FullID}} {{.Name}}'
        syslog-address: "tcp://10.8.8.8:1514"
      driver: syslog

volumes:
  mongo_data:

This is a little bit of a hack as the logging endpoint would normally be external, rather than a container in the same group. This is why the logging uses the external address and port mapping to access the syslog server. This connection is between the docker daemon and the log server, rather than container to container.