"pymongo.errors.OperationFailure: auth failed"

[ch40s]

Hey all, I'm using the arm image "webhippie/mongodb:latest" and I get the following error, any idea how to troubleshoot and solve this?

$ docker service logs -f walkoff_core_api
Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 331, in main
    run(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 354, in run
    server.run()
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 382, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.7/site-packages/uvicorn/main.py", line 389, in serve
    config.load()
  File "/usr/local/lib/python3.7/site-packages/uvicorn/config.py", line 288, in load
    self.loaded_app = import_from_string(self.app)
  File "/usr/local/lib/python3.7/site-packages/uvicorn/importer.py", line 20, in import_from_string
    module = importlib.import_module(module_str)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked                                                                                  [8/1955]
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./api/server/app.py", line 15, in <module>
    from api.server.db.mongo import mongo, get_mongo_c
  File "./api/server/db/mongo/__init__.py", line 87, in <module>
    mongo = MongoManager()
  File "./api/server/db/mongo/__init__.py", line 21, in __init__
    self.init_db()
  File "./api/server/db/mongo/__init__.py", line 31, in init_db
    self.reg_client.walkoff_db.apps.create_indexes([id_index, name_index])
  File "/usr/local/lib/python3.7/site-packages/pymongo/collection.py", line 1841, in create_indexes
    with self._socket_for_writes(session) as sock_info:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1222, in _get_socket
    self.__all_credentials, checkout=exhaust) as sock_info:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/pymongo/pool.py", line 1138, in get_socket
    sock_info.check_auth(all_credentials)
  File "/usr/local/lib/python3.7/site-packages/pymongo/pool.py", line 712, in check_auth
    auth.authenticate(credentials, self)
  File "/usr/local/lib/python3.7/site-packages/pymongo/auth.py", line 564, in authenticate
    auth_func(credentials, sock_info)
  File "/usr/local/lib/python3.7/site-packages/pymongo/auth.py", line 543, in _authenticate_default
    return _authenticate_mongo_cr(credentials, sock_info)
  File "/usr/local/lib/python3.7/site-packages/pymongo/auth.py", line 525, in _authenticate_mongo_cr
    sock_info.command(source, query)
  File "/usr/local/lib/python3.7/site-packages/pymongo/pool.py", line 613, in command
    user_fields=user_fields)
  File "/usr/local/lib/python3.7/site-packages/pymongo/network.py", line 167, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/usr/local/lib/python3.7/site-packages/pymongo/helpers.py", line 159, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: auth failed

[adpham95]

Going to address #256 in this as well because they are similar issues.

When Walkoff is built for the first time, the Bootloader will generate Docker secrets that are used as credentials for the resources like Mongo, Redis, Minio, etc.

These secrets are mounted as files in their respective containers under /run/secrets/walkoff_<resourcename>_key. If you are changing the image used by the resources, the new images should also be able to accept files as credential sources by setting them in environment variables like the images currently used. This why we use Bitnami images instead of official images for some resources, as they all support this pattern, though it doesn't look like they support ARM.

If your alternate images can't do this, for example they only accept the password directly in the environment variable itself (I think the webhippie/mongodb image falls into this category), or the password is inside a larger file (like the redis.conf in official Redis images), you can work around this albeit not in an ideal manner.

For testing purposes, when you use ./walkoff.sh up you can also specify the -d debug flag, which sets all credentials to walkoff123456. You can then hardcode that into the appropriate environment variable in base-compose.yml for webhippie/mongodb and into a redis.conf that you bind mount into the Redis service.

I think the real solution may be to support templating in the compose file, but roadmap is unclear at this point in time.

[ch40s]

Thanks @adpham95 for taking the time to comment on this. That makes sense and being able to easily support other architectures and images via the compose file sounds like a great feature request. I'll try to resolve this following your recommendation above and will provide an update here.

[ch40s]

@adpham95 :

• "aioredis.errors.AuthError: ERR Client sent AUTH, but no password is set" #256 was resolved by using a redis.conf file for the redis service.
• Mongo is still failing, even when using the debug flag. Any idea what am I missing?

  resource_mongo:
    image: webhippie/mongodb:latest
    networks:
      - walkoff_network
    ports:
      - 27016:27016
    environment:
      #Reference: https://hub.docker.com/r/webhippie/mongodb
      - MONGODB_ROOT_USERNAME=walkoff
      - MONGODB_ROOT_PASSWORD=walkoff123456
    volumes:
      - walkoff_resource_mongo_volume:/data/db
    command: mongod --port 27016

Error:

$ docker service logs -f walkoff_resource_mongo
authenticate db: admin { authenticate: 1, user: "walkoff", nonce: "xxx", key: "xxx" }
Failed to authenticate walkoff@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user walkoff@admin

[adpham95]

Can you see if that image has the MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD environment variables or variations thereof? They're not on the dockerhub page so I'm not sure, but in the official Mongo image, the INITDB variant of those environment variables actually creates said user w/ password.

[adpham95]

Hmm.

https://github.com/dockhippie/mongodb/blob/master/latest/overlay/etc/s6/mongodb/run#L92

If this is the correct source code, then at least it seems like that MONGODB_ROOT_USERNAME should create the user. Maybe check the logs of the Mongo container to see if there's anything going wrong with that.

[ch40s]

Nothing interesting in the Mongo container logs.

$ docker container logs -f f41494f.....
warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
[initandlisten] MongoDB starting : pid=1 port=27016 dbpath=/data/db 32-bit host=f41494f8a4c7
[initandlisten]
[initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
[initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
[initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
[initandlisten] **       See http://dochub.mongodb.org/core/32bit
[initandlisten]
[initandlisten] db version v2.6.10
[initandlisten] git version: nogitversion
[initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
[initandlisten] build info: Linux kishi10 3.2.0-84-highbank #121-Ubuntu SMP PREEMPT Tue May 5 19:44:15 UTC 2015 armv7l BOOST_LIB_VERSION=1_58
[initandlisten] allocator: tcmalloc
[initandlisten] options: { net: { port: 27016 } }
[initandlisten] allocating new ns file /data/db/local.ns, filling with zeroes...
[FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
[FileAllocator] creating directory /data/db/_tmp
[FileAllocator] done allocating datafile /data/db/local.0, size: 64MB,  took 0.002 secs
[initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
[initandlisten]     added index to empty collection
[initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0  reslen:37 1192ms
[initandlisten] waiting for connections on port 27016
[clientcursormon] mem (MB) res:39 virt:208
[clientcursormon]  mapped:80
[clientcursormon]  connections:0
[initandlisten] connection accepted from 10.0.12.89:32868 #1 (1 connection now open)
[initandlisten] connection accepted from 10.0.12.89:32870 #2 (2 connections now open)
[conn2] authenticate db: admin { authenticate: 1, user: "walkoff", nonce: "xxx", key: "xxx" }
[conn2] Failed to authenticate walkoff@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user walkoff@admin
[conn2] end connection 10.0.12.89:32870 (1 connection now open)
[conn1] end connection 10.0.12.89:32868 (1 connection now open)