kamal app exec includes secrets

[andrewjanssen]

Currently, containers created by kamal app exec don't have secrets. It would be helpful if they did have secrets, as env vars, like Kamal's other containers. This PR does that.

Resolves #1180.

[djmb]

Not requiring the secrets allows someone to run kamal app exec without having access to the secrets manager set up.

This means you can do things like giving someone access to a Rails console without having to set up authentication for wherever the secrets come from.

It is a little bit janky in that you need a deployment first to ensure the right secrets are in place, but I think that's worth it.

[andrewjanssen]

Not requiring the secrets allows someone to run kamal app exec without having access to the secrets manager set up.

Not requiring secrets is certainly beneficial, that's how I want Kamal to be too. This PR doesn't change that; if you haven't set any secrets, the behavior remains the same as it was without this PR.

This means you can do things like giving someone access to a Rails console without having to set up authentication for wherever the secrets come from.

Yes— that specific low-friction onboarding experience is important to me too and should be kept. This PR doesn't affect it. (In my case, the Rails console is not usable by default without the secrets, because RAILS_MASTER_KEY is a secret, which is what prompted me to write this PR. Certainly some meaningful percentage other people have a similar set-up as me.)

It is a little bit janky in that you need a deployment first to ensure the right secrets are in place, but I think that's worth it.

I don't intend for this PR to make secrets required. :) I only wanted this PR to inject any secrets which have been set. Lmk if I'm on the same page as you! Thanks for the review.

[djmb]

Not requiring secrets is certainly beneficial, that's how I want Kamal to be too. This PR doesn't change that; if you haven't set any secrets, the behavior remains the same as it was without this PR.

Oh sorry, I wasn't very clear there! What I mean is a situation where you have multiple users and some have the secret access set up and some don't. So you do have secrets, but you want the commands that don't need them to work for everyone.

This is for simplicity and not security though - anyone who can run commands can SSH to the servers and read the secrets anyway.

Where you have integrated a secret manager, your PR will make setting up secret access required for the app exec command because the secrets are lazy loaded by KAMAL.primary_role.env(KAMAL.primary_host).secrets.to_h.

If you have deployed to a server already, the secrets should be there, so I think this should only be an issue if that's not the case.

[cromega]

I would be more than happy with a secrets push command or even an optional argument to exec that would do this (and would fail if some secrets are missing)

[djmb]

Yeah an option makes sense here - we can add a boolean --push-secrets to kamal app exec defaulting to false.

[anatoly-pashin-axon21]

I was trying to run migrations using the pre-deploy hook, and spent a lot of time trying to figure out why it doesn't see the secrets.

My hook is the following:

kamal app exec --primary --roles=web --version="$KAMAL_VERSION" "./bin/console doctrine:migrations:migrate --no-interaction --allow-no-migration -vvv"

Turns out the secrets’ file is not there until the first deployment happens. But the hook will fail, and the deployment won't finish. So, currently, to do the first deployment to the new host, I have to make a deployment without the hooks.

Would be really nice to have the mentioned flag --push-secrets or maybe a separate command that just places the secrets’ files.

The error:

DEBUG [e33ea858] 	  INFO [3893cf09] Running docker run --rm --name redacted-web-exec-c4a6e73d6c664b4b1b04345c9ffcdd0600622a7d-27e910 --network kamal --env redacted="true" --env redacted --env redacted="1" --env redacted="dev" --env-file .kamal/apps/redacted/env/roles/web.env --log-opt max-size="10m" ghcr.io/redacted/redacted:c4a6e73d6c664b4b1b04345c9ffcdd0600622a7d ./bin/console doctrine:migrations:migrate --no-interaction --allow-no-migration -vvv on app.staging.internal

 DEBUG [e33ea858] 	  ERROR (SSHKit::Runner::ExecuteError): Exception while executing on web: Exception while executing on host app.staging.internal: docker exit status: 125

 DEBUG [e33ea858] 	docker stdout: Nothing written

 DEBUG [e33ea858] 	docker stderr: docker: open .kamal/apps/redacted/env/roles/web.env: no such file or directory

Looking at the PR contents, I guess with the updated code, the hook will still fail because it doesn't create the env file and just updates the env variable? I'm not sure because I don't speak ruby much.

[anatoly-pashin-axon21]

I've tried to implement a new flag to push secrets: #1742. Please review when you have time.

Also, NB about the pre-deploy hook: the hook is executed before the docker network is created, so app exec would fail with a docker error (network kamal not found). The correct way is to use pre-app-boot hook instead that is run after the proxy and the network is set up.

So, in my case, it's like this .kamal/hooks/pre-app-boot:

#!/bin/bash

kamal app exec --push-secrets --primary --roles=web --version="$KAMAL_VERSION" "./bin/console doctrine:migrations:migrate --no-interaction --allow-no-migration -vvv"

Then kamal deploy works fine for the fresh servers.

[djmb]

Closing in favour of #1742