Secrets

Modified on Wed, 14 Sep 2022 at 06:20 AM

Q: When using secrets in Tower workflow run, the process executed with an error Missing AWS execution role arn

The ECS Agent must be empowered to retrieve Secrets from the AWS Secrets Manager. Tower-launched pipelines that use Secrets and execute in an AWS Batch Compute Environment will encounter this error if an IAM Execution Role is not provided. Please see Pipeline Secrets for remediation steps.

Q: Why do work tasks which use Secrets fail when running in AWS Batch?

Users may encounter a few different errors when executing pipelines that use Secrets, via AWS Batch:

  • If you use nf-sqldb version 0.4.1 or earlier and have Secrets in your nextflow.config, you may see this error in your Nextflow Log: nextflow.secret.MissingSecretException: Unknown config secret {SECRET_NAME}.
    You can resolve this error by explicitly defining the xpack-amzn plugin in your configuration.
    Example:

plugins {
  id 'xpack-amzn'
  id 'nf-sqldb'
}
  • If you have two or more processes that use the same container image, but only a subset of these processes use Secrets, your Secret-using processes may fail during the initial run, but succeed when resumed. This is due to a bug in how Nextflow (22.07.1-edge and earlier) registers jobs with AWS Batch.

    To resolve the issue, please upgrade your Nextflow version to 22.08.0-edge. If you cannot upgrade, you can use the following as workarounds:

    1. Use a different container image for each process.
    2. Define the same set of Secrets in each process that uses the same container image.


Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select atleast one of the reasons

Feedback sent

We appreciate your effort and will try to fix the article