diff --git a/submodule05_Nextflow_AWSBatch.ipynb b/submodule05_Nextflow_AWSBatch.ipynb
index 0503a62..3e33079 100755
--- a/submodule05_Nextflow_AWSBatch.ipynb
+++ b/submodule05_Nextflow_AWSBatch.ipynb
@@ -9,18 +9,13 @@
]
},
{
+ "attachments": {},
"cell_type": "markdown",
- "id": "4126cb07-34ee-4780-838f-872015a882b3",
- "metadata": {},
- "source": [
- "## Overview"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f15ea992-faa6-4705-8384-eb5d81f5daff",
+ "id": "c090b42f-ffd5-41c0-b599-9e1f8a463369",
"metadata": {},
"source": [
+ "## Overview\n",
+ "\n",
"This short tutorial demonstrates how to run a comparative genomics workflow using a bacteria data set. Steps in the workflow combine the analyses you performed in previous submodules, and include:\n",
"- Read QC and trimming (fastqc, fastp)\n",
"- Genome assembly (SPAdes)\n",
@@ -33,18 +28,14 @@
"\n",
"The tutorial uses a popular workflow manager called [Nextflow](https://www.nextflow.io) run via [AWS Batch](https://aws.amazon.com/batch/) to automate the processes run in Submodules 1 & 2.\n",
"\n",
- "AWS Batch will create the needed permissions, roles and resources to run Nextflow in a serverless manner. You can set up AWS Batch manually or deploy it **automatically** with a stack template. The Launch Stack button below will take you to the cloud formation create stack webpage with the template with required resources already linked. \n",
- "\n",
- "If you prefer to skip manual deployment and deploy automatically in the cloud, click the Launch Stack button below. For a walkthrough of the screens during automatic deployment please click [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/HowToLaunchAWSBatch.md). The deployment should take ~5 min and then the resources will be ready for use. \n",
- "\n",
- "[](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?stackName=aws-batch-nigms&templateURL=https://nigms-sandbox.s3.us-east-1.amazonaws.com/cf-templates/AWSBatch_template.yaml )\n",
"\n",
- "Before begining this tutorial, if you do not have required roles, policies, permissions or compute environment and would like to **manually** set those up please click [here](https://github.com/NIGMS/NIGMS-Sandbox/blob/main/docs/AWS-Batch-Setup.md) to set that up.\n"
+ "#### About AWS Batch\n",
+ "AWS Batch will create the needed permissions, roles and resources to run Nextflow in a serverless manner. You can set up AWS Batch manually or deploy it **automatically** with a stack template. Please see **Setting up AWS Batch** in the Get Started section below to learn more about how to use it.\n"
]
},
{
"cell_type": "markdown",
- "id": "56d718a8",
+ "id": "80e981d2-010c-4139-80d5-7e7c4b14f8a5",
"metadata": {},
"source": [
"## Prerequisites\n",
@@ -55,8 +46,9 @@
"+ Please ensure you have a VPC, subnets, and security group set up before running this tutorial.\n",
"+ Role with AdministratorAccess, AmazonSageMakerFullAccess, S3 access and AWSBatchServiceRole.\n",
"+ Instance Role with AmazonECS_FullAccess, AmazonEC2ContainerRegistryFullAccess, and S3 access.\n",
- "+ If you do not have the required set-up for AWS Batch please follow this tutorial [here](https://github.com/STRIDES/NIHCloudLabAWS/blob/zbyosufzai-awsbatch-1/notebooks/AWSBatch/Intro_AWS_Batch.ipynb#install_nextflow). ***When making the instance role, make another for SageMaker notebooks with the following permissions: AdminstratorAccess, AmazonEC2ContainerRegistryFullAccess, AmazonECS_FullAccess, AmazonS3FullAccess, AmazonSageMakerFullAccess, and AWSBatchServiceRole.***\n",
- "It is recommended that specific permission to folders are added through inline policy. An example of the JSON is below:\n",
+ "+ If you do not have the required set-up for AWS Batch please follow this tutorial [here](https://github.com/STRIDES/NIHCloudLabAWS/blob/main/notebooks/AWSBatch/Intro_AWS_Batch.ipynb).\n",
+ "+ ***When making the instance role, make another for SageMaker notebooks with the following permissions: AdminstratorAccess, AmazonEC2ContainerRegistryFullAccess, AmazonECS_FullAccess, AmazonS3FullAccess, AmazonSageMakerFullAccess, and AWSBatchServiceRole.***\n",
+ "+ It is recommended that specific permission to folders are added through inline policy. An example of the JSON is below:\n",
"\n",
"\n",
"{\n",
@@ -84,21 +76,68 @@
" ]\n",
"}\n",
"\n",
- "For AWS bucket naming conventions, please click [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html)."
+ "For AWS bucket naming conventions, please click [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html).\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "
\n",
+ "
\n",
+ "Before begining this tutorial, if you do not have required roles, policies, permissions or compute environment and would like to manually set those up please click here to set that up.\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "
\n",
+ "After successfull creation of your stack you must attatch a new role to SageMaker to be able to submit batch jobs. Please following the the following steps to change your SageMaker role:
\n",
+ "
- Navigate to your SageMaker AI notebook dashboard (where you initially created and launched your VM)
- Locate your instance and click the Stop button
- Once the instance is stopped:
- Click Edit
- Scroll to the \"Permissions and encryption\" section
- Click the IAM role dropdown
- Select the new role created during stack formation (named something like aws-batch-nigms-SageMakerExecutionRole)
\n",
+ "- Click Update notebook instance to save your changes
\n",
+ "- After the update completes:
- Click Start to relaunch your instance
- Reconnect to your instance
- Resume your work from this point
\n",
+ "\n",
+ "
Warning: Make sure to replace the
stack name to the stack that you just created.
STACK_NAME = \"your-stack-name-here\"\n",
+ " \n",
+ "
\n",
+ "Install Java and Nextflow if needed in other systems
\n",
+ "If using other system other than AWS SageMaker Notebook, you might need to install java and nextflow using the code below:\n",
+ "
# Install java\n",
+ " sudo apt update\n",
+ " sudo apt-get install default-jdk -y\n",
+ " java -version\n",
+ "
\n",
+ " # Install Nextflow\n",
+ " curl https://get.nextflow.io | bash\n",
+ " chmod +x nextflow\n",
+ " ./nextflow self-update\n",
+ " ./nextflow plugin update\n",
+ "
\n",
+ " "
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "id": "3124016c",
+ "cell_type": "markdown",
+ "id": "d1d3c2cf-c12c-4343-871e-14e147d79929",
"metadata": {},
- "outputs": [],
"source": [
- "#Install nexflow, make it exceutable, and update it\n",
- "! curl https://get.nextflow.io | bash\n",
- "! chmod +x nextflow\n",
- "! ./nextflow self-update\n",
- "! ./nextflow plugin update"
+ "#### Create additional .config file needed\n",
+ "A configuration template (aws_batch_template.config) is customized below with your actual AWS values:\n",
+ "- S3 bucket name\n",
+ "- AWS job queue name\n",
+ "- AWS region\n",
+ "\n",
+ "This file tells Nextflow later how to communicate with AWS Batch and where to find the resources it needs."
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "cfa2e068",
+ "id": "44c9fb02-dbe9-41d2-a984-7634f6aebd91",
"metadata": {},
"outputs": [],
"source": [
+ "# copy the aws batch configuration file \n",
+ "! cp wgsbac/conf/aws_batch_template.config aws_batch_submodule5.config \n",
"# replace batch bucket name in nextflow configuration file\n",
- "! sed -i \"s/aws-batch-nigms-batch-bucket-/$BUCKET_NAME/g\" wgsbac/nextflow.config"
+ "! sed -i \"s/aws-batch-nigms-batch-bucket-/$BUCKET_NAME/g\" aws_batch_submodule5.config \n",
+ "# replace job queue name in configuration file \n",
+ "! sed -i \"s/aws-batch-nigms-JobQueue/$AWS_QUEUE/g\" aws_batch_submodule5.config \n",
+ "# replace the region placeholder with your region\n",
+ "! sed -i \"s/us-east-1/$AWS_REGION/g\" aws_batch_submodule5.config "
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "id": "9417e8be",
+ "cell_type": "markdown",
+ "id": "fa9cbd3e-d003-48d1-8b41-924c74b9486f",
"metadata": {},
- "outputs": [],
"source": [
- "# replace job queue name in configuration file \n",
- "! sed -i \"s/aws-batch-nigms-JobQueue/$AWS_QUEUE/g\" wgsbac/nextflow.config"
+ "### Step 2. Enable AWS Batch for the nextflow script \n",
+ "Run the pipeline in a cloud-native, serverless manner using AWS Batch. AWS Batch offloads the burden of provisioning and managing compute resources. When you execute this command:\n",
+ "- Nextflow uploads tasks to AWS Batch. \n",
+ "- AWS Batch pulls the necessary containers.\n",
+ "- Each process/task in the pipeline runs as an isolated job in the cloud."
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "09770029",
+ "id": "065b7e42-53f4-4cd9-a067-d03c7b87f482",
"metadata": {},
"outputs": [],
"source": [
- "# replace the region placeholder with your region\n",
- "! sed -i \"s/aws-region/$AWS_REGION/g\" wgsbac/nextflow.config"
+ "# Run nextflow script with parameters \n",
+ "! nextflow run wgsbac/main.nf -profile docker,awsbatch \\\n",
+ " --input s3://$INPUT_FOLDER/samplesheet_test.csv \\\n",
+ " -c aws_batch_submodule5.config \\\n",
+ " -resume"
]
},
{
"cell_type": "markdown",
- "id": "d6424c54-108f-459e-8f14-f2866bfc0141",
+ "id": "309fff1d-9c02-4386-895d-787fb8c3b5b4",
"metadata": {},
"source": [
- "### Step 2. Enable AWS Batch for the nextflow script "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d6cb4617-b98c-41b6-995d-711d2f722cbf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Run nextflow script with parameters \n",
- "! ./nextflow run wgsbac/main.nf --input s3://$INPUT_FOLDER/samplesheet_test.csv -profile docker,awsbatch -c wgsbac/nextflow.config --awsqueue $AWS_QUEUE --awsregion $AWS_REGION"
+ "#### Key Differences from Local Execution:\n",
+ "\n",
+ "\n",
+ " \n",
+ " \n",
+ " | Feature | \n",
+ " Local Execution (e.g., SageMaker) | \n",
+ " AWS Batch Execution | \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ " | Compute | \n",
+ " Uses notebook’s limited CPU/memory | \n",
+ " Uses scalable EC2/Fargate resources | \n",
+ "
\n",
+ " \n",
+ " | Data Location | \n",
+ " Reads from local disk | \n",
+ " Reads directly from S3 | \n",
+ "
\n",
+ " \n",
+ " | Job Management | \n",
+ " Manual or single-threaded | \n",
+ " Distributed via job queues | \n",
+ "
\n",
+ " \n",
+ " | Reproducibility | \n",
+ " Depends on local environment | \n",
+ " Fully containerized via Docker | \n",
+ "
\n",
+ " \n",
+ " | Fault Tolerance | \n",
+ " Limited | \n",
+ " Retries & logs handled by AWS Batch | \n",
+ "
\n",
+ " \n",
+ "
"
]
},
{
"cell_type": "markdown",
- "id": "7187f8dc-bcb6-4caf-b17c-183fc776147f",
+ "id": "7e18731d-59ef-4c45-bc9e-6e5773f1b786",
"metadata": {},
"source": [
- "### Step 3: Explore Results"
+ "### Step 3: Explore Results\n",
+ "This command lets you preview the full paths of output files within the S3 bucket. These results should be identical to the ones generated in Submodule 4, where the pipeline was run locally. The only differences lie in: (1) Execution environment: Local notebook vs. AWS Batch; (2)Data paths: Local file system vs. S3 input/output directories. Everything else: tools, parameters, and pipeline structure, remains the same, ensuring consistency across both local and cloud executions."
]
},
{
@@ -293,9 +385,60 @@
"from IPython.display import Image\n",
"Image('wgsbac/assets/results/final_results/blobtools_plots/SRR10056829_T1_blobplot_read_cov.png', width=1200)"
]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bb75509d-3000-4cfd-93ac-d910d2a97dca",
+ "metadata": {},
+ "source": [
+ "## Clean Up the AWS Environment\n",
+ "\n",
+ "Once you've successfully run your analysis and downloaded the results, it's a good idea to clean up unused resources to avoid unnecessary charges.\n",
+ "\n",
+ "#### Recommended Cleanup Steps:\n",
+ "\n",
+ "- **Delete Output Files from S3 (Optional)** \n",
+ " If you've downloaded your results locally and no longer need them stored in the cloud.\n",
+ "- **Delete the S3 Bucket (Optional)** \n",
+ " To remove the entire bucket (only do this if you're sure!)\n",
+ "- **Shut Down AWS Batch Resources (Optional but Recommended):** \n",
+ " If you used a CloudFormation stack to set up AWS Batch, you can delete all associated resources in one step (⚠️ Note: Deleting the stack will also remove IAM roles and compute environments created by the template.):\n",
+ " + Go to the AWS CloudFormation Console\n",
+ " + Select your stack (e.g., aws-batch-nigms-test1)\n",
+ " + Click Delete\n",
+ " + Wait for all resources (compute environments, roles, queues) to be removed\n",
+ " \n",
+ "\n",
+ "
\n",
+ "
\n",
+ "It’s always good practice to periodically review your EC2 instances, ECR containers, S3 storage, and CloudWatch logs to ensure no stray resources are incurring charges.\n",
+ "
\n",
+ "