HPC Cluster on AWS

This repository provides an introduction to key Terraform and Ansible concepts in the context of HPC on AWS. It also includes the Terraform configurations and Ansible playbooks used to provision an AWS-based HPC environment, including head nodes, compute nodes, shared storage, and supporting services.

Contents

• Cluster Design
• Automated Cluster Creation
  • AWS IAM User
  • Rocky Linux Subscription
  • HCP Account and Terraform
  • Ansible
• Virtual Machine Setup
  • Networking Architecture
  • Virtual Private Cloud (VPC)
  • Public Subnet
  • Internet Gateway
  • Public Route Table
  • Route Table Association
  • Security Groups (SG)
    • SSH Access Security Group
    • Internal Node Communication Security Group
    • Monitoring Security Group
  • EC2 Instance Setup
    • SSH Key Pair
    • Security Groups
    • Storage Configuration
    • EC2 Instances Setup
• Preparing AWS instances
• Passwordless SSH
  • Create the SSH keys
  • Set the hostname
  • Hosts File Setup
  • Append Keys
  • Disable cloud configurations
• BeeGFS File System
  • Disk and Directory setup
  • Management Node
  • Storage Nodes
  • Configuration and setup
    • Configure Management Node
    • Configure Metadata server
    • Storage Server Configuration
    • Install BeeGFS Client and Dependencies
• PBS Scheduler
  • Initial installation
  • Build OpenPBS from Source
  • Configuration Files
    • PBS Server
    • Client Node
    • Compute Nodes
  • Configuring and Verifying PBS Nodes
• LDAP
  • LDAP Server Configuration
  • LDAP Client Configuration
• Prometheus and Grafana
  • Prometheus
  • Node Exporter
  • Grafana
  • Dashboard
• Reference
• Contributers