diff --git a/code-backup/README.md b/code-backup/README.md index 8b2eacd..ffe1513 100644 --- a/code-backup/README.md +++ b/code-backup/README.md @@ -1,22 +1,24 @@ -# GitHub Projects Backup Script +# GitHub Projects Backup Scripts -This script automatically backs up all of your GitHub repositories (both public and private) by: +This directory contains two scripts for backing up your GitHub repositories: -- Fetching all repositories from your GitHub account using the GitHub API -- Creating a `~/Projects` directory if it doesn't exist -- Cloning new repositories or updating existing ones to their latest default branch -- Creating a timestamped zip backup of all projects: `~/Projects-Backup_YYYY-MM-DD_HH-MM-SS.zip` +1. **`code-backup-local.sh`** - Creates a local, zipped directory of all your non-archived projects +2. **`code-backup-gitlab.sh`** - Mirrors all non-archived public and private projects to similarly named GitLab projects --- ## πŸ›  Requirements +Both scripts require: + - Bash shell (version 4.0+) - `git` CLI -- `curl` (for GitHub API calls) +- `curl` (for GitHub/GitLab API calls) - `jq` (for JSON parsing) + +The local backup script also requires: + - `zip` (for creating backup archives) -- GitHub account with API access ### Installing Dependencies @@ -41,105 +43,142 @@ sudo yum install git curl jq zip --- -## πŸ” GitHub Authentication +## πŸ“¦ Script 1: Local Backup (`code-backup-local.sh`) -The script will attempt to automatically detect your GitHub username from: +Creates a local, zipped directory of all your non-archived GitHub repositories. -1. Git global configuration (`git config --global user.name`) -2. GitHub API (if authenticated) -3. Manual input prompt +### Features -For private repositories, you'll need to authenticate with GitHub. You can either: +- Fetches all non-archived repositories from your GitHub account +- Creates a `~/Projects` directory (or custom via `PROJECTS_DIR` env var) +- Clones new repositories or updates existing ones to their latest default branch +- Creates a timestamped zip backup: `~/Projects-Backup_YYYY-MM-DD_HH-MM-SS.zip` +- Excludes `.git` directories and system files from the zip archive -- Use a Personal Access Token (recommended) -- Set up SSH keys for Git operations +### πŸ” Authentication -### Using a Personal Access Token +The script supports both public and private repositories: -1. Go to GitHub Settings β†’ Developer settings β†’ Personal access tokens -2. Generate a new token with `repo` scope for full repository access -3. Set the token as an environment variable: +**For private repositories**, set a GitHub Personal Access Token: - ```bash - export GITHUB_TOKEN="your_token_here" - ``` +```bash +export GITHUB_TOKEN="your_token_here" +``` ---- +To create a token: -## πŸ“¦ Directory Structure +1. Go to GitHub Settings β†’ Developer settings β†’ Personal access tokens +2. Generate a new token with `repo` scope for full repository access -**Before running:** +**Optional environment variables:** -```text -~/Projects/ (created if doesn't exist) +```bash +export GITHUB_USERNAME="your-username" # Auto-detected if token provided +export PROJECTS_DIR="$HOME/MyProjects" # Default: ~/Projects +export USE_GITHUB_SSH="true" # Use SSH instead of HTTPS (default: false) ``` -**After running:** - -```text -~/Projects/ -β”œβ”€β”€ repo-a/ -β”œβ”€β”€ repo-b/ -β”œβ”€β”€ private-repo/ -└── … +### Usage -~/Projects-Backup_2025-01-15_14-30-25.zip +```bash +chmod +x code-backup-local.sh +./code-backup-local.sh ``` +### Output + +- **Local repositories**: `~/Projects/` (or `$PROJECTS_DIR`) +- **Backup archive**: `~/Projects-Backup_YYYY-MM-DD_HH-MM-SS.zip` +- **Logs**: `logs/code-backup-YYYYMMDD-HHMMSS.log` +- **Errors**: `logs/errors-YYYYMMDD-HHMMSS.log` + --- -## πŸš€ Usage +## πŸ“¦ Script 2: GitLab Mirror (`code-backup-gitlab.sh`) -1. Make the script executable: +Mirrors all non-archived public and private GitHub repositories to similarly named GitLab projects. + +### Features + +- Lists all non-archived GitHub repos you can access +- Creates/updates a local mirror clone (bare repo) for each +- Ensures a same-named GitLab project exists under your namespace +- Pushes a full mirror to GitLab (all branches, tags, and refs) +- Automatically creates GitLab projects if they don't exist (optional) + +### πŸ” Authentication + +**Required environment variables:** ```bash -chmod +x code-backup.sh +export GITLAB_TOKEN="your_gitlab_pat" # GitLab.com Personal Access Token +export GITLAB_NAMESPACE="your-username" # Your GitLab username or group ``` -2. Run the script: +**Optional environment variables:** ```bash -./code-backup.sh +export GITHUB_TOKEN="your_github_token" # For private GitHub repos +export GITHUB_USERNAME="your-username" # Auto-detected if token provided +export USE_GITHUB_SSH="true" # Use SSH for GitHub (default: false) +export AUTO_CREATE_GITLAB_PROJECTS="true" # Auto-create missing projects (default: true) +export GITLAB_VISIBILITY="private" # Visibility for new projects: private/internal/public (default: private) +export GITLAB_HOST="https://gitlab.com" # GitLab instance URL (default: gitlab.com) +export BACKUP_ROOT="$HOME/GitHub-GitLab-Backup" # Where to store local mirrors ``` -3. The script will: - - Check all dependencies - - Create necessary directories - - Fetch your GitHub username - - Download/update all repositories - - Create a timestamped backup zip file +### Creating GitLab Token ---- +1. Go to GitLab.com β†’ Settings β†’ Access Tokens +2. Create a token with `api` scope (and `write_repository` if needed) +3. Set it as `GITLAB_TOKEN` environment variable + +### Usage -## πŸ“Š Features +```bash +chmod +x code-backup-gitlab.sh -### βœ… What it does: +# Set required environment variables +export GITLAB_TOKEN="your_token" +export GITLAB_NAMESPACE="your-username" -- **Automatic Discovery**: Uses GitHub API to find all your repositories -- **Smart Updates**: Clones new repos, updates existing ones -- **Default Branch Detection**: Automatically detects and checks out the correct default branch -- **Comprehensive Logging**: Detailed logs with timestamps and error tracking -- **Error Handling**: Robust error handling with detailed error reporting -- **Progress Tracking**: Shows progress and summary statistics -- **Clean Backups**: Excludes `.git` directories and system files from backups +# Optional: for private GitHub repos +export GITHUB_TOKEN="your_github_token" -### πŸ”§ Advanced Features: +./code-backup-gitlab.sh +``` -- **Pagination Support**: Handles users with many repositories -- **Branch Management**: Automatically switches to default branch before updating -- **Log Management**: Separate log files for general output and errors -- **Dependency Checking**: Validates all required tools before starting -- **Cleanup**: Automatic cleanup of temporary files +### How It Works + +1. Fetches all non-archived repositories from GitHub +2. For each repository: + - Creates/updates a local bare mirror clone + - Checks if a GitLab project exists (creates it if `AUTO_CREATE_GITLAB_PROJECTS=true`) + - Pushes all branches, tags, and refs to GitLab as a mirror +3. Assumes GitHub and GitLab usernames are the same, and projects have the same name + +### Output + +- **Local mirrors**: `$BACKUP_ROOT/mirrors-YYYYMMDD-HHMMSS/` (bare repos) +- **Logs**: `logs/gh-gl-backup-YYYYMMDD-HHMMSS.log` +- **Errors**: `logs/gh-gl-errors-YYYYMMDD-HHMMSS.log` --- ## πŸ“ Logging -The script creates detailed logs in the `logs/` directory: +Both scripts create detailed logs in the `logs/` directory: + +**Local Backup:** - `code-backup-YYYYMMDD-HHMMSS.log` - General execution log - `errors-YYYYMMDD-HHMMSS.log` - Error-specific log +**GitLab Mirror:** + +- `gh-gl-backup-YYYYMMDD-HHMMSS.log` - General execution log +- `gh-gl-errors-YYYYMMDD-HHMMSS.log` - Error-specific log + Logs include: - Timestamped entries @@ -150,20 +189,9 @@ Logs include: --- -## βš™οΈ Configuration - -You can customize the script by modifying these variables at the top: - -```bash -readonly PROJECTS_DIR="$HOME/Projects" # Where to store repositories -readonly LOG_DIR="$SCRIPT_DIR/logs" # Where to store logs -``` - ---- - ## 🚨 Troubleshooting -### Common Issues: +### Common Issues 1. **"Missing required dependencies"** - Install missing tools using the commands above @@ -171,7 +199,7 @@ readonly LOG_DIR="$SCRIPT_DIR/logs" # Where to store logs 2. **"Failed to fetch repositories from GitHub API"** - Check your internet connection - Verify GitHub API access - - Consider using a Personal Access Token + - For private repos, ensure `GITHUB_TOKEN` is set correctly 3. **"Could not determine default branch"** - Repository might be empty or have no branches @@ -179,37 +207,95 @@ readonly LOG_DIR="$SCRIPT_DIR/logs" # Where to store logs 4. **"Failed to clone/update repository"** - Check repository permissions - - Verify SSH keys or authentication + - Verify SSH keys or authentication tokens - Check error log for specific details -### Getting Help: +5. **GitLab: "Could not find GitLab namespace"** + - Verify `GITLAB_NAMESPACE` is set correctly + - Ensure your GitLab token has proper permissions + - Check that the namespace exists (username or group) + +6. **GitLab: "Failed to push mirror"** + - Verify `GITLAB_TOKEN` has `write_repository` scope + - Check that the GitLab project exists or auto-create is enabled + - Review error log for specific GitLab API errors + +### Getting Help -Check the error log file for detailed error messages: +Check the error log files for detailed error messages: ```bash +# Local backup errors cat logs/errors-*.log + +# GitLab mirror errors +cat logs/gh-gl-errors-*.log ``` --- ## πŸ”„ Automation -To run this script automatically, you can set up a cron job: +To run these scripts automatically, you can set up cron jobs: ```bash # Edit crontab crontab -e -# Add entry to run daily at 2 AM -0 2 * * * /path/to/code-backup.sh +# Run local backup daily at 2 AM +0 2 * * * /path/to/code-backup-local.sh + +# Run GitLab mirror weekly on Sundays at 3 AM +0 3 * * 0 /path/to/code-backup-gitlab.sh +``` + +**Note:** When using cron, make sure to set environment variables in your crontab or in a script that sources them: + +```bash +# In crontab +0 2 * * * source ~/.bashrc && /path/to/code-backup-gitlab.sh ``` --- -## πŸ“‚ Related +## πŸ“‚ Notes + +### Repository Filtering + +Both scripts only process **non-archived** repositories. Archived repositories are automatically excluded. + +### Private Repositories + +- **Local backup**: Requires `GITHUB_TOKEN` to access private repos +- **GitLab mirror**: Requires both `GITHUB_TOKEN` (for GitHub) and `GITLAB_TOKEN` (for GitLab) + +### SSH vs HTTPS + +Both scripts support both SSH and HTTPS for GitHub operations: + +- Set `USE_GITHUB_SSH="true"` to use SSH (requires SSH keys configured) +- Default is HTTPS with token authentication + +### GitLab Project Creation + +The GitLab mirror script can automatically create GitLab projects if they don't exist: + +- Set `AUTO_CREATE_GITLAB_PROJECTS="true"` (default) +- New projects will be created with visibility set by `GITLAB_VISIBILITY` (default: `private`) + +### Submodules For repositories with submodules, ensure they're properly initialized: ```bash git submodule update --init --recursive ``` + +--- + +## πŸ”— Related + +- [GitHub API Documentation](https://docs.github.com/en/rest) +- [GitLab API Documentation](https://docs.gitlab.com/ee/api/) +- [Git Personal Access Tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) +- [GitLab Personal Access Tokens](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) diff --git a/code-backup/code-backup-gitlab.sh b/code-backup/code-backup-gitlab.sh new file mode 100644 index 0000000..74470c5 --- /dev/null +++ b/code-backup/code-backup-gitlab.sh @@ -0,0 +1,376 @@ +#!/usr/bin/env bash +# GitHub -> GitLab.com Backup Script (mirror push) +# - Lists all non-archived GitHub repos you can access +# - Creates/updates a local mirror clone for each +# - Ensures a same-named GitLab project exists under your namespace +# - Pushes a full mirror to GitLab (all branches/tags/refs) + +set -euo pipefail + +# ---------------------------- +# Configuration +# ---------------------------- +readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +readonly LOG_DIR="$SCRIPT_DIR/logs" +mkdir -p "$LOG_DIR" + +readonly RUN_TS="$(date +%Y%m%d-%H%M%S)" +readonly LOG_FILE="$LOG_DIR/gh-gl-backup-$RUN_TS.log" +readonly ERROR_LOG="$LOG_DIR/gh-gl-errors-$RUN_TS.log" + +# Where to store local mirror clones (bare repos) +readonly BACKUP_ROOT="${BACKUP_ROOT:-$HOME/GitHub-GitLab-Backup}" +readonly MIRRORS_DIR="$BACKUP_ROOT/mirrors-$RUN_TS" +mkdir -p "$MIRRORS_DIR" + +# Required: +: "${GITLAB_TOKEN:?Set GITLAB_TOKEN (GitLab.com PAT) in env}" +: "${GITLAB_NAMESPACE:?Set GITLAB_NAMESPACE (your GitLab username or group full path) in env}" + +# Optional: +# If set, script can access private GitHub repos you can see. +# Export a GitHub classic PAT with repo read access (or fine-grained equivalent). +GITHUB_TOKEN="${GITHUB_TOKEN:-}" + +# If empty, we'll try to auto-detect your GitHub username from /user (needs GITHUB_TOKEN), +# otherwise we’ll prompt. +GITHUB_USERNAME="${GITHUB_USERNAME:-}" + +# Prefer SSH clone from GitHub? (requires your SSH keys set up for GitHub) +# If false, use HTTPS with token injection if GITHUB_TOKEN is set. +USE_GITHUB_SSH="${USE_GITHUB_SSH:-false}" + +# Create GitLab projects automatically if missing +AUTO_CREATE_GITLAB_PROJECTS="${AUTO_CREATE_GITLAB_PROJECTS:-true}" + +# Default visibility for newly created GitLab projects: private/internal/public +GITLAB_VISIBILITY="${GITLAB_VISIBILITY:-private}" + +# GitLab host (you said GitLab.com) +GITLAB_HOST="${GITLAB_HOST:-https://gitlab.com}" +GITLAB_API="$GITLAB_HOST/api/v4" + +# ---------------------------- +# Pretty logging +# ---------------------------- +readonly RED='\033[0;31m' +readonly GREEN='\033[0;32m' +readonly YELLOW='\033[1;33m' +readonly BLUE='\033[0;34m' +readonly NC='\033[0m' + +log() { + local level="$1"; shift + local msg="$*" + local ts; ts="$(date '+%Y-%m-%d %H:%M:%S')" + echo -e "${ts} [${level}] ${msg}" | tee -a "$LOG_FILE" +} + +log_info() { log "INFO" "${BLUE}$*${NC}"; } +log_success() { log "SUCCESS" "${GREEN}$*${NC}"; } +log_warn() { log "WARN" "${YELLOW}$*${NC}"; } +log_error() { + local ts; ts="$(date '+%Y-%m-%d %H:%M:%S')" + log "ERROR" "${RED}$*${NC}" + echo -e "${ts} [ERROR] $*" >> "$ERROR_LOG" +} + +error_exit() { + log_error "Fatal: $1" + exit 1 +} + +# ---------------------------- +# Dependencies +# ---------------------------- +check_dependencies() { + log_info "Checking dependencies..." + local missing=() + for cmd in git curl jq; do + command -v "$cmd" >/dev/null 2>&1 || missing+=("$cmd") + done + if [ ${#missing[@]} -ne 0 ]; then + error_exit "Missing dependencies: ${missing[*]} (install and retry)" + fi + log_success "All dependencies found" +} + +# ---------------------------- +# GitHub: determine username +# ---------------------------- +get_github_username() { + if [ -n "${GITHUB_USERNAME:-}" ]; then + log_success "Using GitHub username from env: $GITHUB_USERNAME" + return 0 + fi + + if [ -n "${GITHUB_TOKEN:-}" ]; then + log_info "Detecting GitHub username via API (/user)..." + local resp + resp="$(curl -sS -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user 2>>"$ERROR_LOG" || true)" + local login + login="$(echo "$resp" | jq -r '.login // empty' 2>>"$ERROR_LOG" || true)" + if [ -n "$login" ]; then + GITHUB_USERNAME="$login" + log_success "Detected GitHub username: $GITHUB_USERNAME" + return 0 + fi + log_warn "Could not detect GitHub username from token; will prompt." + fi + + read -r -p "Enter your GitHub username: " GITHUB_USERNAME + [ -n "$GITHUB_USERNAME" ] || error_exit "GitHub username is required" + log_success "Using GitHub username: $GITHUB_USERNAME" +} + +# ---------------------------- +# GitHub: list repos (non-archived) +# ---------------------------- +# Returns lines: "/\t" +get_github_repos() { + log_info "Fetching GitHub repos (excluding archived) for: $GITHUB_USERNAME" + + local page=1 + local per_page=100 + + while true; do + local url + local resp + + if [ -n "${GITHUB_TOKEN:-}" ]; then + # Authenticated: includes private repos you can access + url="https://api.github.com/user/repos?page=$page&per_page=$per_page&type=all&sort=updated" + resp="$(curl -sS -H "Authorization: token $GITHUB_TOKEN" "$url" 2>>"$ERROR_LOG" || true)" + else + # Unauthenticated: only public repos + url="https://api.github.com/users/$GITHUB_USERNAME/repos?page=$page&per_page=$per_page&type=all&sort=updated" + resp="$(curl -sS "$url" 2>>"$ERROR_LOG" || true)" + fi + + # Error? + if echo "$resp" | jq -e '.message? // empty' >/dev/null 2>&1; then + local msg; msg="$(echo "$resp" | jq -r '.message' 2>>"$ERROR_LOG" || echo "unknown")" + error_exit "GitHub API error: $msg" + fi + + # Choose clone URL style + local jq_clone_field + if [ "$USE_GITHUB_SSH" = "true" ]; then + jq_clone_field='.ssh_url' + else + jq_clone_field='.clone_url' + fi + + # Emit owner/name + url, excluding archived + local lines + lines="$(echo "$resp" | jq -r --argjson _ 0 \ + ".[] | select(.archived == false) | (.full_name + \"\t\" + ${jq_clone_field})" 2>>"$ERROR_LOG" || true)" + + [ -n "$lines" ] || break + + # Print for caller + echo "$lines" + + # Last page? + local count + count="$(echo "$lines" | wc -l | tr -d ' ')" + if [ "$count" -lt "$per_page" ]; then + break + fi + + page=$((page + 1)) + done +} + +# ---------------------------- +# GitLab helpers +# ---------------------------- +urlencode() { + # Minimal urlencode for path_with_namespace usage (spaces unlikely) + # Encodes: / -> %2F + echo -n "$1" | sed 's/%/%25/g; s/\//%2F/g; s/#/%23/g; s/\?/%3F/g; s/&/%26/g; s/ /%20/g' +} + +gitlab_api_get() { + local path="$1" + curl -sS --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$GITLAB_API$path" 2>>"$ERROR_LOG" || true +} + +gitlab_api_post_json() { + local path="$1" + local json="$2" + curl -sS --request POST \ + --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ + --header "Content-Type: application/json" \ + --data "$json" \ + "$GITLAB_API$path" 2>>"$ERROR_LOG" || true +} + +get_gitlab_namespace_id() { + # Find namespace id by searching and matching full_path exactly. + # Works for user namespaces and groups. + local search="$GITLAB_NAMESPACE" + local resp + resp="$(gitlab_api_get "/namespaces?search=$(urlencode "$search")")" + local ns_id + ns_id="$(echo "$resp" | jq -r --arg fp "$GITLAB_NAMESPACE" '.[] | select(.full_path==$fp) | .id' | head -n1)" + [ -n "$ns_id" ] || return 1 + echo "$ns_id" +} + +gitlab_project_exists() { + local path_with_ns="$1" # e.g. mygroup/myrepo + local enc; enc="$(urlencode "$path_with_ns")" + local resp + resp="$(gitlab_api_get "/projects/$enc")" + # If exists, response has an "id" + echo "$resp" | jq -e '.id? != null' >/dev/null 2>&1 +} + +create_gitlab_project() { + local repo_name="$1" # just "myrepo" + local ns_id="$2" + + local payload + payload="$(jq -n \ + --arg name "$repo_name" \ + --argjson namespace_id "$ns_id" \ + --arg visibility "$GITLAB_VISIBILITY" \ + '{name:$name, namespace_id:$namespace_id, visibility:$visibility}')" + + local resp + resp="$(gitlab_api_post_json "/projects" "$payload")" + + if echo "$resp" | jq -e '.id? != null' >/dev/null 2>&1; then + return 0 + fi + + local msg + msg="$(echo "$resp" | jq -r '.message? // .error? // empty' 2>/dev/null || true)" + log_error "GitLab project creation failed for $repo_name: ${msg:-unknown error}" + return 1 +} + +gitlab_remote_url() { + # Use GitLab's HTTPS token auth. + # GitLab PAT can be used as the password over HTTPS. [oai_citation:3‑GitLab Docs](https://docs.gitlab.com/user/profile/personal_access_tokens/?utm_source=chatgpt.com) + # The conventional form for GitLab is username "oauth2" with token as password. + local path_with_ns="$1" + echo "https://oauth2:${GITLAB_TOKEN}@gitlab.com/${path_with_ns}.git" +} + +# ---------------------------- +# Mirror clone/update and push +# ---------------------------- +process_repo() { + local full_name="$1" # owner/name on GitHub + local clone_url="$2" # GitHub clone url + local repo_name + repo_name="$(basename "$full_name")" + + local local_path="$MIRRORS_DIR/${repo_name}.git" + local gl_path="${GITLAB_NAMESPACE}/${repo_name}" + + log_info "Repo: $full_name -> GitLab: $gl_path" + + # If using HTTPS to GitHub and token exists, inject it (so private clones work non-interactively) + local effective_clone_url="$clone_url" + if [ "$USE_GITHUB_SSH" != "true" ] && [ -n "${GITHUB_TOKEN:-}" ]; then + # GitHub supports token auth via x-access-token username. + effective_clone_url="$(echo "$clone_url" | sed "s#https://#https://x-access-token:${GITHUB_TOKEN}@#")" + fi + + # Clone/update local mirror + if [ -d "$local_path" ]; then + log_info "Updating local mirror: $repo_name" + if ! git -C "$local_path" remote update --prune 2>>"$ERROR_LOG"; then + log_error "Failed to update mirror for $repo_name" + return 1 + fi + else + log_info "Cloning local mirror: $repo_name" + if ! git clone --mirror "$effective_clone_url" "$local_path" 2>>"$ERROR_LOG"; then + log_error "Failed to mirror-clone $repo_name" + return 1 + fi + fi + + # Ensure GitLab project exists (optional auto-create) + if gitlab_project_exists "$gl_path"; then + log_info "GitLab project exists: $gl_path" + else + if [ "$AUTO_CREATE_GITLAB_PROJECTS" = "true" ]; then + log_warn "GitLab project missing, creating: $gl_path" + local ns_id + ns_id="$(get_gitlab_namespace_id)" || { + log_error "Could not resolve GitLab namespace id for: $GITLAB_NAMESPACE" + return 1 + } + create_gitlab_project "$repo_name" "$ns_id" || return 1 + log_success "Created GitLab project: $gl_path" + else + log_warn "GitLab project missing and AUTO_CREATE_GITLAB_PROJECTS=false; skipping push: $repo_name" + return 0 + fi + fi + + # Push mirror to GitLab + local gl_url + gl_url="$(gitlab_remote_url "$gl_path")" + + # Avoid printing token-bearing URL + log_info "Pushing mirror to GitLab (all refs): $gl_path" + if ! git -C "$local_path" push --mirror "$gl_url" 2>>"$ERROR_LOG"; then + log_error "Failed to push mirror to GitLab for $repo_name" + return 1 + fi + + log_success "Backed up to GitLab: $gl_path" + return 0 +} + +# ---------------------------- +# Main +# ---------------------------- +main() { + log_info "Starting GitHub -> GitLab backup" + log_info "Local mirrors dir: $MIRRORS_DIR" + log_info "Log: $LOG_FILE" + log_info "Errors: $ERROR_LOG" + + check_dependencies + get_github_username + + local ns_id="" + if [ "$AUTO_CREATE_GITLAB_PROJECTS" = "true" ]; then + ns_id="$(get_gitlab_namespace_id)" || error_exit "Could not find GitLab namespace: $GITLAB_NAMESPACE" + log_success "Resolved GitLab namespace id: $ns_id" + fi + + local total=0 ok=0 fail=0 + + # Stream repos line-by-line + while IFS=$'\t' read -r full_name clone_url; do + [ -n "${full_name:-}" ] || continue + [ -n "${clone_url:-}" ] || continue + total=$((total + 1)) + + if process_repo "$full_name" "$clone_url"; then + ok=$((ok + 1)) + else + fail=$((fail + 1)) + fi + done < <(get_github_repos) + + log_success "Done." + log_info "Total repos processed: $total" + log_info "Successful: $ok" + log_info "Failed: $fail" + + if [ "$fail" -gt 0 ]; then + log_warn "Some repos failed. See: $ERROR_LOG" + exit 1 + fi +} + +main "$@" diff --git a/code-backup/code-backup.sh b/code-backup/code-backup-local.sh similarity index 59% rename from code-backup/code-backup.sh rename to code-backup/code-backup-local.sh index 6c591d0..0ee96e8 100755 --- a/code-backup/code-backup.sh +++ b/code-backup/code-backup-local.sh @@ -1,15 +1,26 @@ -#!/bin/bash - -# GitHub Projects Backup Script -# Backs up all GitHub repositories for a user by cloning/updating them and creating a timestamped zip backup +#!/usr/bin/env bash +# GitHub Projects Local Backup Script +# - Lists all non-archived GitHub repos you can access +# - Clones/updates them locally +# - Creates a timestamped zip backup set -euo pipefail # Configuration readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly LOG_DIR="$SCRIPT_DIR/logs" -readonly LOG_FILE="$LOG_DIR/code-backup-$(date +%Y%m%d-%H%M%S).log" -readonly ERROR_LOG="$LOG_DIR/errors-$(date +%Y%m%d-%H%M%S).log" +readonly RUN_TS="$(date +%Y%m%d-%H%M%S)" +readonly LOG_FILE="$LOG_DIR/code-backup-$RUN_TS.log" +readonly ERROR_LOG="$LOG_DIR/errors-$RUN_TS.log" + +# Optional: GitHub token for private repos +GITHUB_TOKEN="${GITHUB_TOKEN:-}" + +# Optional: GitHub username (auto-detected if token provided) +GITHUB_USERNAME="${GITHUB_USERNAME:-}" + +# Prefer SSH clone from GitHub? (requires your SSH keys set up for GitHub) +USE_GITHUB_SSH="${USE_GITHUB_SSH:-false}" # Backup directory will be created with date format readonly BACKUP_DATE=$(date +%m-%d-%y) @@ -25,7 +36,6 @@ readonly BLUE='\033[0;34m' readonly NC='\033[0m' # No Color # Global variables -GITHUB_USERNAME="" TOTAL_REPOS=0 SUCCESSFUL_REPOS=0 FAILED_REPOS=0 @@ -103,62 +113,57 @@ check_dependencies() { # Get GitHub username get_github_username() { - log_info "Getting GitHub username..." - - # Try to get from git remote if available (most reliable) - GITHUB_USERNAME=$(git config --get remote.origin.url 2>/dev/null | sed -n 's/.*github\.com[:/]\([^/]*\).*/\1/p' || echo "") - - if [ -z "$GITHUB_USERNAME" ]; then - # Try to get from GitHub API (requires authentication) - GITHUB_USERNAME=$(curl -s https://api.github.com/user 2>/dev/null | jq -r '.login' 2>/dev/null || echo "") + if [ -n "${GITHUB_USERNAME:-}" ]; then + log_success "Using GitHub username from env: $GITHUB_USERNAME" + return 0 fi - if [ -z "$GITHUB_USERNAME" ] || [ "$GITHUB_USERNAME" = "null" ]; then - read -p "Please enter your GitHub username: " GITHUB_USERNAME - if [ -z "$GITHUB_USERNAME" ]; then - error_exit "GitHub username is required" + if [ -n "${GITHUB_TOKEN:-}" ]; then + log_info "Detecting GitHub username via API (/user)..." + local resp + resp="$(curl -sS -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user 2>>"$ERROR_LOG" || true)" + local login + login="$(echo "$resp" | jq -r '.login // empty' 2>>"$ERROR_LOG" || true)" + if [ -n "$login" ] && [ "$login" != "null" ]; then + GITHUB_USERNAME="$login" + log_success "Detected GitHub username: $GITHUB_USERNAME" + return 0 fi + log_warning "Could not detect GitHub username from token; will prompt." fi + read -r -p "Enter your GitHub username: " GITHUB_USERNAME + [ -n "$GITHUB_USERNAME" ] || error_exit "GitHub username is required" log_success "Using GitHub username: $GITHUB_USERNAME" } # Create necessary directories setup_directories() { - log_info "Setting up directories..." - - # Create backup directory (dated) - if [ -d "$BACKUP_DIR" ]; then - log_warning "Backup directory already exists: $BACKUP_DIR" - log_info "Removing existing backup directory..." - rm -rf "$BACKUP_DIR" || { - error_exit "Failed to remove existing backup directory" - } - fi - - log_info "Creating backup directory: $BACKUP_DIR" - mkdir -p "$BACKUP_DIR" || { - error_exit "Failed to create backup directory" + # Create log directory first (needed for logging) + mkdir -p "$LOG_DIR" || { + echo "Error: Failed to create log directory: $LOG_DIR" >&2 + exit 1 } - # Create log directory - if [ ! -d "$LOG_DIR" ]; then - log_info "Creating log directory: $LOG_DIR" - mkdir -p "$LOG_DIR" || { - error_exit "Failed to create log directory" + # Create Projects directory + if [ ! -d "$PROJECTS_DIR" ]; then + log_info "Creating Projects directory: $PROJECTS_DIR" + mkdir -p "$PROJECTS_DIR" || { + log_error "Failed to create Projects directory: $PROJECTS_DIR" + exit 1 } fi log_success "Directories set up successfully" } -# Get all GitHub repositories +# Get all GitHub repositories (non-archived only) +# Returns lines: "" get_github_repos() { - log_info "Fetching GitHub repositories for user: $GITHUB_USERNAME" >&2 + log_info "Fetching GitHub repos (excluding archived) for: $GITHUB_USERNAME" local page=1 local per_page=100 - local all_repos=() # Check for GitHub token for private repos if [ -n "${GITHUB_TOKEN:-}" ]; then @@ -166,66 +171,52 @@ get_github_repos() { fi while true; do - log_info "Fetching page $page..." >&2 - - # Use /user/repos endpoint for authenticated access to private repos - # Fall back to /users/$GITHUB_USERNAME/repos if no token local url + local resp + if [ -n "${GITHUB_TOKEN:-}" ]; then + # Authenticated: includes private repos you can access url="https://api.github.com/user/repos?page=$page&per_page=$per_page&type=all&sort=updated" + resp="$(curl -sS -H "Authorization: token $GITHUB_TOKEN" "$url" 2>>"$ERROR_LOG" || true)" else + # Unauthenticated: only public repos url="https://api.github.com/users/$GITHUB_USERNAME/repos?page=$page&per_page=$per_page&type=all&sort=updated" + resp="$(curl -sS "$url" 2>>"$ERROR_LOG" || true)" fi - local response - local repos_json - - # Make API call with authentication if token is available - if [ -n "${GITHUB_TOKEN:-}" ]; then - if ! response=$(curl -s -H "Authorization: token $GITHUB_TOKEN" "$url" 2>/dev/null); then - error_exit "Failed to fetch repositories from GitHub API" - fi - else - if ! response=$(curl -s "$url" 2>/dev/null); then - error_exit "Failed to fetch repositories from GitHub API" - fi + # Error? + if echo "$resp" | jq -e '.message? // empty' >/dev/null 2>&1; then + local msg; msg="$(echo "$resp" | jq -r '.message' 2>>"$ERROR_LOG" || echo "unknown")" + error_exit "GitHub API error: $msg" fi - # Check if we got an error response - if echo "$response" | jq -e '.message' >/dev/null 2>&1; then - local error_msg=$(echo "$response" | jq -r '.message') - error_exit "GitHub API error: $error_msg" + # Choose clone URL style + local jq_clone_field + if [ "$USE_GITHUB_SSH" = "true" ]; then + jq_clone_field='.ssh_url' + else + jq_clone_field='.clone_url' fi - # Get repository data, filtering out archived repos - if ! repos_json=$(echo "$response" | jq -r '.[] | select(.archived == false) | .clone_url' 2>/dev/null); then - error_exit "Failed to parse repository data from GitHub API" - fi + # Emit clone URLs, excluding archived + local lines + lines="$(echo "$resp" | jq -r --argjson _ 0 \ + ".[] | select(.archived == false) | ${jq_clone_field}" 2>>"$ERROR_LOG" || true)" - if [ -z "$repos_json" ] || [ "$repos_json" = "null" ]; then - break - fi + [ -n "$lines" ] || break - while IFS= read -r repo_url; do - if [ -n "$repo_url" ] && [ "$repo_url" != "null" ]; then - all_repos+=("$repo_url") - fi - done <<< "$repos_json" + # Print for caller + echo "$lines" - # Check if we got fewer repos than requested (last page) - local repo_count=$(echo "$repos_json" | grep -c . || echo "0") - if [ "$repo_count" -lt "$per_page" ]; then + # Last page? + local count + count="$(echo "$lines" | wc -l | tr -d ' ')" + if [ "$count" -lt "$per_page" ]; then break fi - ((page++)) + page=$((page + 1)) done - - TOTAL_REPOS=${#all_repos[@]} - log_success "Found $TOTAL_REPOS repositories (excluding archived)" >&2 - - # Return the array to stdout only - printf '%s\n' "${all_repos[@]}" } # Get default branch for a repository @@ -286,7 +277,14 @@ clone_repository() { local repo_name="$3" local original_dir=$(pwd) - if git clone "$repo_url" "$repo_path" 2>>"$ERROR_LOG"; then + # If using HTTPS and token exists, inject it (so private clones work non-interactively) + local effective_clone_url="$repo_url" + if [ "$USE_GITHUB_SSH" != "true" ] && [ -n "${GITHUB_TOKEN:-}" ]; then + # GitHub supports token auth via x-access-token username. + effective_clone_url="$(echo "$repo_url" | sed "s#https://#https://x-access-token:${GITHUB_TOKEN}@#")" + fi + + if git clone "$effective_clone_url" "$repo_path" 2>>"$ERROR_LOG"; then log_success "Successfully cloned: $repo_name" ((SUCCESSFUL_REPOS++)) @@ -390,86 +388,49 @@ main() { # Setup directories first before any logging setup_directories - log_info "Starting GitHub Projects Backup Script" + log_info "Starting GitHub Projects Local Backup" + log_info "Projects directory: $PROJECTS_DIR" log_info "Log file: $LOG_FILE" log_info "Error log: $ERROR_LOG" - # Continue with other setup check_dependencies get_github_username - # Get all repositories - local repos=() - local temp_file=$(mktemp) - # get_github_repos outputs logs to stderr and repos to stdout - get_github_repos > "$temp_file" + local total=0 ok=0 fail=0 - # Read repos from temp file + # Stream repos line-by-line while IFS= read -r repo_url; do - if [ -n "$repo_url" ] && [[ "$repo_url" =~ ^https:// ]]; then - repos+=("$repo_url") + [ -n "${repo_url:-}" ] || continue + total=$((total + 1)) + + if process_repository "$repo_url"; then + ok=$((ok + 1)) + else + fail=$((fail + 1)) fi - done < "$temp_file" - rm -f "$temp_file" + done < <(get_github_repos) - log_info "Loaded ${#repos[@]} repository URLs into array" + TOTAL_REPOS=$total + SUCCESSFUL_REPOS=$ok + FAILED_REPOS=$fail - if [ ${#repos[@]} -eq 0 ]; then + if [ "$total" -eq 0 ]; then log_warning "No repositories found" exit 0 fi - # Debug: Show first few repos - if [ ${#repos[@]} -gt 0 ]; then - log_info "First repository URL: ${repos[0]}" - fi - - # Process each repository - local total_repos=${#repos[@]} - log_info "Starting to process $total_repos repositories..." - local repo_count=0 - - # Temporarily disable ALL strict error handling for the loop - set +e - set +u - set +o pipefail - - local i=0 - log_info "Entering repository processing loop (i=$i, total=$total_repos)..." - while [ $i -lt ${#repos[@]} ]; do - local repo_url="${repos[$i]}" - if [ -z "$repo_url" ]; then - log_warning "Skipping empty repository URL at index $i" - i=$((i + 1)) - continue - fi - repo_count=$((repo_count + 1)) - local repo_name=$(basename "$repo_url" .git 2>/dev/null || echo "unknown") - log_info "Processing repository $repo_count of ${#repos[@]}: $repo_name" - if ! process_repository "$repo_url"; then - log_warning "Repository processing failed for: $repo_name" - fi - i=$((i + 1)) - done - - # Re-enable strict error handling - set -e - set -u - set -o pipefail - - log_info "Finished processing loop. Processed $repo_count repositories." - # Create backup create_backup # Summary log_success "Backup process completed!" - log_info "Total repositories: $TOTAL_REPOS" - log_info "Successful: $SUCCESSFUL_REPOS" - log_info "Failed: $FAILED_REPOS" + log_info "Total repositories: $total" + log_info "Successful: $ok" + log_info "Failed: $fail" - if [ $FAILED_REPOS -gt 0 ]; then + if [ "$fail" -gt 0 ]; then log_warning "Some repositories failed to process. Check error log: $ERROR_LOG" + exit 1 fi }