Sudoers as Code: CI/CD Approach to Privileged Access

Introduction

Managing privileged access across a fleet of Linux systems can be a daunting task. Traditionally, updating the sudoers file has been a manual process—prone to human error, inconsistent across environments, and often lacking proper version control or audit trails. This approach simply doesn’t scale in a modern infrastructure where security, traceability, and speed are paramount.

To solve this, I designed and implemented a secure, automated workflow to manage sudoers policies using a CI/CD pipeline integrated with HashiCorp Vault. This pipeline pulls system credentials securely, compares existing configurations with source-controlled definitions, and updates only what's missing—without requiring interactive SSH sessions.

The result is a robust, versioned, and auditable process that eliminates the guesswork from managing elevated permissions, ensuring our systems remain compliant and consistent.

In this post, we’ll walk through the motivation, design, implementation, and outcomes of our solution—sharing lessons learned and future improvements along the way.

I wanted to start taking some of the configurable security configurations in our Linux Environment and try to turn them into zero-tough solutions that can be used to take out some of the daya to day drift that happens. This will be the first of a few in this line of posts.

Background & Motivation

Our environment consists of numerous Linux systems that support mission-critical workloads. Each system may require slightly different privilege rules based on its role, but those rules must be consistently deployed and easy to audit. The problem we faced was clear:

Manual edits to /etc/sudoers were not scalable
Copy-paste deployments lacked traceability
Onboarding/offboarding engineers required error-prone touchpoints
Vault was already in use for secret management, but not leveraged fully in this context

We knew there had to be a better way to:

Treat sudo privileges as code
Securely store and retrieve per-system credentials
Automate and audit every change
Ensure consistent configuration without overstepping

The goal was to build a solution that could be:

Modular: so teams could define their own sudoers templates
Secure: no plaintext passwords or shell prompt risks
Intelligent: only update what’s missing—don’t overwrite
Hands-off: no need to log into each box or manage keys manually

By integrating Vault, GitLab CI/CD, and some smart scripting, we built a pipeline that met all of these needs.

Architecture Overview

The backbone of our automated sudoers deployment is a simple, scalable CI/CD pipeline driven by GitLab and powered by secure credential handling through HashiCorp Vault.

This section breaks down the major components involved and how they interact to ensure sudoers configurations are safely and consistently updated across our Linux infrastructure.

🔧 Key Components

GitLab Repository:
Houses our version-controlled sudoers configuration files (e.g., 00-operational-accounts). These are reviewed and approved through merge requests.
GitLab CI/CD Pipeline:
Orchestrates the workflow by:
- Fetching secrets securely from Vault
- Connecting via SSH to target systems (non-interactive)
- Comparing current /etc/sudoers to the source-controlled version
- Appending any missing lines—no overwrites
Vault:
Stores per-host credentials. Accessed via AppRole or token auth depending on the stage environment. Secrets are pulled just-in-time and never stored in CI logs.
Linux Hosts:
The destination systems where sudo rules are managed. These are accessed via SSH using Vault-sourced credentials. The script logic ensures only required changes are applied.
Logs & Audit Trail:
All pipeline actions (success, failure, skipped updates) are logged. Vault access logs and GitLab job logs provide traceability for all changes.

🖼️ Diagram of the Flow

Here’s a high-level overview of how the system works from Git commit to system update:

Implementation Details

At the heart of this system is a GitLab CI/CD pipeline that securely pulls per-host credentials from Vault and updates the /etc/sudoers file only when necessary. This ensures idempotent, traceable privilege management with zero manual intervention.

Here’s how we structured our repository, authentication flow, and SSH update logic.

📁 Repo Layout

The repository is structured in a way that aligns sudoers configurations to specific environments, teams, or host groups. Each system (or group of systems) has its own file containing rules to append if not already present.

/sudoers-pipeline/
├── environments/
│   ├── prod/
│   │   ├── host01
│   │   └── host02
│   ├── dev/
│   │   └── host-dev01
│   └── shared/
│       └── 00-operational-accounts
├── scripts/
│   └── update_sudoers.sh
└── .gitlab-ci.yml

🔐 Vault Integration

The pipeline uses HashiCorp Vault to dynamically pull credentials just-in-time. We use an AppRole auth method, which lets the GitLab runner authenticate securely without embedding long-lived tokens.

Example: Fetching Vault credentials

# Authenticate using AppRole
VAULT_TOKEN=$(vault write -field=token auth/approle/login \
    role_id="$ROLE_ID" \
    secret_id="$SECRET_ID")

# Pull system password
export HOST_PASS=$(vault kv get -field=password secret/linux/host01)

🚀 CI/CD SSH Logic

To connect to target systems, the CI/CD pipeline invokes a shell script (update_sudoers.sh) that:

Disables shell prompts and banners to ensure non-interactive SSH
Pulls the remote /etc/sudoers file via sudo cat
Compares it to the desired version using grep or diff
Appends any missing lines safely using a temporary file and visudo -c

# Example snippet (simplified)
REMOTE_LINES=$(ssh -o BatchMode=yes user@$HOST "sudo cat /etc/sudoers")
LOCAL_LINES=$(cat ./environments/prod/host01)

for line in $LOCAL_LINES; do
    if ! grep -qF "$line" <<< "$REMOTE_LINES"; then
        echo "$line" | ssh user@$HOST "sudo tee -a /etc/sudoers.d/temp-sudoers"
    fi
done

ssh user@$HOST "sudo visudo -c -f /etc/sudoers.d/temp-sudoers"

🧪 Validation & Fail-Safes

Validation: Every file is tested with visudo -c before being committed or deployed.
Dry Run Mode: A toggle allows team members to simulate the updates and view diff output without applying changes.
Error Handling: SSH and Vault errors are logged in the CI pipeline and sent to our alerting system for review.

CI/CD Pipeline Flow

The automation process is powered by a GitLab CI/CD pipeline that executes in clearly defined stages—ensuring safety, traceability, and modular control over how and when sudoers configurations are updated.

Each job in the pipeline is responsible for a distinct step, and secret handling is isolated from logic execution to minimize exposure.

🧬 Pipeline Stages

We structured our .gitlab-ci.yml file into the following stages:

Preparation
- Authenticate to Vault
- Retrieve target host credentials
- Validate that the right configuration files exist
Validation
- Run visudo -c on each sudoers config locally to catch syntax errors early
Deployment
- SSH into each host non-interactively
- Compare live /etc/sudoers content with version-controlled inputs
- Append only missing lines
- Re-validate on the remote side with visudo -c
Reporting
- Log job success/failure
- Optional: Notify team via Slack, email, or internal dashboard

🧾 Sample GitLab CI Job Definition

stages:
  - prepare
  - validate
  - deploy
  - report

variables:
  VAULT_ADDR: https://vault.internal
  ROLE_ID: $VAULT_ROLE_ID
  SECRET_ID: $VAULT_SECRET_ID

prepare-vault-auth:
  stage: prepare
  script:
    - echo "Authenticating with Vault..."
    - VAULT_TOKEN=$(vault write -field=token auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID)
    - export VAULT_TOKEN

validate-sudoers:
  stage: validate
  script:
    - for file in environments/prod/*; do visudo -c -f "$file"; done

deploy-to-hosts:
  stage: deploy
  script:
    - bash scripts/update_sudoers.sh environments/prod

report-status:
  stage: report
  script:
    - echo "All updates complete."

Challenges & Lessons Learned

No automation is complete without some trial and error. Along the way, we encountered a few key challenges:

Shell Banner Interference
Many systems printed login banners or shell prompt scripts, breaking non-interactive SSH sessions. We resolved this by forcing BatchMode and disabling MOTD and shell output in .bashrc.
sudoers Validation Risks
Mistakes in a sudoers file could lock us out. Running visudo -c pre- and post-deployment was essential.
Secret Sprawl Prevention
Secrets needed to be isolated to each job and never written to disk. GitLab’s masked environment variables and Vault's TTL limits helped prevent leaks.
Idempotency Logic
Ensuring we only appended missing lines took time to get right. Using grep -F and diff logic helped avoid duplication and system bloat.

Security Considerations

Security was a priority at every stage:

Secrets were never stored in plaintext or left behind in job logs
Vault tokens used short TTLs and ephemeral auth (AppRole) to reduce risk
Only protected branches can trigger the pipeline
CI logs were scrubbed to redact command output that may leak sensitive data
Change requests and reviews ensured no one pushed dangerous sudo rules unnoticed

You can also enhance security further by enabling Just-In-Time SSH access or integrating Vault’s SSH CA features.

Outcome & Benefits

This approach gave our team significant advantages:

Speed: New permissions could be rolled out in minutes, not hours
Auditability: Every change is logged, traceable, and stored in version control
Safety: visudo checks and minimal diffs reduced breakage risk
Scalability: Whether 5 systems or 500, the same code handles it all
Compliance: Aligns with STIG and organizational policy enforcement

Future Improvements

A few enhancements are on our roadmap:

Rollback Support: Automatically revert changes if validation fails
Central Logging: Send update and error logs to our SIEM for visibility
Multi-OS Support: Extend to RHEL or other Unix variants
User Self-Service: Allow team leads to submit requests for privilege changes through a web interface

If your organization uses Vault and GitLab, this model is highly extensible and easy to scale.

Conclusion

Managing sudo privileges manually is a relic of the past. With version-controlled rules, CI/CD validation, and secure secrets delivery, we’ve turned a risky process into a reliable one.

We hope this breakdown helps your team improve your access control practices. Treat sudoers like code—secure it, test it, and automate it.

If you’re curious about implementation or want to adapt this to your own environment, feel free to reach out!

Sudoers as Code: A CI/CD Approach to Privileged Access Management