The Great Migration: From Global API Keys to Secure Tokens

In the early days of our infrastructure automation, we built a custom Nginx container image. This image was designed to serve as a reverse proxy and SSL certificate management layer for our internal services. It was a workhorse—handling routing, SSL termination, and automatic certificate generation and renewal through Let’s Encrypt.

The architecture was elegant in its simplicity:

• A Docker container with Nginx and Certbot
• Cloudflare DNS integration for DNS-01 challenges
• Automatic certificate lifecycle management
• Support for both local development and production environments

The system worked flawlessly. Certificates were generated, renewed automatically, and SSL termination happened seamlessly. For months, it ran like clockwork, managing certificates for dozens of internal domains without intervention.

Security Audit

During a routine security review, a critical vulnerability was identified in our authentication approach. We were using Cloudflare’s Global API Key for DNS authentication—a method that, while functional, posed significant security risks.

The Problem with Global API Keys:

• Global API Keys provide access to the entire Cloudflare account
• They can’t be scoped to specific zones or operations
• If compromised, they expose all zones and settings
• They don’t follow the principle of least privilege
• Revocation affects all services using the key

The security team’s recommendation was clear ( I was that Security team member ): migrate to Zone-Specific API Tokens or Account API Tokens. These tokens could be:

• Scoped to specific zones
• Limited to specific operations (DNS read/write only)
• Revoked independently without affecting other services
• Rotated more easily
• Audited more effectively

This was not just a best practice—it was a security imperative.

The Migration Plan

The migration plan was straightforward, or so we thought:

Phase 1: Token Creation

• Create zone-specific API tokens in Cloudflare dashboard
• Grant minimal permissions: DNS read and write for specific zones
• Document token scopes and associated zones

Phase 2: Image Update

• Update the Nginx container image to use token-based authentication
• Modify the entrypoint script to write tokens to /etc/cloudflare.ini
• Remove references to Global API Key and email
• Update environment variable requirements

Phase 3: Deployment

• Update production configurations with new token variables
• Deploy updated image
• Verify certificate generation works
• Monitor renewal cycles

Phase 4: Cleanup

• Revoke old Global API Keys
• Update documentation
• Close security findings

Simple. Clean. What could go wrong?

The Implementation

The image update was implemented with careful attention to detail:

Changes Made:

1. Updated setup_cloudflare_ini() to use CLOUDFLARE_API_TOKEN instead of Global API Key
2. Modified the credentials file format from:dns_cloudflare_api_key = <key> dns_cloudflare_api_email = <email> To:dns_cloudflare_api_token = <token>
3. Updated environment variable requirements in documentation
4. Removed all references to CF_API_KEY and CF_API_EMAIL from the codebase
5. Updated Docker Compose templates to use token variables

The code review passed. The changes looked correct. The credentials file format matched Cloudflare’s documentation. Everything was ready.

The Deployment

The new image with token-based authentication was built, pushed to the docker registry, and deployed. The migration to token-based authentication was complete:

• Code updated to use tokens
• INI file creation updated to use dns_cloudflare_api_token
• Token environment variable set in Docker Compose
• Deployment process was smooth—containers restarted, configurations loaded, and the system appeared operational

The first certificate generation was scheduled for a test domain. The process started normally:

• Container initialized
• Environment variables loaded
• Credentials file (/etc/cloudflare.ini) created with the new token format
• Certbot invoked with DNS-01 challenge

Then, silence. The authentication failed.

The Failure

The first failure came in the logs:

dns-01 challenge for bakra.com
Cleaning up challenges
Error determining zone_id: 6003 Invalid request headers. 
Please confirm that you have supplied valid Cloudflare API credentials.
(Did you copy your entire API token/key? To use Cloudflare tokens, 
you'll need the python package cloudflare>=2.3.1. This certbot is 
running cloudflare 2.19.4)

The error was immediate and absolute. The DNS-01 challenge failed before it could even begin. Certbot couldn’t authenticate with Cloudflare’s API, which meant:

• No certificate generation
• No certificate renewal
• SSL termination broken
• Production services potentially impacted

Initial Assessment:

• ✅ Migration to token-based authentication was complete
• ✅ Token was correctly formatted in the environment variable
• ✅ Credentials file (/etc/cloudflare.ini) was being created properly with dns_cloudflare_api_token
• ✅ Token was valid and working (confirmed by manual test)
• ✅ Certbot command syntax was correct

Everything was configured correctly for the new token-based authentication. So why was authentication failing?

The Investigation Begins

The investigation started with verification of the obvious:

1. ✅ Token was correctly set in environment variables
2. ✅ Token was properly written to /etc/cloudflare.ini
3. ✅ File permissions were correct (600)
4. ✅ Certbot command included --dns-cloudflare-credentials
5. ✅ Token format matched Cloudflare’s documentation

But the error persisted. The 6003 error code specifically indicated “Invalid request headers,” which suggested the authentication headers sent to Cloudflare’s API were malformed.

The Hypothesis: Perhaps the token was being truncated during file write? Or maybe there was whitespace? Or the token itself was invalid?

We added verbose logging, checked file contents, verified token validity in Cloudflare dashboard. Everything checked out. The token was valid, the file was correct, but authentication still failed.

The Version Update Attempt

First Suspect: Python Package Version

The error message mentioned the Python Cloudflare package version: “This certbot is running cloudflare 2.19.4”. While 2.19.4 is greater than the required 2.3.1, we wondered if there might be a bug in this specific version or if we needed the absolute latest version.

The Decision: Update the Cloudflare Python package to the latest version in the Dockerfile. Perhaps a newer version had bug fixes or better token support.

The Change:

RUN pip install --upgrade certbot certbot-dns-cloudflare cloudflare

We rebuilt the image, pushed it to the docker registry, and deployed it to production. The anticipation was high—surely a version update would fix this.

The Result: The same error. The exact same error. Certificate generation failed with the identical “6003 Invalid request headers” error. The version update changed nothing.

The Realization: This wasn’t a version issue. The problem was deeper, more fundamental. Something was wrong with how the authentication was being attempted, not with the library version itself.

The Manual Test Mystery

The Smoking Gun

Frustrated and confused, we decided to test manually. We exec’d into the running container and manually executed the exact same certbot command that was failing in the automated process:

docker exec -it <container> bash
certbot certonly -n --agree-tos -m "$CERTBOT_EMAIL" \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/cloudflare.ini \
  --preferred-challenges dns-01 \
  -d "$CERTBOT_DOMAIN" \
  -v

The Result: It worked. Perfectly. The certificate was generated successfully. The DNS-01 challenge completed. Everything functioned as expected. This proved beyond doubt that:

• ✅ The token was valid
• ✅ The credentials file (/etc/cloudflare.ini) was correct
• ✅ The token-based authentication method was working
• ✅ The configuration was correct

The Confusion: How could the same command work when executed manually but fail when run through the entrypoint script? The command was identical. The credentials file was the same. The token was the same. The container was the same. The token-based authentication was clearly correct—so why was it failing in automated execution?

The Investigation: We compared the environments:

• Same container
• Same credentials file
• Same certbot command
• Same Python version
• Same Cloudflare package version

But wait—what about the environment? We checked the environment variables in both contexts:

Manual execution (working):

• Clean shell environment
• Only the variables we explicitly set
• No inherited variables

Automated execution (failing):

• Environment from entrypoint script
• Possibly inherited variables from Docker Compose
• Variables from the container image itself
• Variables from the orchestration layer

The Hypothesis: Something in the environment during automated execution was different. Something was interfering with the authentication process.

The Research Dead End

Searching for Solutions

We searched everywhere:

• GitHub issues for certbot-dns-cloudflare
• Stack Overflow questions
• Cloudflare community forums
• Certbot documentation
• Python Cloudflare library documentation

What We Found:

• Many people had similar errors
• Most solutions were about token format
• Some were about file permissions
• A few were about version issues
• None addressed our specific scenario: working manually but failing automatically

The Pattern: People who had this issue typically:

• Had incorrect token format
• Had file permission issues
• Had version compatibility problems
• Had incorrect credentials file format

None of these applied to us. We had successfully migrated to token-based authentication—our token was valid (it worked manually), our INI file was correct, our file permissions were correct, our version was fine, our format was correct. The migration was complete and correct.

The Frustration: We were stuck. The manual test proved the token-based authentication was correct and working. The automated execution proved something was interfering with our correct configuration. But what? The difference was subtle, invisible, maddening. We had done everything right, yet it was failing.

The Insight: If the manual test worked but automated execution failed, the difference had to be in the execution environment. The environment variables during automated execution must be different from the manual execution environment.

The Discovery

Breakthrough Moment

Armed with the insight from the manual test—that the environment was different—we began a systematic examination of the container’s environment during automated execution. We added debugging to print all environment variables, and we discovered something unexpected.

The Critical Discovery: We had successfully migrated to the new token-based authentication:

• ✅ The new /etc/cloudflare.ini file was correctly created with dns_cloudflare_api_token
• ✅ The token was valid (proven by the manual test working)
• ✅ The file format was correct
• ✅ The file permissions were correct

But the container environment still contained old environment variables from the previous Global API Key setup:

• CF_API_KEY (set to empty or old/invalid value from previous image)
• CF_API_EMAIL (set to empty or old/invalid value from previous image)
• CF_API_TOKEN (potentially set)
• CLOUDFLARE_CFG (potentially set)
• CLOUDFLARE_EMAIL (potentially set)

These variables were remnants from:

• Previous container image versions that used Global API Keys
• Base image environment
• Docker Compose inheritance
• Orchestration layer defaults

The Critical Insight: The Python Cloudflare library used by certbot checks authentication methods in a specific priority order:

1. First: CF_API_KEY + CF_API_EMAIL (Global API Key method)
2. Second: CF_API_TOKEN environment variable
3. Third: Credentials file (/etc/cloudflare.ini)

The Problem: Even though we had correctly migrated to the new token-based authentication in /etc/cloudflare.ini, the library was checking the old environment variables first. The container had stale CF_API_KEY and CF_API_EMAIL variables from the previous image version. These were being checked first, causing the library to attempt authentication using empty or invalid values. The authentication failed with “Invalid request headers” before the library ever reached the credentials file containing the valid token.

Why This Happened:

• We had updated the code to use tokens
• We had updated the INI file creation to use tokens
• But we hadn’t removed the old environment variables from the container environment
• The library’s authentication priority meant it tried the old method first, failed, and never reached the new method

This was a classic authentication priority conflict—the migration from Global API Key to Token was correct, but the old authentication method was still being attempted first due to lingering environment variables, preventing the new method from ever being used.

The Root Cause Analysis

Deep Dive

The root cause was a combination of factors:

1. Backward Compatibility: The Cloudflare Python library prioritizes environment variables for backward compatibility with older setups using Global API Keys.
2. Environment Variable Persistence: In containerized environments, environment variables can persist from:
   • Base images
   • Previous image versions
   • Docker Compose service definitions
   • Orchestration platform defaults
   • Shell initialization scripts
3. Silent Fallback Failure: The library doesn’t explicitly log when it tries environment variables first and fails—it just fails with a generic authentication error.
4. Migration Blind Spot: When migrating from Global API Key to Token, we successfully updated the code to use tokens and created the new INI file correctly. However, we didn’t account for the fact that old environment variables from the previous setup might still exist in the container environment and interfere with the new authentication method.
5. Authentication Priority Chain: The library’s authentication priority meant that even though we correctly configured token-based auth in the INI file, the old environment variables were checked first, causing authentication to fail before the library could reach the new token-based method.

Why This Happened During Migration:

• We successfully migrated to token-based authentication
• The new INI file was correctly created with valid tokens
• But old environment variables from previous image versions persisted
• Base images might have included them
• Docker Compose files might have inherited them from parent services
• Orchestration platforms might have injected them
• The migration focused on “implementing token support” but didn’t explicitly “remove old auth environment variables”

The Solution Design

Architectural Decision – The Unset Solution

After exhausting all research and finding no existing solutions for our specific scenario, we had to think outside the box. The manual test had proven the configuration was correct. The environment variable discovery had revealed the culprit. Now we needed a solution.

The Realization: We had correctly migrated to token-based authentication in the INI file, but old environment variables were interfering. The library was checking those old variables first, causing failures before it could reach our valid token in the INI file. Since we couldn’t find existing solutions, we needed to create our own. The answer was simple: remove the interfering variables before certbot runs, forcing the library to use only the INI file.

The Approach: Create a cleanup function that explicitly unsets all Cloudflare authentication-related environment variables before certbot runs. This forces the library to skip environment variable checks entirely and go directly to the credentials file where our valid token resides.

The Innovation: This was a defensive programming approach—don’t trust the environment, make it trustworthy. By explicitly cleaning the environment, we eliminate the possibility of interference from stale or inherited variables.

The Function:

certbot_cleanup_env() {
    # This prevents the library from attempting Global API Key auth
    # and forces it to use the API token from /etc/cloudflare.ini
    unset CF_API_KEY CF_API_EMAIL CF_API_TOKEN CLOUDFLARE_CFG CLOUDFLARE_EMAIL
}

Why Each Variable:

• CF_API_KEY and CF_API_EMAIL: Legacy Global API Key authentication method
• CF_API_TOKEN: Alternative token environment variable (we want file-based only)
• CLOUDFLARE_CFG: Configuration file path override (could cause conflicts)
• CLOUDFLARE_EMAIL: Additional email variable (might confuse the library)

The Strategy:

1. Create the credentials file with the valid token
2. Clean the environment of all authentication variables
3. Execute certbot with only the credentials file as authentication source
4. Ensure consistent behavior across all certificate operations

The Implementation

Code Changes

The cleanup function was strategically integrated into the certificate management workflow:

Location 1: generate_cert() Function

generate_cert() {
    log_info "Generating new certificate..."
    setup_cloudflare_ini          # Create credentials file
    certbot_cleanup_env           # Clean environment
    check_renewal_config          # Verify renewal config
    
    if verify_domain_ip; then
        if certbot certonly ...; then
            # Success path
        fi
    fi
}

Location 2: renew_cert() Function

renew_cert() {
    # ... certificate expiry checks ...
    
    log_info "Certificate renewal needed..."
    setup_cloudflare_ini          # Create credentials file
    certbot_cleanup_env           # Clean environment
    check_renewal_config          # Verify renewal config
    
    if certbot renew ...; then
        # Success path
    fi
}

Execution Flow:

setup_cloudflare_ini() 
    ↓
certbot_cleanup_env() 
    ↓
check_renewal_config() 
    ↓
certbot execution (with clean environment)
    ↓
Authentication via /etc/cloudflare.ini only

This ensures that:

• The credentials file is always created first
• The environment is always cleaned before certbot runs
• Certbot has no choice but to use the credentials file
• Both generation and renewal follow the same clean path

The Technical Deep Dive

Why This Solution Works

Authentication Priority Elimination: The Cloudflare Python library’s authentication mechanism is designed for flexibility—it tries multiple methods to make it easy for users. However, in containerized environments with complex inheritance chains, this flexibility becomes a liability.

By explicitly unsetting all authentication environment variables, we:

1. Break the Priority Chain: The library can’t find environment variables, so it skips those checks
2. Force File-Based Auth: The credentials file becomes the only viable authentication source
3. Eliminate Ambiguity: There’s no question about which authentication method will be used
4. Prevent Stale Credentials: Old or invalid credentials can’t interfere

The Error Code 6003 Explained: Cloudflare’s 6003 error indicates malformed HTTP request headers. This happens when:

• Authentication headers are missing required fields
• Headers contain invalid or empty values
• Multiple authentication methods are attempted simultaneously
• Token/API key format is incorrect

In our case, we had correctly migrated to token-based authentication with a valid token in /etc/cloudflare.ini. However, the library was attempting to use old CF_API_KEY and CF_API_EMAIL environment variables first, constructing headers using empty or invalid values, resulting in malformed headers that Cloudflare rejected. The library never reached the credentials file containing our valid token because it failed on the old environment variables first.

Defensive Programming Principle: This solution follows the defensive programming principle: “Don’t trust the environment—make it trustworthy.” Instead of assuming the environment is clean, we explicitly clean it. This prevents:

• Environment variable leakage from base images
• Configuration inheritance issues
• Orchestration platform variable injection
• Future regressions from environment changes

The Migration Impact

Broader Implications

This issue revealed several important lessons about security migrations:

1. Migration Complexity: Moving from one authentication method to another isn’t just about updating code—it’s about ensuring the old method is completely removed from the execution path. Legacy code paths can persist in unexpected places.

2. Environment Variable Hygiene: In containerized environments, environment variables can come from multiple sources. Assuming a clean environment is dangerous. Explicit cleanup is essential.

3. Authentication Priority Awareness: When libraries support multiple authentication methods, understanding the priority order is critical. Higher-priority methods can interfere with lower-priority ones, even when the lower-priority method is correct.

4. Security Migration Best Practices:

• Audit all authentication-related environment variables
• Explicitly remove old authentication methods
• Test with clean environments
• Verify authentication method priority
• Add defensive cleanup steps

5. Container Image Lifecycle: Container images can carry environment variables from:

• Base images
• Build-time settings
• Previous versions
• Dockerfile ENV instructions
• Runtime inheritance

These must be explicitly managed during migrations.

The Verification Process

Testing and Validation

After implementing the fix, comprehensive verification was performed:

1. Environment Variable Audit:

• Verified all Cloudflare-related variables are unset before certbot execution
• Confirmed no variable leakage from base images
• Checked Docker Compose inheritance chains

2. Authentication Flow Verification:

• Confirmed certbot uses only /etc/cloudflare.ini for authentication
• Verified no environment variable fallback attempts
• Validated token-based authentication works correctly

3. Certificate Operations Testing:

• Tested initial certificate generation
• Tested certificate renewal
• Verified DNS-01 challenge completion
• Confirmed SSL termination works

4. Edge Case Testing:

• Tested with old environment variables present (should be ignored)
• Tested with multiple Cloudflare variables set (all should be cleared)
• Tested with empty/invalid variables (should be cleared and not used)
• Tested with missing token (should fail gracefully with clear error)

5. Configuration Verification:

• Verified Docker Compose files only set CLOUDFLARE_API_TOKEN
• Confirmed no legacy variables in production configurations
• Validated credentials file format matches Cloudflare requirements

Results: ✅ All authentication variables properly cleared
✅ Certbot uses credentials file exclusively
✅ Certificate generation successful
✅ Certificate renewal successful
✅ No environment variable conflicts
✅ Clean authentication flow verified

The Production Deployment

Go-Live

With verification complete, the updated image was deployed to production:

Deployment Steps:

1. Built new image with cleanup function
2. Pushed to container registry
3. Updated production Docker Compose configurations
4. Deployed new containers
5. Monitored certificate generation
6. Verified SSL termination

Monitoring:

• Certificate generation logs
• Authentication success/failure rates
• DNS-01 challenge completion
• SSL certificate validity
• Service availability

Results:

• ✅ Certificate generation successful on first attempt
• ✅ No authentication errors
• ✅ DNS-01 challenges completing successfully
• ✅ SSL certificates valid and working
• ✅ No service disruptions

The migration was complete. The system was now using secure, zone-scoped API tokens instead of Global API Keys, and the authentication flow was clean and predictable.

The Security Improvements

Security Posture Enhancement

The migration achieved several security improvements:

1. Principle of Least Privilege:

• Zone-specific tokens with minimal required permissions
• Scoped to specific DNS operations only
• No access to account-level settings

2. Token Management:

• Tokens can be rotated independently
• Revocation doesn’t affect other services
• Easier to audit and track usage

3. Reduced Attack Surface:

• No Global API Keys in use
• Compromised tokens only affect specific zones
• Easier to isolate and contain issues

4. Operational Security:

• Better separation of concerns
• Clearer authentication boundaries
• More granular access control

5. Compliance Alignment:

• Follows security best practices
• Aligns with industry standards
• Supports audit requirements

Lessons Learned

This incident taught us several critical lessons:

1. Migration Complexity: Security migrations are more complex than they appear. Changing authentication methods requires understanding the entire authentication chain, including priority orders, fallback mechanisms, and environment variable persistence.

2. Environment Variable Management: In containerized environments, environment variables are not always what they appear to be. They can come from multiple sources, persist across image versions, and interfere with intended configurations. Explicit cleanup is essential.

3. Authentication Library Behavior: When libraries support multiple authentication methods, understanding their priority order is critical. Higher-priority methods can interfere with lower-priority ones, even when the lower-priority method is correct.

4. Manual Testing as a Debugging Tool: When automated execution fails but manual execution works, the difference is almost always in the execution environment. Manual testing provides a clean baseline that can reveal environment variable issues, inheritance problems, and configuration conflicts that aren’t visible in automated contexts.

5. Version Updates Don’t Always Fix Issues: Updating libraries to the latest version is often a good first step, but it doesn’t always solve the problem. The issue might be in how the library is used, not in the library itself. Don’t assume version updates will fix configuration or environment-related issues.

6. Research Dead Ends Can Be Valuable: When existing solutions don’t address your specific scenario, it’s a signal that the problem might be unique to your environment or configuration. This forces creative problem-solving and can lead to innovative solutions that benefit others facing similar issues.

7. Defensive Programming: Don’t assume the environment is clean—make it clean. Explicit cleanup steps prevent unexpected behavior from stale or inherited values. When in doubt, clean the environment explicitly.

8. Testing Strategy: Test migrations with realistic environments, including potential legacy configurations. Edge cases matter, especially when dealing with authentication mechanisms. Manual testing provides insights that automated tests might miss.

9. Documentation Importance: Document authentication method priority, environment variable sources, and cleanup requirements. Future developers need this context.

10. Security Migration Best Practices:

• Audit all authentication-related code paths
• Remove old authentication methods explicitly
• Test with clean and dirty environments
• Verify authentication method priority
• Add defensive cleanup steps
• Monitor for unexpected behavior

The Code Changes

Technical Implementation Details

File Modified:

• docker-entrypoint.sh

Function Added:

certbot_cleanup_env() {
    # This prevents the library from attempting Global API Key auth
    # and forces it to use the API token from /etc/cloudflare.ini
    unset CF_API_KEY CF_API_EMAIL CF_API_TOKEN CLOUDFLARE_CFG CLOUDFLARE_EMAIL
}

Functions Modified:

1. generate_cert() – Added certbot_cleanup_env() call after setup_cloudflare_ini() and before check_renewal_config()
2. renew_cert() – Added certbot_cleanup_env() call after setup_cloudflare_ini() and before check_renewal_config()

Environment Variables Cleared:

• CF_API_KEY – Legacy Global API Key
• CF_API_EMAIL – Legacy Global API Key email
• CF_API_TOKEN – Alternative token environment variable
• CLOUDFLARE_CFG – Configuration file path override
• CLOUDFLARE_EMAIL – Additional email variable

Authentication Method:

• File-based: /etc/cloudflare.ini containing dns_cloudflare_api_token = <token>

Execution Order:

1. setup_cloudflare_ini() – Creates credentials file
2. certbot_cleanup_env() – Clears environment variables
3. check_renewal_config() – Verifies renewal configuration
4. Certbot execution – Uses credentials file exclusively

The Aftermath

With the fix in place, the SSL certificate management system now operates with:

✅ Clean Authentication Environment

• No conflicting environment variables
• Single, unambiguous authentication source
• Predictable authentication flow

✅ Secure Token-Based Authentication

• Zone-scoped API tokens
• Principle of least privilege
• Independent token management

✅ Reliable Certificate Operations

• Successful certificate generation
• Successful certificate renewal
• DNS-01 challenges completing correctly

✅ Defensive Programming

• Environment cleanup prevents future issues
• Explicit authentication method selection
• Protection against environment variable leakage

✅ Production Stability

• No authentication errors
• Consistent certificate lifecycle
• Reliable SSL termination

The Resolution

Final Status: ✅ Resolved

Impact:

• Critical authentication failures eliminated
• Security posture significantly improved
• Migration from Global API Keys to secure tokens completed

Risk Level:

• Low – Defensive cleanup with no side effects
• Improves system reliability and security

Maintenance:

• Self-maintaining – no ongoing action required
• Cleanup function runs automatically before each certbot execution

Documentation:

• Migration process documented
• Authentication flow documented
• Environment variable management documented
• Security improvements documented

Epilogue

What started as a security hardening initiative—a migration from Global API Keys to secure zone-scoped tokens—revealed a deeper issue with authentication priority in containerized environments.

The troubleshooting journey was instructive: we had successfully migrated to token-based authentication with a valid token in the INI file, but authentication was failing. We first suspected a version issue and updated the Cloudflare Python package, only to find the same error persisted. We then manually tested the certbot command inside the container, and to our surprise, it worked perfectly—proving our token-based authentication was correct and the configuration was valid, but something in the automated execution environment was different. Exhaustive research revealed no existing solutions for our specific scenario, forcing us to think creatively and develop our own solution.

The discovery was that old environment variables from the previous Global API Key setup were still present in the container environment. Even though we had correctly migrated to token-based authentication in the INI file, the library was checking those old environment variables first, causing failures before it could reach our valid token. The fix, while simple in execution—a function that unsets environment variables—addresses a fundamental problem with how authentication libraries handle multiple credential sources in complex containerized systems. The manual test had been the key insight: if it works manually but fails automatically, the difference must be in the environment.

The solution ensures that our SSL certificate management system operates in a predictable, clean environment, eliminating a class of authentication failures that could have plagued the system indefinitely. More importantly, it demonstrates the importance of defensive programming, explicit environment management, and the value of manual testing when automated processes fail mysteriously.

The migration is complete. The system is more secure. The authentication flow is clean. And we’ve learned valuable lessons about security migrations, environment variable management, authentication library behavior, and creative problem-solving when standard solutions don’t apply.