The Mystery

Sometimes the most frustrating bugs are the ones hiding in plain sight. Last week, the application I am working on is not working - the login page was returning "not found" errors. The timing was suspicious: the issue started right after deploying what should have been a minor configuration update. Let me walk you through how we diagnosed and fixed this surprisingly subtle issue.

The Setup

Our application uses AWS CloudFormation via the AWS CDK to provision resources, including:

  1. Route53 DNS records for our application endpoints
  2. SSM Parameters to store configuration values

One critical piece of infrastructure is our login route configuration, which tells our application how to handle login requests. This configuration is stored in an SSM Parameter that our application reads during startup.

Here's the relevant code that creates this parameter:

def _create_login_route_parameter(self, config, login_dns_record):
    """
    Creates an SSM Parameter for the Login Route.
    """
    if hasattr(config, 'login_route'):
        param_value = config.login_route
    else:
        param_value = login_dns_record.name

    login_route_parameter = ssm.CfnParameter(
        self,
        "LoginRouteParameter",
        name=(
            f"/application/{config.stack_name}/user.ini/"
            "resources.router.routes.login.route"
        ),
        type="String",
        value=param_value
    )

    self.resources['login_route_parameter'] = login_route_parameter

    return login_route_parameter

The Error

When I deployed our infrastructure, I started seeing this cryptic error:

Resource handler returned message: "Error occurred during operation 'PutParameter'." 
(RequestToken: 19b98abc-1b33-4822-aar4-c12wffer3a73, HandlerErrorCode: GeneralServiceException)

Looking at the CloudFormation template that was generated, we discovered something suspicious:

"ApplicationLoginRouteParameterC8560C99": {
  "Type": "AWS::SSM::Parameter",
  "Properties": {
    "Name": "/application/Dev-app/user.ini/resources.router.routes.login.route",
    "Tags": {
      "Business Unit": "IT",
      "Environment": "Development",
      "Owner": "DevOps",
    },
    "Type": "String",
    "Value": ""
  }
}

The Value property was an empty string! AWS SSM Parameter Store doesn't allow parameters with empty values, which explained our immediate deployment error. And it took me 2 days to figure this out.

The Investigation

So why was our parameter value empty? Looking back at our configuration, we discovered we had included this in our config:

"login_route": ""

We had explicitly set an empty value for login_route. This was the key insight: our code was checking if the attribute existed but not whether it contained a meaningful value.

The if-statement if hasattr(config, 'login_route') was evaluating to True because the attribute existed, even though its value was an empty string. This led to our code choosing the empty string instead of falling back to the DNS record name as intended.

The Fix

The solution was simple but highlights an important debugging lesson. We updated our code to check not only if the attribute exists but also if it contains a non-empty value:

def _create_login_route_parameter(self, config):
    """
    Creates an SSM Parameter for the Login Route.
    """
    # Check if login_route exists AND is not empty
    if hasattr(config, 'login_route') and config.login_route:
        param_value = config.login_route
    else:
        param_value = f"login-{config.root_dns_name}"

    # Add a final check to ensure we never have an empty value
    if not param_value:
        raise ValueError("Cannot create SSM Parameter with empty value. Please provide a valid login_route or root_dns_name.")

    # Rest of the function...

With this change, our deployment succeeded, and the login page started working again!

Lessons Learned

This debugging adventure taught us several valuable lessons:

  1. Empty strings are truthy for existence checks but falsy in boolean context - When checking if an attribute should be used, we need to verify both its existence AND its value.

  2. Understand your service constraints - AWS SSM Parameter Store doesn't accept empty string values. Understanding these constraints helps prevent similar issues.

  3. Validate inputs thoroughly - Even if you're certain the input will never be empty, add validation to fail fast and provide clear error messages.

  4. Use debug prints effectively - Adding a simple print(f"SSM Parameter Value: {param_value}") would have immediately revealed the issue during development.

The Python Gotcha: Empty Strings

This bug illustrates a common Python gotcha. In Python:

# This checks if the attribute exists, not if it has a meaningful value
if hasattr(obj, 'attribute'):  # True even if obj.attribute = ""
    do_something()

# This checks both existence and truthiness (empty string is falsy)
if hasattr(obj, 'attribute') and obj.attribute:
    do_something()

Remember: an empty string exists as an attribute, but evaluates to False in boolean context. When working with configuration values that might be empty, always check both!

Conclusion

A simple empty string caused our application to be unavailable. It's a reminder that sometimes the most frustrating bugs aren't complex logic errors but simple oversights in handling edge cases.

Next time you're setting up conditional logic, especially with configuration values, remember to check not just if the value exists, but if it contains meaningful data!

Happy debugging!