Nowadays, it is standard procedure to make use of HTTPS to ensure secure communication with websites. Sometimes, this protocol could be applied in environments that comprise many machines that serve tens of thousands of internet sites or web applications. The most simple and economical way to do that would be to use SSL certificates from ‘Let’s Encrypt’.
To manage network traffic going towards hundreds of internet sites, it is recommended not to use a single machine (physical or not), but in favor of a cluster of them grouped within an autoscaling and a ‘Load Balancer’ to manage and allocate the traffic in the most balanced way possible. With autoscaling in front of your application and a fleet of EC2 instances, the use of the “HTTP-01 challenge” would be challenging or maybe you simply don’t want to install Certbot directly on your servers. In our case, we decided to avoid the installation of Certbot CLI on the servers and differentiate infrastructure-as-code between web applications and certificate flow.
I’m going to divide this argument into a series of three articles:
- How to generate certificates from ‘Let’s Encrypt’ using the ‘Certbot’ Python library and AWS Lambda.
- How to renew a certificate using Certbot and certificate files saved in the first step.
- Infrastructure as code using AWS CDK to deploy a Step-Function to automatically renew certificates.
Certbot utilization in AWS Lambda
Understanding the structure of the working directory is important because we must know how to reconstruct it when we need to renew the certificate. The following is an example of the Certbot’s working directory structure. You can find more information about directory structure in the Let’s Encrypt forum.
After the creation of a domain certificate, we need to save the resulting 4 files, with the .pem extension, that constitutes it. These files are found within the path
<certbot-work-dir>/archive/<domain>/. The name of these files ends with an incremental number starting from 1, which identifies how many times a certificate was created or renewed for the same domain.
During multiple tests for the same domain on the development environment with AWS Lambda, we should come across a “warm start” of the function where the storage is the same as the previous execution. In the storage path
<certbot-work-dir>/archive/<domain>/, Certbot will save another four files with the numeric suffix incremented by 1 compared to the previous one. A solution can be to copy the latest version of the certificate following the symbolic links we can find in
<certbot-work-dir>/live/<domain>/<file>.pem and that is what I have done.
In addition to the previously mentioned files, we need even to save the following files to allow the next renewal of the certificate:
- <certbot-work-dir>/renewal/<domain>.conf →Certbot information for the renewal of the certificate
- <certbot-work-dir>/accounts/<apihost>/directory/<account>/* →Account representation files on an ACME server. <apihost> parameter has to be extrapolated from the previous file <domain>.conf.
Create certificate
The AWS Lambda responsible for creating the certificate will have a straightforward input and output. Our goal is to receive a domain name as input for which to create a certificate and store the resulting files in an S3 bucket. I decided to categorize the certificate files on S3 by date and domain name: <bucket-name>/ssl/<year>/<month>/<day>/<domain>/. This structure will be useful in the next step to find the certificates that need to be renewed.
The Python code presented below enables you to perform all the operations we have discussed so far. Remember to create the Lambda function with the correct permissions to write to the S3 bucket. Add the environment variable called CERTS_S3_BUCKET to the Lambda to specify the name of the created S3 bucket.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
import boto3 from botocore.exceptions import ClientError import certbot.main from datetime import date import os DESTINATION_BUCKET = os.environ.get('CERTS_S3_BUCKET') step_functions = boto3.client('stepfunctions') client = boto3.client('s3') def read_file(path): with open(path, 'r') as file: contents = file.read() return contents def delete_files(path: str): for file_name in os.listdir(path): # construct full file path file = path.join(file_name) if os.path.isfile(file): print(f'Deleting {file}') os.remove(file) def provision_cert(email, domain, test_cert=False): certbot_options = [ 'certonly', # Obtain a cert but don't install it '-n', # Run in non-interactive mode '--agree-tos', # Agree to the terms of service, '--email', email, '--dns-route53', # Use dns challenge with route53 '-d', f'{domain},www.{domain}', # Domains to provision certs for # Override directory paths '--config-dir', '/tmp/config-dir/', '--work-dir', '/tmp/work-dir/', '--logs-dir', '/tmp/logs-dir/', ] if test_cert: certbot_options.append('--test-cert') certbot.main.main(certbot_options) path = '/tmp/config-dir/live/' + domain + '/' return { 'certificate': read_file(path + 'cert.pem'), 'private_key': read_file(path + 'privkey.pem'), 'certificate_chain': read_file(path + 'chain.pem') } def should_provision(domain, bucket=None): """The certificate exist only if the corresponding Nginx conf exists""" try: client.head_object( Bucket=bucket, Key=f'nginx/server-blocks/{domain}.conf', ) logger.warning(f'The certificate for {domain} already exists') return False except ClientError: logger.info(f'Certificate for {domain} does not exists, we need to create it ...') return True def parse_cert_conf_file(domain): domain_conf_file_name = os.path.join('/tmp/config-dir/', 'renewal', f"{domain}.conf") with open(domain_conf_file_name) as conf_file: for line in conf_file: if 'account' in line: renewal_account = line.split('=')[1].strip() if 'server' in line: renewal_server = line.split('=')[1].strip().split('//')[1] return renewal_account, renewal_server def upload_cert_s3(domain: str, cert_local_path: str, bucket: str): today = date.today() # Upload certs for dirpath, _dirnames, filenames in os.walk(cert_local_path): filenames.remove('README') # Remove useless file created by certbot for filename in filenames: local_path = os.path.join(dirpath, filename) s3_key = f'ssl/{today.year}/{today.month}/{today.day}/{domain}/{filename}' client.upload_file( local_path, bucket, s3_key, ExtraArgs={'ServerSideEncryption': 'AES256', 'ChecksumAlgorithm': 'SHA1'}) def upload_renewal_conf_s3(domain: str, conf_local_path: str, bucket: str): today = date.today() # upload renew config file domain_conf_file_name = f"{domain}.conf" s3_key = f'ssl/{today.year}/{today.month}/{today.day}/{domain}/{domain_conf_file_name}' try: client.upload_file( f"{conf_local_path}/{domain_conf_file_name}", bucket, s3_key, ExtraArgs={'ServerSideEncryption': 'AES256', 'ChecksumAlgorithm': 'SHA1'}) except ClientError as ex: raise ex return True def upload_account_files_s3(domain: str, accounts_dir: str, account_id: str, server, bucket: str): today = date.today() s3_key = f'ssl/{today.year}/{today.month}/{today.day}/{domain}' accounts_dir = os.path.join(accounts_dir, server, account_id) for file in os.listdir(accounts_dir): client.upload_file( os.path.join(accounts_dir, file), bucket, f"{s3_key}/{file}", ExtraArgs={'ServerSideEncryption': 'AES256', 'ChecksumAlgorithm': 'SHA1'}) def response(created: bool, reason: str = 'No description'): return {"IssuedCertificate": created, "Reason": reason} def handler(event, context): domain = event.get('domainName') root_path = '/tmp/config-dir' live_path = os.path.join(root_path, 'live', domain) renewal_path = os.path.join(root_path, 'renewal') archive_path = os.path.join(root_path, 'archive', domain) accounts_path = os.path.join(root_path, 'accounts') if should_provision(domain, DESTINATION_BUCKET): cert = provision_cert(os.environ['LETSENCRYPT_EMAIL'], domain) account, server = parse_cert_conf_file(domain) upload_cert_s3(domain, live_path, DESTINATION_BUCKET) upload_renewal_conf_s3(domain, renewal_path, DESTINATION_BUCKET) upload_account_files_s3(domain, accounts_path, account, server, DESTINATION_BUCKET) for path in [live_path, archive_path, renewal_path]: delete_files(path) status = response(True, f'Certificate for {domain} created and uploaded') else: status = response(False, f'Certificate for {domain} already issued') return status |
After creating the Lambda function, it could be executed by providing a JSON as input with a single parameter domainName :
|
1 2 3 |
{ "domainName": "mydomain.com" } |
After the successful execution of our function, we can check the S3 bucket to confirm the presence of certificate files.
It is now possible to establish HTTPS connections by downloading .pem files from the S3 bucket onto the web server.
Conclusion
The one presented here is just one of the methods to generate certificates through AWS serverless services. We could use a CloudFront + ACM solution or one similar to the one presented using ACM itself as the certificate store (that’s the link to the GitHub Gist).
The generated certificate will expire in 90 days… so be sure not to miss the next article in the series before that date to learn how to renew it. Cheerio!
