Well-Architected Update 2023

AWS recently released a new set of changes to the AWS Well-Architected Framework. This is a rare event and since the inception of Well-Architected in 2012 has only happened 8 times in the 11 years since the framework’s inception.

The key changes to the framework expand on the Security pillar concentrating on the security best practices of Application Development.

There was also a change to the Cost pillar and some change in the priority of some best practices, but for this post, I’m concentrating on the main changes to the Security Pillar.

A whole new question…

AWS have this time created a whole new question and a series of best practices to go with it, all about Application Security.

SEC 11 - Application Security

“How do you incorporate and validate the security properties of applications throughout the design, development, and deployment lifecycle?”

Security is no longer just about implementing security around your application or data encryption. Security should be implemented from the first line of code to build and deployment pipelines, software distribution, and then implementation.

This new question to the Well-Architected Framework is designed to create some thought leadership and best practices to enable your team to develop your application, product or business securely.

Werner Vogels is known for his many clever quotes, coined during events like re: Invent. In this case;

“Dance like no one is watching, encrypt like everyone is”

This is no longer enough. Security must have a top-down approach, both within your team (Executives down to developers) structure and your team practices.

This new approach reinforces the messaging, “Security is job zero”.

Best Practices

SEC11-BP01 Train for application security

“Provide training to the builders in your organization on common practices for the secure development and operation of applications. Adopting security-focused development practices helps reduce the likelihood of issues that are only detected at the security review stage.”

The key message from this best practice is to train your team to develop secure applications. This includes threat modelling* and internal cyber security training. It also talks about the use-case of supporting access to AWS Training & Certification and AWS Partner training and using Game Days as a way to increase the skill sets of developers and builders and to reduce the number of issues in the backlog or bugs.

*[ https://catalog.workshops.aws/threatmodel/en-US ]

Avoid leaving security until a security review has happened though. A lot of customers have said to me in the past, “As an organisation, we don’t review security until pen testing has happened”. This is no longer acceptable.

Your people are your best friends when it comes to security, train them in the basics of security within your Software Development Lifecycle and they will do what’s best.

SEC11-BP02 Automate testing throughout the development and release lifecycle

“Automate the testing for security properties throughout the development and release lifecycle. Automation makes it easier to consistently and repeatably identify potential issues in software prior to release, which reduces the risk of security issues in the software being provided.”

This is one of the new best practices that hold the most weight from the newest question in the security pillar. This starts to look at technologies that can be used through the software development lifecycle to help with security best practices. This includes;

• A standard set of development tools (IDE, Frameworks & Libraries) used across the whole business which can have security testing integrated.
• Blocking pipelines rather than just alerting for security issues to stop the release of security issues
• Testing and analysing code using tools such as Amazon CodeWhisperer and Amazon CodeGuru Reviewer. Adding these tools to your IDE or pipeline can help best practices and stop security issues at the development level.
• Using Amazon Inspector for AWS Lambda function.
• Automating testing where possible within the CI/CD pipeline.

SEC11-BP03 Perform regular penetration testing

“Perform regular penetration testing of your software. This mechanism helps identify potential software issues that cannot be detected by automated testing or a manual code review. It can also help you understand the efficacy of your detective controls. Penetration testing should try to determine if the software can be made to perform in unexpected ways, such as exposing data that should be protected or granting broader permissions than expected.”

Penetration testing has always been something I’ve recommended for workloads within AWS but now it goes beyond perimeter checks to application penetration testing. SEC11-BP03 now looks at penetration testing in line with your threat modelling and also makes sure it’s added to your software development life cycles.

More importantly, also train your developers, builders and support staff on what to expect from a penetration test to ensure there is a business-wide benefit and how to get information on remediation.

Penetration testing should be automated, where possible, using best practice toolsets to ensure consistency in the results and the results should create further automation to create additional tests

SEC11-BP04 Manual code reviews

“Perform a manual code review of the software that you produce. This process helps verify that the person who wrote the code is not the only one checking the code quality.”

Surprisingly, in a new question which talks a lot about automation of your checks within your software development life cycle, there are some manual actions to take. By manually reviewing someone else’s code before it goes to release you’re ensuring the quality of the code being released is good and also looking for any security flaws in the product.

There are tools that can help here, such as AWS CodeGuru or similar partner products.

At this point, pair programming is a concept which could also be very useful. This is used in many consultancies and has many benefits from code quality to skills uplift between partners.

As well as manual code review, implement code review and approval stages within your software development life cycle. Repository merge approvals and CI/CD manual approvals before progressing to the next stage are advised.

As you find issues within your development processes manually, start to look for patterns and automation that can help with these processes. (See, we finally came back to automation!)

SEC11-BP05 Centralize services for packages and dependencies

“Provide centralized services for builder teams to obtain software packages and other dependencies. This allows the validation of packages before they are included in the software that you write, and provides a source of data for the analysis of the software being used in your organization.”

This best practice is closely related to SEC11-BP02 (above).

When working in large teams it’s sometimes difficult to control the version of packages being used across all developers, e.g. JSON parsers, encryption software or image manipulation packages. This introduces a level of risk that parts of your software may be impacted by some CVEs, despite implicit trust in a package or library.

Logically centralizing the sources for shared packages and frameworks ensures that security teams can pre-vet and approve the use of libraries. Don’t be fooled though by allowing the use of packages from repositories on the internet or blindly approving packages, this can lead to disaster.

Overall, look at creating a central repo, fork your favourite frameworks or libraries and control the versions that are fed into your repo. By running a fork of a repo, you can also have your teams contribute back to open source communities as well, crowdsourcing security improvements. Maybe run a Core Intellectual Property (Core IP) library in your organisation. This way teams are also able to contribute back to libraries.

SEC11-BP06 Deploy software programmatically

“Perform software deployments programmatically where possible. This approach reduces the likelihood that a deployment fails or an unexpected issue is introduced due to human error.”

It’s common to allow developers access to wherever they need to deploy to as sometimes it just feels like the easiest way to allow any of the following;

• Zip files
• Upload/download
• git push/git pull
• MSI install
• pip install
• npm install
• …

I could go on all day with the different deployment practices seen in the wild, however, this new best practice is designed to remove the developers from the production environment. In fact, a production environment should not need anyone logged into it. Production should be considered sacred, clinical and sanitised.

I’m going to be open and controversial here. People make mistakes! We’re only human after all and that’s ok. So don’t make mistakes in production that might cause a customer outcome to go sour.

Set yourself up a good CI/CD pipeline which will first take your code, build it, test it and then deploy it into a dev/test/sandbox/staging/uat environment (or account). Take your pick, any or all of the suggested names are good. Once testing is passed, automatically deploy, via some sort of pipeline, to production. There should be no manual updates to databases, and no code or libraries “adjusted to fit” on the server.

To take this one step further, cryptographically sign your packages as they are built and deployed, this means you can prove that software passed your testing and QA process prior to deployment.

Finally, allow for rollback in your automated deployment, in case not everything goes well.

Overall, you’ll reduce the risk of mistakes and increase confidence of your production environment being stable.

SEC11-BP07 Regularly assess the security properties of the pipelines

“Apply the principles of the Well-Architected Security Pillar to your pipelines, with particular attention to the separation of permissions. Regularly assess the security properties of your pipeline infrastructure. Effectively managing the security of the pipelines allows you to deliver the security of the software that passes through the pipelines.”

If you don’t already, consider your CI/CD pipeline as its own workload. It should be assessed over time against the Well-Architected framework. Assessing delivery pipelines separately ensures another level of security is applied. As we said at the top of the piece, Security is no longer how to just implement your product.

Things to avoid here are things like CI/CD pipelines using overly broad permission sets, ensuring they only have the permissions they need to deploy. Make sure builders can’t just bypass tests either and make sure your chosen deployment method is not using hard-coded credentials such as access keys.

Lastly, ensure your pipelines are integrated with your monitoring solution. It’s important to know when a build failed, why it failed and take action. Watch out for abnormal activity in your pipelines. This can be achieved by leveraging EventBridge, Lambda or SNS.

SEC11-BP08 Build a program that embeds security ownership in workload teams

“Build a program or mechanism that empowers builder teams to make security decisions about the software that they create. Your security team still needs to validate these decisions during a review, but embedding security ownership in builder teams allows for faster, more secure workloads to be built. This mechanism also promotes a culture of ownership that positively impacts the operation of the systems you build.”

This best practice really ties into SEC11-BP01 and honestly is one of my more favoured best practices. Your key asset in fighting the good fight for security is your people.

“Empower your people to do the right thing” - Me.

Put together a program in your organisation such as a Cloud Guild or a Champions League or Cloud Advocates team. This can promote education and collaboration with your organisation far more effectively than formal training. Encourage side projects and leverage the platform to bring in guest speakers to share knowledge. Ensure that such an initiative has top-down support though, and engage with an executive sponsor to ensure success on an organisational level.

The rest of these best practices should come naturally if you achieve the above; Tracking metrics of training and documentation of who has what skills or certifications. Identifying project sponsors will become easier and their varying roles across build, security and other stakeholder groups. Do not rely on a single security team to control security in your organization. Instead have your security team, power the rest of your team.

“It’s incredible how passionate people are about having the tools to control their own destiny.” - Andy Jassy