Building Secure Microservice Architectures

We shall use a video streaming platform in the vein of Netflix as our example for such an event-driven microservice architecture. Users can request video content, the prices for that content and what content they are already entitled to from the API Gateway, which retrieves the corresponding data from a catalogue service, pricing service and entitlement service respectively and merges the data to a client response. They can also get streaming URLs, for which the entitlements are retrieved and checked, the DRM system is prepared through a DRM connector service and the URLs are assembled by a stream service. There are also some clients (set-top boxes) that can access the middleware directly, circumventing the API Gateway. The services are populated with data by events the middleware system publishes to the message queue when content is received from an external content provider.

The first solution that comes to mind when thinking about how to secure a microservice-based system is probably to handle the security aspects at the network perimeter of the system. If only trustworthy actors can operate inside the network and the network is sufficiently secured against intruders, then one should be able to trust all actors inside the network’s perimeter implicitly. This approach is of course not exclusive to microservice-based architectures and the same limitations as with other architecture types apply.

For one, there is of course the question of how the network perimeter is actually secured. The first thing one should ensure is that the network segmentation is properly implemented and that the firewalls to external networks are properly configured. Only clients from within the system’s network should be able to directly access individual microservices. Only those components that must be accessible from outside the network should be exposed to other networks. In our example, the API Gateway should be the only part of the microservice architecture that is exposed to the internet. Outside access to third party systems like the middleware should be limited to specific networks or hosts.

Other than network segregation, one would also need to secure the outside-facing components against unauthorized access. There, one would need to consider which measures could be taken to secure these components:

One approach would be to harden the API Gateway. Depending on the underlying technology, this could be a question of adjusting the configuration of the existing application. It may also involve hardening the host operation system or the application platform the API Gateway runs on top of. In the end, the objective should be to minimize the attack surface presented to a potential attacker. But one problem with this approach is that only calls accessing the API Gateway would be covered. Third party systems would need to be secured separately.

An additional measure that could be taken would be the introduction of a Web Application Firewall (WAF). A WAF could inspect all incoming requests for attempts to exploit common or specific web application vulnerabilities (such as SQL or Cross-Site Scripting injection attempts) and prevent suspicious requests from being sent to the API Gateway. The scope of the WAF could also be expanded to cover third party systems if they provide web services to external clients.

To secure third party systems one might well be dependant on the support of the system’s vendor, if it can not be sufficiently covered with a WAF (or an XML firewall, if they expose SOAP interfaces). If the vendor does not provide enough security measures, one could adjust the network segmentation and firewall rules to be much more restrictive as compensation.

Even with the aforementioned measures implemented, relying solely on perimeter security remains problematic. If an attack manages to circumvent these measures, they will have unchecked access to the whole system. Since all actors within the network are considered trustworthy, an attacker within the perimeter will be trusted and allowed access to any component.

Therefore, a more deliberate approach should be applied if possible. To keep the perimeter security from being a single point of failure, a layered approach to security should be implemented. Multiple security measures should complement each other and components inside the network should be appropriately secured. This approach is called "Defence in Depth" [BS-DD] and should certainly be considered when designing the security measures of a microservice architecture.

Most security layers that should be considered in a microservice architecture are very similar to the ones used to secure regular network infrastructures. Therefore, we will not discuss them in too much detail here and concentrate on how they can help us in a microservice environment.

The first layer of security when considering Defence in Depth should likely remain a WAF or at least a regular firewall. The ability to detect common attack patterns before they reach the API Gateway will still be very helpful and can aid in guarding against automated attack attempts and vulnerability scans. Firewalls are especially important for third party systems for which the integration of additional security layers can not be realized without the support of the vendor.

A very basic security measure on the microservice layer would also be to require the use of secure connections (HTTPS/SSL) for all communication between microservices. The performance penalty that the use of SSL once implied can mostly be disregarded nowadays [ITFY]. Therefore, there are few reasons not to implement Transport Layer Security, at the very least to secure the connection and to verify its server-side. One issue would be the need to generate certificates for all microservices (preferably in an automated way). Here, projects like Let’s Encrypt could be evaluated to facilitate automatic certificate generation. To further improve the level of trust in the system, the use of client certificates could also be considered. But there, the certificate management is an even greater obstacle. The client certificates would need to be distributed and the revocation and reissue of certificates would need to be handled.

An Intrusion Detection and/or Prevention System (IDS/IPS) could be used to monitor the network traffic between the API Gateway and the microservices, the microservices and the external systems and in-between the microservices themselves. Since the interfaces between those services should be very well defined, the definition of the rules that detect irregular behaviour might actually be easier than in a normal network environment. An IDS could also help to detect unwanted dependencies within the architecture if it were to be configured to send out alerts if sensitive services are accessed from components that do not have explicit permission to do so.

The implementation of Log File Aggregation and Correlation should be considered for any microservice architecture with certain complexity, even without the security benefits in mind. Log Aggregation will help with understanding and retracing runtime behaviour and request paths over multiple services and is a very important tool to debug a microservice architecture. From a security standpoint it will also be indispensable to reconstruct what happened during a security incident and to follow an attackers actions after the fact. It should therefore be part of a Defence in Depth approach. It may also foil an attackers attempts to hide their traces by deleting or altering the log files of the services they compromised.

On the database layer, one should consider encrypting the data stores of services that keep sensitive data. Sensitive data is usually one of the main targets of attackers, either for exfiltration or to hold as ransom, and encrypting it while at rest would make such attacks much more difficult. It would not foil a ransomware attack encrypting the whole data store, were a strong backup policy would be much more helpful. But the encryption of sensitive data would also make these backups much less valuable as a target for attackers. In a microservice environment, the separation of data into service specific stores would enable us to be very selective in what we consider to encrypt and lower the barrier of entry considerably.

With a modern, software-defined networking [ONF-SDN] based approach to network segregation, one could prevent unauthorized components from accessing microservices that manage sensitive data on a network basis. Only those components that are given permission explicitly would even be allowed to connect to the sensitive services. This approach would make it much more difficult for an attacker to access sensitive data, since they would have to identify and compromise a privileged component before they could even attempt to breach the intended target.

When deciding what security measures should be implemented for any given system, considering the resources of the implementing organization and team, there are a few tools and processes that can support the decision making process.

To identify the components that should be considered high-value targets and therefore need additional protection, one should first identify what information handled or stored by the system would be sensitive to the owner and/or could be valuable to an attacker. Microservices that store or process sensitive data should warrant additional protection measures and should be prioritized when implementing security measures.

Thread Modeling is the process of decomposing an application into its components, determining and ranking threats to the application and its components and determining countermeasures and mitigation measures for those threats [OW-TM]. This process can help with identifying weaknesses and low-hanging fruits in the application’s architecture, but also with finding high-value attack vectors within the application. There are different approaches to Threat Modeling and accompanying tools for them, including these:

For a more fine-grained solution to authentication, one should also consider to authenticating the individual microservices to each other. Since most of the implementation approaches to service-to-service authentication require some sort of shared secret, we will discuss Secret Management in a separate paragraph. These authentication methods may also be viable solutions to authenticate clients to the API Gateway and a common approach to client and service-to-service authentication might be advantageous.

The most basic approach would certainly be to submit a username and password pair over HTTP Basic Authentication. But this would also make the use of Transport Layer Security for service-to-service communication strongly advisable. Otherwise, the authentication data is sent over an open channel and could be visible to an attacker.

Another common approach would be to integrate the microservices into an (existing) Single-Sign On (SSO) solution. Incorporation SSO into the service-to-service communication would also help with the authorization of requests. There are a host of open and proprietary SSO solutions and we only want to introduce a few widely used ones:

As already discusses in Transport Security, the use of client certificates for service-to-service connections would enable a very strong authentication, but would entail a lot of effort to get the certificate management right.

To ensure requests were sent by a known service and were not altered during transport, one could implement Hash-based Message Authentication Codes (HMAC) [IT-HMAC]. Were the messages sent from one service to another and a secret key known to both services would be run through a cryptographic hashing function and the resulting hash would accompany the message (i.e. as a header). The receiving service can then reproduce the process with the received message and check if the result matches the given HMAC, thus verifying both the integrity and authentication of the message.

Another rather simple approach to authenticating services to each others would be the use of API keys. There are various implementation approaches to API keys. They can be implemented as simple shared secrets, as keys for a HMAC as discussed above or as public and private key pairs.

When discussing authorization within a microservice architecture, we should both consider service-to-service authorization and the propagation of the external clients authorization principal. Usually, the same tools can be used to solve both problems.

One potential stumbling block most approaches have in common is the Confused Deputy Problem [Hardy]. Where an actor tricks a service to make requests to another service they should not have access to. The second service would only see that the request is coming from the first service and assume it to be authorized to execute the request. Therefore, it should be investigated if it is possible to pass the original caller’s authorization principal down to downstream services if such a scenario is viable.

Most Single Sign-On solutions allow the propagation of roles. Therefore, the integration of microservices into the SSO solution discussed above could be a good approach to authorization within a microservice architecture.

The OAuth 2.0 protocol was specifically designed as an open standard to allow secure authorization in an HTTP environment and is therefore well suited for REST-based microservices. It is widely used and there is very good support in variety of frameworks (i.e. Spring Security) because of that.

JSON Web Token (JWT) is a JSON-based open standard for access tokens. The tokens are signed by the issuing server’s key to verify their legitimacy. Tooling support might not be as widespread as with OAuth, but due to their compact design, they could be a good basis for a custom authentication implementation.