How to Encrypt and Decrypt Kafka Message using Custom (De)Serializers

Sensitive data always need to be handled with extra careful. Thus, in some cases, we need to encrypt before delivering message to Kafka topic.

Sharing is caring!

Overview

Today’s post still has correlation with two previous articles, which talk about Kafka and security. Even though we already guarded our Kafka with authorization just like in the first post; and we cautiously reserved our client with authentication and roles-based access like in the second post, in my own perspective it is not enough. Because occasionally our message contains sensitive information, such as ID number, Credit Card number, etc. We have to be careful to store those kind of information into memory or disk as a plain text. The last thing you want is your customer sensitive data spread all over the internet. It will destroy your company reputations and can impact in financial loss. Therefore, I will address message encryption and decryption in Kafka.

Approach

There are many ways to do this kind of approach. First, there are various encryption algorithms. We have two types of keys, which are symmetric and asymmetric. AES, 3-DES, Blowfish are the examples of symmetric encryption. For asymmetric, we have RSA, PKCS and DSA as examples. We will use symmetric keys for Kafka message encryption. The reason is because there is a possibility one message can be consumed with multiple consumer group. Meanwhile, asymmetric use public key to encrypt and private key to decrypt. It will raise a question, which consumer-group’s public key would be used for encryption? Consequently, we need to share the same key for encryption and decryption to eliminate confusion and reach some consensus between consumer-groups. For this, I pick AES as encryption algorithm due to its acceptance worldwide.

Next, the way we will encrypt the message. Of course there are some options to do this. We can encrypt sensitive value, set the value into POJO using setter method, then send the message to Kafka. Another way is we optimize Kafka Serializer/De-Serializer mechanism. Thus, we send POJO to Kafka. Kafka then will convert to JSON, encrypt it then store into topic. I will choose the latter approach in this article.

Why do we need to convert to JSON first then encrypt? Why don’t we serialize object using ByteArrayOutputStream since the encryption only need byte after all? The answer is because our consumer should not be limited with Java client. We want every service that will subscribe to our topic, regardless its programming language, can still process the message. Because JSON is very common across languages, we should find no issue with this approach.

Sample Code

We need three additional java files to fulfill the approach and two configuration changes.

AES.java

AESSerializer.java

AESDeserializer.java

KafkaConfiguration.java

Conclusion

In this example, I hard coded password and salt values. Of course, it is not a good practice. You can shift those two values inside properties file with Jasypt encrypted values, or better to put them into Hashicorp Vault. The samples are already written in my previous posts.

As a reminder, every approach has its own benefits and drawbacks. Tighten up security, it will drain performance. You should know the best option for your solution. As usual, the code is on my GitHub repository, inside branch encrypt (https://github.com/ru-rocker/spring-kafka-sasl-scram/tree/encrypt).

References

Author: ru rocker

I am a professional software developer with more than 10 years experiences. I am a certified Java Developer (SCJP and SCWCD). However, In the recent months, I have more interest in DevOps and start to become a polyglot developer. Python and Go-lang become my favorite programming languages besides Java.

Leave a Reply

Your email address will not be published. Required fields are marked *