[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Kafka consumer, is there a way to filter out messages using key only?


I'm afraid that there is no out-of-the box solution for this, but what you could do is to generate from KeyedDeserializationSchema some marker (Optional, null value...) based on the message key, that would allow you later to filter it out. So assuming the Optional solution the result of KeyedDeserializationSchema#deserialize could be Optional.empty() for invalid keys and Optional.of(deserializedValue) for valid keys.



On 18/12/2018 20:22, Hao Sun wrote:
Hi, I am using 1.7 on K8S.

I have a huge amount of data in kafka, but I only need a tiny portion of it.
It is a keyed stream, the value in JSON encoded. I want to avoid deserialization of the value, since it is very expensive. Can I only filter based on the key?
I know there is a KeyedDeserializationSchema, but can I use it to filter data?

Hao Sun
Team Lead
1019 Market St. 7F
San Francisco, CA 94103

Attachment: signature.asc
Description: OpenPGP digital signature