Bento

I am trying to work in more Test-Driven-Development style - which effectively means writing the tests for some functionality before writing the implementation. When the functionality requires connecting to another service, I think that it makes sense to start with an integration test. Docker is a great tool for spinning up an ephemeral service to use in the test. Recently I found myself wanting to implement Kafka <–> Bento (stream processor) connection using OAUTHBEARER. To use OAUTHBEARER effectively you should also set-up that connection with tls/ssl. ...

A while back we had a request for a “kubernetes events input” for the data streaming tool Bento - at the time an etcd input connector was being worked on and because kubernetes uses etcd as the default datastore, the etcd input component should enable streaming kubernetes events. This blog post walks through the steps to connect a minikube based local kubernetes cluster’s etcd with Bento. ⚠️ WARNING ⚠️ THIS FOR DEV/TESTING ONLY - EXPOSING ETCD LIKE THIS IS A SECURITY RISK ...

Standard SQS messages are limited in size to 256 KB 😱 - but you can use the Amazon SQS Extended Client to send larger messages up to 2 GB. These libraries store the messages larger than 256 KB in S3 and send a reference to the stored object in the SQS queue. Amazon provides two client libraries, for Python and Java. There are a few unofficial ones knocking around on github: Javascript and Go. Setup using Terraform Ok so, we only need a couple of AWS resources: ...