Data Engineer / OSS Engineer / DevOps
… and general stuff
Hello! Welcome to my blog! 👋
I hope to share insights from my experience about Software & Data Engineering, Open-Source shenanigans, Cloud and DevOps related fun stuff! Here you should find practical nuggets of info that will help you on your journey in the wonderful world of IT (does anyone still call it IT? 🤔).
Intro A while back we had a request for a “kubernetes events input” for the data streaming tool Bento - at the time an etcd input connector was being worked on and because kubernetes uses etcd as the default datastore, the etcd input component should enable streaming kubernetes events. This blog post walks through the steps to connect a minikube based local kubernetes cluster’s etcd with Bento. ⚠️ WARNING ⚠️ THIS FOR DEV/TESTING ONLY - EXPOSING ETCD LIKE THIS IS A SECURITY RISK ...
Intro Standard SQS messages are limited in size to 256 KB 😱 - but you can use the Amazon SQS Extended Client to send larger messages up to 2 GB. These libraries store the messages larger than 256 KB in S3 and send a reference to the stored object in the SQS queue. Amazon provides two client libraries, for Python and Java. There are a few unofficial ones knocking around on github: Javascript and Go. ...