Real-time Data Science Workflows on Kubernetes.

During my army service, our unit heavily relied on real-time, continuous data processing. Every few minutes, we faced the challenge of handling large volumes of new data, requiring the execution of algorithms to extract valuable insights.

Running algorithms at regular intervals is manageable, but dealing with multiple algorithms across various data streams and different cycles gets complicated. To streamline this complexity, we, like many in the industry, utilized orchestration software that automates the majority of the laborious tasks.

Dissatisfied with our existing orchestration platform, I took it upon myself to set up Argo-Workflows, a task that proved challenging within the constraints of the army’s closed system. Afterward, I rigorously tested its usage and documented my experiences, including key examples that I found particularly interesting.

Disclaimer: This guide is unclassified and represents my own personal work, allowing me to share it freely.

Click here to view the guide!