From Ops to Dev: A Dual-Layer Programmable Data Lake With Kubernetes and WebAssembly-Based Offloading

Abstract

This paper presents a dual-layer integration approach that evolves the data lake from a passive repository into a platform that developers can directly program. While its building blocks-WebAssembly, io_uring, and Kubernetes-are established technologies, the contribution lies in integrating them through an eDMA abstraction that bridges a WebAssembly sandbox with high-performance asynchronous I/O for programmable storage. Although modern object stores expose limited server-side operations, control typically resides with infrastructure operators, preventing data consumers, such as data scientists, from leveraging computing resources near data. This results in repeated inefficiency when large datasets are shipped across a network for processing. To address this, we design a WebAssembly-based "Connected Data Lake" framework that enables data consumers to deploy and execute storage applications in a WebAssembly sandbox within storage nodes, so that custom APIs can be offloaded onto storage nodes. Using WebAssembly's portability, we abstract and integrate Linux's io_uring-based I/O and offloadable operations to be invoked on storage nodes, delivering high-performance, low-resource I/O on general-purpose hardware. We evaluated our prototype in two complementary experiments. First, in a log filtering scenario using approximately 29.61 GiB of raw data, the proposed data operation offloading reduced the network transfer volume by 99.978% and achieved a 45.5% faster end-to-end processing time compared to the conventional "compute-pulls-data" model, even with WebAssembly sandbox overhead. Second, in an S3-compatible object I/O benchmark against a conventional object store, our prototype improved GET throughput by 10.1-22.2% for small-to-medium objects, PUT throughput by up to 438.6% for small objects, and STAT throughput by 15.9%, while consistently reducing latency across all PUT percentiles for objects up to 1 MB and lowering CPU per operation by 24-86% for GET and 35-77% for PUT, with a 4.0- 6.2 & times; smaller memory footprint. These results show that data consumers can safely program object storage and that the architecture provides a pragmatic path to data lake implementations.