Dinda Sabela
Student, Universitas Gadjah Mada, Indonesia
My experience with ILUM has highlighted many unique features and advantages. ILUM has proven to be an invaluable platform for managing distributed computation with Kubernetes-based orchestration. ILUM, with this kind of architecture ensures that tasks are efficiently orchestrated across multiple nodes, for this case I was joining 6 HPCs to build distributed computation. In this environment, ILUM is able to support local computation within individual nodes. This makes my thesis experiment run on just one platform in the same capacity of resources.
As an information, my thesis research had the objectives to compare 3 computing environments- local Python, local Spark, and cluster Spark- to analyze their effectiveness in managing and processing my data also in the usage of resources for each environment. I was able to test and perform these comparisons within the same ecosystem. Having the same “ground level” across all environments made my research more accurate.
Another standout feature carried by the ILUM platform is its storage layer powered by MinIO and its module driven by Jupyter Notebook. With its compatibility with ILUM, I don’t have to worry about setting up the storage for my data processing, its logs, and where it can be run in. This makes handling large datasets and tracking logs incredibly hassle-free.
I also heavily relied on ILUM Workloads feature when it came to distributed computation that made me have to make a group of Spark jobs. Using this feature, I could easily monitor the cluster performance, tracked resource utilization, and ensured everything was running smoothly. The one that makes it even better is the simplicity of ILUM’s interface. It eliminates the need for manual coding to create and start Spark sessions, making the process accessible, even for those who don’t really understand the technical expertise. With the auto-configuration feature, all I had to do was just specify the number of drivers, executors, and cores I needed, and with just 1 “click”, I could start a Spark session. Once the session was running, the effort for managing and adjusting was just so simple. This saved so much time and effort.
Although ILUM’s documentation, somehow, is still quite limited, ILUM’s customer support team is really incredible. Anytime I had questions or ran into issues, their responses had been fast and very helpful. It is reassuring to know that even if the documentation doesn’t have the answers, there’s always someone ready to assist.
As a conclusion, I would like to appreciate ILUM, which has been such a fantastic tool for managing my distributed computation using Kubernetes, starting from the built-in storage solution, user-friendly interface, and simplified-management for the Spark session.
___
DTETI UGM bekerja sama dengan Ilum, sebuah platform dengan tagline “Free Data Lakehouse for a Cloud Native World” yang mendukung pengelolaan data secara efisien melalui arsitektur cloud-native. Kolaborasi ini memungkinkan mahasiswa DTETI untuk menjalankan penelitian berbasis data menggunakan fitur-fitur canggih seperti manajemen cluster, eksekusi tugas dengan Spark, dan monitoring terpusat. Dengan kerja sama ini, DTETI memfasilitasi mahasiswa untuk memperoleh pengalaman praktis dalam teknologi data terkini, mendukung inovasi di bidang teknologi informasi dan rekayasa data.