Peplum: F/OSS Distributed Parallel Computing and Supercomputing At Home With Ruby Infrastructure (ecsypno.com) 20
Slashdot reader Zapotek brings an update from the Ecsypno skunkworks, where they've been busy with R&D for distributed computing systems:
Armed with Cuboid, Qmap was built, which tackled the handling of nmap in a distributed environment, with great results. Afterwards, an iterative clean-up process led to a template of sorts, for scheduling most applications in such environments.
With that, Peplum was born, which allows for OS applications, Ruby code and C/C++/Rust code (via Ruby extensions) to be distributed across machines and tackle the processing of neatly grouped objects.
In essence, Peplum:
- Is a distributed computing solution backed by Cuboid.
- Its basic function is to distribute workloads and deliver payloads across multiple machines and thus parallelize otherwise time consuming tasks.
- Allows you to combine several machines and built a cluster/supercomputer of sorts with great ease.
After that was dealt with, it was time to port Qmap over to Peplum for easier long-term maintenance, thus renamed Peplum::Nmap.
We have high hopes for Peplum as it basically means easy, simple and joyful cloud/clustering/super-computing at home, on-premise, anywhere really. Along with the capability to turn a lot of security oriented apps into super versions of themselves, it is quite the infrastructure.
Yes, this means there's a new solution if you're using multiple machines for "running simulations, to network mapping/security scans, to password cracking/recovery or just encoding your collection of music and video" -- or anything else: Peplum is a F/OSS (MIT licensed) project aimed at making clustering/super-computing affordable and accessible, by making it simple to setup a distributed parallel computing environment for abstract applications... TLDR: You no longer have to only imagine a Beowulf cluster of those, you can now easily build one yourself with Peplum.
Some technical specs: It is written in the Ruby programming language, thus coming with an entire ecosystem of libraries and the capability to run abstract Ruby code, execute external utilities, run OS commands, call C/C++/Rust routines and more...
Peplum is powered by Cuboid, a F/OSS (MIT licensed) abstract framework for distributed computing — both of them are funded by Ecsypno Single Member P.C., a new R&D and Consulting company.
With that, Peplum was born, which allows for OS applications, Ruby code and C/C++/Rust code (via Ruby extensions) to be distributed across machines and tackle the processing of neatly grouped objects.
In essence, Peplum:
- Is a distributed computing solution backed by Cuboid.
- Its basic function is to distribute workloads and deliver payloads across multiple machines and thus parallelize otherwise time consuming tasks.
- Allows you to combine several machines and built a cluster/supercomputer of sorts with great ease.
After that was dealt with, it was time to port Qmap over to Peplum for easier long-term maintenance, thus renamed Peplum::Nmap.
We have high hopes for Peplum as it basically means easy, simple and joyful cloud/clustering/super-computing at home, on-premise, anywhere really. Along with the capability to turn a lot of security oriented apps into super versions of themselves, it is quite the infrastructure.
Yes, this means there's a new solution if you're using multiple machines for "running simulations, to network mapping/security scans, to password cracking/recovery or just encoding your collection of music and video" -- or anything else: Peplum is a F/OSS (MIT licensed) project aimed at making clustering/super-computing affordable and accessible, by making it simple to setup a distributed parallel computing environment for abstract applications... TLDR: You no longer have to only imagine a Beowulf cluster of those, you can now easily build one yourself with Peplum.
Some technical specs: It is written in the Ruby programming language, thus coming with an entire ecosystem of libraries and the capability to run abstract Ruby code, execute external utilities, run OS commands, call C/C++/Rust routines and more...
Peplum is powered by Cuboid, a F/OSS (MIT licensed) abstract framework for distributed computing — both of them are funded by Ecsypno Single Member P.C., a new R&D and Consulting company.
Ruby? (Score:2, Informative)
And with ruby being so slow, you need a cluster to run something that would have been fine on a single machine if it were written in a more efficient language.
Re: Ruby? (Score:3)
You beat me to it. Maybe they'll add Perl, too.
And COBOL... (Score:2)
Seriously around me Ruby is gone... projects are cutting number of languages used and Ruby is one of the victims...
Re: (Score:2)
Re: (Score:2)
You've kinda missed the entire objective of the project here.
To get attention by wasting time? Mission accomplished!
Every time I have to screw with something written in Ruby it's a headache just getting the tools going.
Re: (Score:2)
Re: (Score:2)
I think he means for this particular project. Though in general, I think ruby doesn't really have any worthwhile niche, which it shares in common with perl.
Nightmares (Score:2)
The very thought of this computing framework is going to keep me up at night at best. At worst it is nightmare fuel. Why did anyone ever bring this into the light of day?
Re: (Score:2)
Guilty btw, but why nightmare fuel?
Re: (Score:2)
Lets just say
At least it's in Ruby and not something that AWS will bring to my researchers in 6 months to tell them that they can do all their simulations on AWS and don't need the local HPC cluster.
They'll find out that AWS are wrong real fast, but I'll have to clean up the mess after.
Sounds interesting, I hope it delivers, great job. (Score:1)
...but there's no way there aren't tons of other solutions for this already, for Python alone, that don't require you to learn how to compile C++ or Rust to include bindings into fecking Ruby.
Ruby seems late, but what's the selling point? (Score:1)
AI has mostly gone the Python route. Cool languages went to Rust. K8s is mostly Go.
I don't understand the market for this. Hadoop was usually for giant batch jobs. Mesos came and went.
What's needed is a modern replacement for Rock Clusters: an integrated, manageable HPC Linux distro.
This is um. Ok. I guess? (Score:2)
The standard solution in the actual supercomputing world is the venerable Message Passing Interface, of which the most accessible implementation is probably OpenMPI.
Note that this distinct from OpenMP, which is a multi-cpu threading shared memory library.
All of these solutions need to be written into the applications that use them, interchangeably with any other threading systems they may use. The quality of the integration varies greatly, and can heavily affect processing outcomes... but also the way they'
Re: (Score:2)
I don't imagine anyone using this for other tasks although the project was just born and high-performance code can be added via native C/C++/Rust extensions to the project. That'd go into Cuboid though, not Peplum, Peplum would just benefit
Re: (Score:2)
yeah i see how the framework is set up, it does make sense to separate it out.
As you say, the divisible workload is the key... but that makes me wonder why it's necessary in this particular implementation.On the other hand, if you're enjoying the work and someone finds it useful... well that's fine then.
From my perspective there's an ongoing conversation around HPC and other types of distributed computing by people who don't know what the words computational complexity even mean... and so I'm used to having
Re: (Score:2)
Re: (Score:2)
This is one of those forever problems. It's surprisingly difficult to get it right.
Imagine... (Score:1)
...a beowolf cluster of these