I might have missed it but is there a particular advantage Rust has over C in this specific use case? Performance, ease of interoperability, etc. or is it preference for Rust over C?
Rust also has a much nicer package ecosystem. It's pretty trivial to wrap existing C libraries as well so I find that I can get up and running much faster rather than messing around with makefiles.
Yeah, this is the killer feature. cmake is a nightmare. Cargo does what it needs to do and gets out of your way, and provides easy access to the whole ecosystem.
There are lots of advantages in general, especially around multithreaded code, and doubly so for anything with relatively simple data parallelism, where you can just use the rayon library.
Another advantage is that there are Rust packages for Ruby interop -- the article ended up using ruru. There are similar libraries for Python interop -- I believe the most actively developed right now is PyO3.
I'm sure people are sick of Rust evangelists on HN but it really is a wonderful language, and in certain cases it really does seem to empower people to do things they wouldn't have been able to do otherwise.
Was doing some work of converting [0,1] signals to 16-bit PCM data with 550Hz tone in numpy.
Python version took ~15 minutes to generate 5,000 4 second files. Broke out the inner loop into Rust with FFI via ctypes and cut that time to ~10 seconds with nearly identical code.
Yeah, I've heard of Julia before but looks like it doesn't have a very clean[1] FFI story. I'm not super-interested in spinning up a full-blown interpreter for an inner loop. Plus I already had Rust installed :).
Seriously though Rust was 43 lines of code + cargo build vs 33 lines of Python and a simple C FFI so it was pretty straightforward to drop in.
I think the whole idea behind using Julia in this situation would be that you get good enough performance "out of the box" so that you don't have to resort to using an FFI in the first place.
this is probably a dumb question but did you just compile the rust binary and then expose the functions? how? I don't remember but does rust face header files?
This is fine for new projects, but OP already has a large Ruby codebase. "Just rewrite the whole thing in Julia and you won't need FFI" isn't a reasonable ask.
Rust's ability to compile to a small C-compatible binary is definitely an advantage here, although Julia will have similar capabilities soon.
A lot of this stuff has worked for ages, but it's rough in various places and needs a better interface. I'm not sure to what extent exporting C functions is actually supported in that wrapper script, or whether you'll need to fiddle with compiler options.
No, but you would have to do with subpar/immature third party ecosystem AND you'd have to port all your Python code to Julia before you can get to resort entirely to it.
So many young Hackers have been asking for the easiest way to hack an email,
Do you know how many email services there are, how many kinds of mail servers, how many forms of custom security?
In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service.
It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate.
I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally.
So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation.
I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality.
So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
A 100x difference to native and having an inner loop in Python code sound exceptionally slow for a numpy app. I guess you could not leverage numpy array operations?
Nope, I agree that base numpy operations are snappy. However I needed to both multiply by a sin wave of 550hz(changing input) and taper the edges of the signal on the transition from 0->1 to not generate a ton of harmonics.
Have you looked at cython? Might have been a great compromise.
I saw similar gains in my code eventually, but the process to get there was very smooth. At first I could just use original python code and covert it piece by piece (of course sometimes you need to refactor). You can drop GIL too.
Not just Python, JVM langs too (and other runtimes w/ fast C interfacing, e.g. node w/ neon). JNI is actually incredibly easy with Rust. I've even used some advanced features of the JVM via JVMTI and exposed it via JNI all in Rust as a learning exercise [0]. Contemplating writing a fuzzer using similar tech soon.
It's a lot easier to write Ruby. For lots of things Ruby also had much more mature libraries. Of course there are other benefits to writing everything in Rust as well. Everything is a trade off. It's quite disappointing . I would have liked free lunch...
Not quite. Rust embraces "zero cost abstractions" which means you can build up pretty high level abstractions without losing any speed. TBH I would much rather write Rust vs Ruby.
I'm talking about developer productivity not computing efficiency. Those high level abstractions are nowhere near as easy to use or easy to understand as they are in higher level languages.
Depending on the application, I might agree. However, if there were a Ruby-with-types language (like TypeScript) I would prefer that over the other two options.
I think you might have that slightly incorrect. Rust is a high level language, capable of C and C++ speeds. As you get to know the language, I find it just as productive as Java, Ruby and Python. In some ways more so, because the compiler catches so many bugs before you even run (not to imply you won’t have bugs, just different ones).
I find Rust to be about as productive as Scala (which I find to be generally more productive than Java). Both are generally functional, with a powerful type system, with a lot of flexibility in the level of abstraction, with an expansive preexisting ecosystem you could shim in (C or JVM).
However, I don't think Rust is a more productive general programming language than Ruby/Python. I think Rust is more productive when solving systems-level concerns, but most problems aren't systems problems. Crystal is also looking to be a nice compromise for a productive, C/C++ level performance language (if the ecosystem catches up).
That's not really my experience, my Rust projects tend to take a lot longer to gain critical features while projects I develop in Go can be MVP way faster.
Rust is quite a good language though, I'll admit that, I'm currently learning it by way of writing a kernel in it. But there are more productive languages out there.
I wonder if your what your working on is a major contributor to the complexity and speed of the project.
When working that low level, there are a lot of lifetime issues your going to need to manage, and that can definitely make Rust have a higher cognitive load.
One major roadblock I had lately was to manage paging in interrupts. Rust doesn't really like static globals but that is the only way to sanely manage paging in interrupts. So I spent probably a day to figure out the correct way to write a small interface to handle a global kernel state variable.
There is also plenty of unsafe stuff that won't go away or stuff that is unsafe not because of the code but because x86. For example, you can't properly handle a null pointer in rust, however, at low-level, the pointer 0x00 is completely valid and I hate to have to waste that address because Rust and LLVM won't allow it.
The major reason I even bothered to learn Rust for this way because I didn't want to manage all of this in C on top of having only printf as my debugging tool.
Rust as productive as Python lol ... things you read on HN. Even if the compiler catches more bugs it doesn't mean it's more productive, I give you a problem and you have to solve it in Python and Rust, Python problem will be solved much faster because the language has higher construct for day to day programming.
It depends on what you mean. You're optimizing for time to first solve; what about when you find bugs that would have been caught at compile-time in Rust? What about existing codebases, rather than writing new code? It's not that simple of a question.
I find personally that if there are libraries to help me out, I'm only slightly slower in Rust than I am in Ruby, but then there's no debugging time in Rust, and invariably, there will be tons of work I have to do later with the Ruby to shake out bugs.
FWIW my last project was in Rust, and the current one is in Python (with which I have more experience), mostly due to Python having some useful libraries for what is mostly an exercise in gluing together different external systems.
Despite the fact that I chose to do this in Python, I'm not at all sure I wouldn't have had a working solution faster in a language that had a good type system and a error handling strategy that made errors explicit and forced them to be handled. I'm almost certain that writing in Rust would have ended up with something that was more reliable in the long term.
(FTR I imagine that Go may also have been a reasonable choice for this kind of work, but I haven't used it).
Rather than building from scratch, you'd probably just want to start by replacing bits of MRI with Rust one component at a time (assuming MRI is appropriately modularized) and see how far that takes you. :P
Does anyone know of ruru’s status? I’ve used it quite a bit and love it, but haven’t seen any action in the repo in a while. I also pinged the Gitter channel a few weeks ago on the topic and got no response. Its a great tool; would hate to see it die.
If one needs speed one does everything it takes. That said, I read the code of the example. It's short and even if I don't know Rust I've been paid to write C in the 90s. They look similar enough.
I see we're back to stuff like
let ptr = path.as_ptr();
let c = unsafe { *ptr.offset(i) };
if c == SEP {
return &path[(i + 1) as usize..end];
};
It's as low level as it can get. I remember that I wrote a small web app in C in 1994 but I wrote the next one in Perl and never looked back. It was some hundreds lines, countless core dumps and many hours vs much less code and pain and time.
Again, Ruby and Python are all about connecting those small pieces of C code that implement their builtin and library methods / functions. Writing them in C or Rust makes no difference. I personally won't write code in those kind of languages again unless I find myself in a scenario where CPU time is worth more than my time. Maybe we'll be back to that with functions in the cloud billed by the millisecond. Programming is going to be a pain again. At least modern languages like Rust don't crash like C and have a better concurrency story. Meanwhile I'll keep using Ruby and Python with C or Rust extensions from GitHub written by somebody else (a big thank you!), and Elixir where concurrency matters.
Of course it looks like C, it's using a C interface to interface with Ruby. Normal Rust does not look like that. Your comment is quite frankly entirely pointless.
Per my comment above linking the real implementation: the comment is perhaps misleading about the cause but its claim is largely true. Rust code tends not to look like this.
Well, unsafe Rust does tend to look like this. You were claiming that it is only for interfacing with C code, but unsafe is also used for performance, as in the snippet above. You were IMHO incorrectly dismissing the parent comment based on that. (On the other hand, the code in question could probably rewritten in safe Rust while still avoiding the bound checks. It would be less straight forward though.)
let mut bytes = path.bytes();
if let Some(end) = bytes.rposition(|&x| x != SEP) {
if let Some(idx) = bytes.rposition(|&x| x == SEP) { &path[idx..end] } else { &path[..end] }
} else { "" }
There are other ways to do this, like using a single iterator where we scan manually, but I just threw this together to demonstrate that this code needn't use unsafe, & by extension needn't use pointers. All that unsafe stuff is more so that if you need to drop down to C from safe-Rust you can instead use unsafe-Rust
I don't see how Rust vs C makes no difference. Both languages have trade-offs and I'm pretty sure that Rust advocates would claim the increased safety of the Rust part as a big advantage, and it will certainly reduce those core dumps. Probably the reason there are so many of those advocates is because they don't have to write such error-prone low-level code. Expressive code is more fun, in my experience.
For what it's worth, I don't think that line needs to be unsafe. I think you can just do this:
let c = s.as_bytes()[i];
In unoptimized/debug mode that's going to do array bounds checks that the unsafe code doesn't, but I think the compiler is smart enough here to see that you're doing the exact same checks in the loop condition too, and it can optimize them out in release mode?
Am I really reading it right that he went to the trouble of doing all this to speed up the string parsing of file paths??? Talk about barking up the wrong tree.
I guess its cool to use something as hipster as Rust to speed up as something as hipster (albeit waning in popularity these days) as Ruby.
However, a better approach would be a straight forward implementation of caching (memcache/redis/static CDN) or to simply rewrite that part of the code to not rely on constantly parsing file paths? Or use something like https://github.com/google/re2, you probably won't be able to do any better than that when doing any kind of string matching and parsing.
If you're concerned at all about performance you shouldn't be doing anything with the file system period.
Author here. Development is indeed on hold for now. However, Crystal core team has plans to build a DSL into Crystal to allow for creating Ruby native extensions.
There are some technical limitations with how I handled defining functions, which has also become obsolete with recent changes to Crystal. However, there is another approach (macro based) demonstrated at https://github.com/spalladino/crystal-ruby_exts. The macro approach is what is needed.
At time of first development, this was just an experiment and I didn't feel like redo-ing it this way. Now that core team has plans for similar functionality, I'd rather not finish a half baked solution that distracts from what theirs will likely be.
78 comments
[ 20.7 ms ] story [ 321 ms ] threadAnother advantage is that there are Rust packages for Ruby interop -- the article ended up using ruru. There are similar libraries for Python interop -- I believe the most actively developed right now is PyO3.
I'm sure people are sick of Rust evangelists on HN but it really is a wonderful language, and in certain cases it really does seem to empower people to do things they wouldn't have been able to do otherwise.
Was doing some work of converting [0,1] signals to 16-bit PCM data with 550Hz tone in numpy.
Python version took ~15 minutes to generate 5,000 4 second files. Broke out the inner loop into Rust with FFI via ctypes and cut that time to ~10 seconds with nearly identical code.
Seriously though Rust was 43 lines of code + cargo build vs 33 lines of Python and a simple C FFI so it was pretty straightforward to drop in.
https://docs.julialang.org/en/release-0.4/manual/embedding/
I can write in anything I want since every language supports C ABI for system calls.
I don't want to have to learn a whole new language and new API at the same time.
Also a cursory glance at Julia's numpy support looks like it requires deep copies which is painful when you want things to be fast.
I'm sure it's a great language but needing to drag in a whole new VM means it doesn't fit my needs.
It removes the complexity of needing to do the FFI at all.
It look me 3 lines of code to import the dll, assign the signature and call the function, I'd hardly call that complex.
Rust's ability to compile to a small C-compatible binary is definitely an advantage here, although Julia will have similar capabilities soon.
Do you have a link to further info on that?
A lot of this stuff has worked for ages, but it's rough in various places and needs a better interface. I'm not sure to what extent exporting C functions is actually supported in that wrapper script, or whether you'll need to fiddle with compiler options.
In my time, I've hacked numerous email servers and services, usually incidentally to compromising the rest of a company or entity. It's always been different. Sometimes it´s as simple as replacing a username in a URL. Other times I've had to compromise the entire support system for an email service.
It varies. It takes imagination. There's no formula. There's no ¨teaching to hack email¨. There's just understanding how systems work and learning to extrapolate. I will reluctantly say that a public favorite seems to be guessing ¨secret answers¨ to ¨secret questions¨ needed to reset passwords on some systems (For Guccifer, hacking email is easy — prison is hard). I never personally used this technique, since it felt like cheating. It's hard to explain why. But certain passwords, once discovered, made sense for certain people (Ih8K1ds for an extremely unpleasant lady at [redacted],SPQR for a CIA officer [on a system with very low password standards]).So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via compositehacks@gmail.com. I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
So many people always think i only work for big companies, that i dont work for private individuals, here is my mail """"""compositehacks@gmail.com,""""" feel free to contact me for any hacking services you need, i'll do your job professionally.
So many people always ask me if i'm a Hacker or a Cracker, as a matter of fact, I don't really care. You can consider me whatever you like - it's entirely subjective. I've been called a white hat, gray hat, and black hat, depending on the attitude and perspective of the writer, and it's not my place to insist that one or the other is correct. I encourage readers to reach conclusions from their own mind and feelings, not arbitrary segregation.
I do what I do, and leave it to others to document it - or not - in the way that suits them best. Distinctions like these work for role playing games, and it's kind of like asking "Is Adrian Lamo True Neutral?" - an interesting intellectual exercise, but with no particular relevance to reality. So many people always think i only work for big companies, that i dont work for private individuals, For urgent requests for professional Hack Advice Or Hack Services, You can contact me directly via """""compositehacks@gmail.com"""". I'm also trying out a new beta ticketing system, which can be accessed at "Composite Hacks & Co"
I'm very tempted to compare that with LuaJIT and/or its FFI[0] (just to use structs, not to call any native code) just to see the results.
[0] - http://luajit.org/ext_ffi.html
0 - https://github.com/cretz/stackparam
I mean, what's the benefit you could get from FFI Rust code into Ruby that you cannot get by directly writing Rust?
[0] https://crystal-lang.org/
YMMV.
The mental effort of writing Erlang is an order of magnitude lower compared to Rust. While Rust is more fun, it does not and can not come close.
I'm not much for Java/Python/Ruby. I don't see those as productive as Erlang so Rust may be closer.
However, I don't think Rust is a more productive general programming language than Ruby/Python. I think Rust is more productive when solving systems-level concerns, but most problems aren't systems problems. Crystal is also looking to be a nice compromise for a productive, C/C++ level performance language (if the ecosystem catches up).
Rust is quite a good language though, I'll admit that, I'm currently learning it by way of writing a kernel in it. But there are more productive languages out there.
When working that low level, there are a lot of lifetime issues your going to need to manage, and that can definitely make Rust have a higher cognitive load.
On a side note have you looked at this blog series? https://www.tockos.org/blog/2017/apsys-paper/
They’re definitely paving an interesting path in the kernel space for Rust.
There is also plenty of unsafe stuff that won't go away or stuff that is unsafe not because of the code but because x86. For example, you can't properly handle a null pointer in rust, however, at low-level, the pointer 0x00 is completely valid and I hate to have to waste that address because Rust and LLVM won't allow it.
The major reason I even bothered to learn Rust for this way because I didn't want to manage all of this in C on top of having only printf as my debugging tool.
I find personally that if there are libraries to help me out, I'm only slightly slower in Rust than I am in Ruby, but then there's no debugging time in Rust, and invariably, there will be tons of work I have to do later with the Ruby to shake out bugs.
YMMV.
Despite the fact that I chose to do this in Python, I'm not at all sure I wouldn't have had a working solution faster in a language that had a good type system and a error handling strategy that made errors explicit and forced them to be handled. I'm almost certain that writing in Rust would have ended up with something that was more reliable in the long term.
(FTR I imagine that Go may also have been a reasonable choice for this kind of work, but I haven't used it).
By leveraging this you could quickly re-write some slow parts in Rust, gaining the benefits with only a few days work. It's certainly worth exploring.
https://github.com/steveklabnik/ruby/tree/rust
I see we're back to stuff like
It's as low level as it can get. I remember that I wrote a small web app in C in 1994 but I wrote the next one in Perl and never looked back. It was some hundreds lines, countless core dumps and many hours vs much less code and pain and time.Again, Ruby and Python are all about connecting those small pieces of C code that implement their builtin and library methods / functions. Writing them in C or Rust makes no difference. I personally won't write code in those kind of languages again unless I find myself in a scenario where CPU time is worth more than my time. Maybe we'll be back to that with functions in the cloud billed by the millisecond. Programming is going to be a pain again. At least modern languages like Rust don't crash like C and have a better concurrency story. Meanwhile I'll keep using Ruby and Python with C or Rust extensions from GitHub written by somebody else (a big thank you!), and Elixir where concurrency matters.
https://doc.rust-lang.org/src/std/path.rs.html#1801-1808
I don't see how Rust vs C makes no difference. Both languages have trade-offs and I'm pretty sure that Rust advocates would claim the increased safety of the Rust part as a big advantage, and it will certainly reduce those core dumps. Probably the reason there are so many of those advocates is because they don't have to write such error-prone low-level code. Expressive code is more fun, in my experience.
It is very much possible, that if you run code on a big enough number of CPUs , it'll worth more than an average developer time.
I guess its cool to use something as hipster as Rust to speed up as something as hipster (albeit waning in popularity these days) as Ruby.
However, a better approach would be a straight forward implementation of caching (memcache/redis/static CDN) or to simply rewrite that part of the code to not rely on constantly parsing file paths? Or use something like https://github.com/google/re2, you probably won't be able to do any better than that when doing any kind of string matching and parsing.
If you're concerned at all about performance you shouldn't be doing anything with the file system period.
We have https://doc.rust-lang.org/std/primitive.str.html#method.as_b... for accessing a str bytewise. No need for pointers and `unsafe`.
The whole point of Rust is being safer and higher-level than C, and if you don't want to use its features, there is no improvement over C.
Write the speed-sensitive bits in Crystal. Aka 'Improving Ruby Performance with Almost-Ruby'.
(Unfortunately development seems to have stopped.)
There are some technical limitations with how I handled defining functions, which has also become obsolete with recent changes to Crystal. However, there is another approach (macro based) demonstrated at https://github.com/spalladino/crystal-ruby_exts. The macro approach is what is needed.
At time of first development, this was just an experiment and I didn't feel like redo-ing it this way. Now that core team has plans for similar functionality, I'd rather not finish a half baked solution that distracts from what theirs will likely be.