Unrelated, but can we please get web browsers to enable users to totally turn off all auto play video and video encoding?
I hate websites like this where you deal with a minefield of video areas that might start playing audio at the smallest tap and drain your bandwidth for content you have zero desire to watch.
Or news sites that play some annoying caster talking about it while you're trying to read and it follows you down the page. I'm here for an article not someone reading it to me.
If course they are. I haven't looked into this model in particular, but unless I'm way off base, these general purpose pretrained things are completely unsuitable for any specialized (say medical or whatever) application. This seems just like a better version of some of the previous semantic segmentation models they trained on coco.
It's probably a good starting point for fine tuning for custom application though, that's the main use of these big models.
Meta does stellar work in AI. I’m quite certain the recent DMCAs were a case of the left hand not knowing what the right hand is doing. This model is open source in every sense of the word, and they’ve now released two impactful models in the span of a month.
That is likely true in the most pedantic sense, but in practice, if I create an algorithm that works by using a series of matrix transformations against a set of carefully chosen (read: "trained") matrices and I open-source only the matrix manipulation code but not the specially chosen matrices, I think there's a fair argument to say that I haven't open-sourced the entire algorithm.
In the phrase "the model is open-source in every sense of the word", that, IMO, must include the weights.
I think of two broad categories of database records: transactional data (data created while running the system) and domain data (data created during development and shipped to production as part of the release process).
The former type of data I wouldn't expect to ever be open-sourced. The latter type might or might not be, depending on the intent of open-sourcing the related system.
If I created a human language translation system that used a SQL database to store the dictionary (domain) data and claimed the system was open-source without shipping the domain data, I think people would rightly say that the system was not fully opened.
The actual source code is not important. The source code can be printed on a single A4 page, the valuable final product is the weights you get after running the code for fifty million dollars of compute time.
If I autogenerated a huge amount of C based on the weights, that added/multipled variables the same way the existing code+weights does, then would it be “source”?
The weights seem to be under the same license, just distributed separately because it doesn’t make sense for the giant binary artifacts of training to be part of the source repository.
> left hand not knowing what the right hand is doing.
No it was trying to stop abuse. Unlike OpenAI, facebook can't get away with releasing a thing that is more than capable of making up libellous, racist or any other type of illegal or PR decimating thing.
The point of the really restrictive release was to at least try to limit that kind of issue.
Fortunatly for meta, most people who use the weights aren't the end user with a large twitter following, so the risk is (now) low.
I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.
It's funny, back in the 90s I disliked Microsoft and thought they were inherently culturally opposed to open source. But big corporations really just follow their business incentives when it comes to open source.
It's not like Meta "loves to be open". But they would hate a world where all the powerful AI development happened on one of the big clouds. Imagine if every researchers used some closed-source API from Google or Azure or AWS. Meta would try to hire some AI people, and they'd be like, ugh if we take the job we have to use this weird Facebook-specific thing. (<cough>Flow</cough>)
So, supporting PyTorch and providing open source models is just a good business strategy for Meta. I'm glad to see it.
> I agree, Meta's work in AI has been really impressive, and it's encouraging how they are open-sourcing so much.
My usual reaction to anything Facebook releases is "yawn, they released yet another model that is practically useless because it's released under a non-commercial license", but I'm pleasantly surprised that this one seems to be actually liberally licensed! Hopefully this continues into the future.
Sure. Any proof at all for that? I mean it's the obvious answer, but generally in business people who don't pay for things tend not to pay for things. In fact, in the enterprise having a big budget is the way you keep your corporate power. Free stuff isn't a way to keep your budget high.
Again, I would like to agree with your thinking. I just don't think the real world backs it up.
In particular, open source software prevents people from getting locked into any of the AWS, Linux, iOS, or Android ecosystems. All of those are popular software development ecosystems that Microsoft does not own or influence very much.
I've been wanting to build an app that will transform a photo into a template for stained glass. Interactive, to choose the "granule" size, merge color groups into one single piece, change border thickness, etc. Something like this would really help there and make it almost a piece of cake.
Yeah it basically does the first part that with a few lines of code! I've been playing with it using it to segment images and just randomly shake each segment in a short gif. Just to kind of mess around with it until I've found a real use for it or more time to implement some of the more interesting use cases I've got. I'm super pumped this was released!
I tested MANY while building various audio tech and this was by far the best (beats the shit out of all the python Pandas Hugging Face lists etc). Incredibly ability to cut noise (tho, it must be noted that perceptual improvements for humans do not usually increase machine transcription as AI models strangely seem to pull info out of the dead space between words, and the noise around words...not just the voiced words themselves...the hidden vibrations, beyond our human ken, oh mere mortals unworthy of the grand perceptive machines...ugh...:p :o ;p xx ;p)
I contacted the authors via their FB emails but never heard back. Right now it's non-commercial and I was building a commercial product.
Somewhat tangential question, have you looked for/found any audio models/tools that can be used for separating out individual voices to separate audio tracks automatically? Perhaps this is already possible with existing tools that I am uninitiated in.
I haven't tested this with multiple voices and it sounds like you want something more specific but it's produced 10/10 results with a couple dozen audio files I've thrown at it, might be of use... https://vocalremover.org/
Izotope RX Pro, which is software for the cleaning and refinement of audio for music and audio post production includes 'Multiple Speaker Detection' which analyzes different voices in a recording and allows you to process them independently.
I can't speak to it's effectiveness because I don't have any need for it, and also RX 10 Advanced is commercial software and pretty expensive for a casual user, but the feature seems to be on the horizon for other apps.
Any idea how/when this will make it into Photoshop?
According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.
But I'm incredibly curious if they'll license this to Adobe, or if Adobe's already got its own team building the same thing that'll be just as good, or what.
It's not super-clear to me why Meta as opposed to Adobe has come out with this first, except maybe to be able to apply filters to just segments of photos in Insta?
However, for sake of fairness, competition, and reproducible science, I hope that if Adobe makes this a part of their commercial offering, they train their own so we can have two variants of the same model available to us.
Very meta (pun intended) but is there a website that allows you to say "I want to find an AI engine / application that does X", where X might be any iteration of terms:
"magic wand tool that cuts an image from the background"
"like photoshop where you cut an object from the background"
"photo segmentation tool"
Google would be the de facto here, but its kind of worthless as its gamed by SEO.
81 comments
[ 2.3 ms ] story [ 173 ms ] threadI hate websites like this where you deal with a minefield of video areas that might start playing audio at the smallest tap and drain your bandwidth for content you have zero desire to watch.
Yep, Google removed the setting to disable it in version 80 for Android.
https://twitter.com/DrJimFan/status/1643647849824161792
It's probably a good starting point for fine tuning for custom application though, that's the main use of these big models.
This tweet chain is a pretty great overview: https://twitter.com/drjimfan/status/1643647849824161792?s=61...
In the phrase "the model is open-source in every sense of the word", that, IMO, must include the weights.
The former type of data I wouldn't expect to ever be open-sourced. The latter type might or might not be, depending on the intent of open-sourcing the related system.
If I created a human language translation system that used a SQL database to store the dictionary (domain) data and claimed the system was open-source without shipping the domain data, I think people would rightly say that the system was not fully opened.
>Currently, the code (without the weights) is available on GitHub
https://github.com/facebookresearch/segment-anything#model-c...
* default/sam_vit_h_4b8939.pth: 2.4GB
* sam_vit_l_0b3195.pth: 1.2GB
* sam_vit_b_01ec64.pth: 358MB
No it was trying to stop abuse. Unlike OpenAI, facebook can't get away with releasing a thing that is more than capable of making up libellous, racist or any other type of illegal or PR decimating thing.
The point of the really restrictive release was to at least try to limit that kind of issue.
Fortunatly for meta, most people who use the weights aren't the end user with a large twitter following, so the risk is (now) low.
It's funny, back in the 90s I disliked Microsoft and thought they were inherently culturally opposed to open source. But big corporations really just follow their business incentives when it comes to open source.
It's not like Meta "loves to be open". But they would hate a world where all the powerful AI development happened on one of the big clouds. Imagine if every researchers used some closed-source API from Google or Azure or AWS. Meta would try to hire some AI people, and they'd be like, ugh if we take the job we have to use this weird Facebook-specific thing. (<cough>Flow</cough>)
So, supporting PyTorch and providing open source models is just a good business strategy for Meta. I'm glad to see it.
My usual reaction to anything Facebook releases is "yawn, they released yet another model that is practically useless because it's released under a non-commercial license", but I'm pleasantly surprised that this one seems to be actually liberally licensed! Hopefully this continues into the future.
I have never, ever seen a convincing business case for their doing so, however.
Not your opinion. A compelling business case.
Again, I would like to agree with your thinking. I just don't think the real world backs it up.
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
In particular, open source software prevents people from getting locked into any of the AWS, Linux, iOS, or Android ecosystems. All of those are popular software development ecosystems that Microsoft does not own or influence very much.
https://github.com/facebookresearch/segment-anything/
ASL 2 license.
You just need to have napari installed. I only tested on Linux, so it might not work on Windows/OSX.
I tested MANY while building various audio tech and this was by far the best (beats the shit out of all the python Pandas Hugging Face lists etc). Incredibly ability to cut noise (tho, it must be noted that perceptual improvements for humans do not usually increase machine transcription as AI models strangely seem to pull info out of the dead space between words, and the noise around words...not just the voiced words themselves...the hidden vibrations, beyond our human ken, oh mere mortals unworthy of the grand perceptive machines...ugh...:p :o ;p xx ;p)
I contacted the authors via their FB emails but never heard back. Right now it's non-commercial and I was building a commercial product.
BTW - in my search just now in response to your question, I see Dolby also has an API: https://dolby.io/products/enhance/
I have no idea what's good or the best, but there's a starting point!
https://www.izotope.com/en/products/rx.html
I can't speak to it's effectiveness because I don't have any need for it, and also RX 10 Advanced is commercial software and pretty expensive for a casual user, but the feature seems to be on the horizon for other apps.
Despite the name, it can also do audio separation.
Here is the announcement from Meta (erstwhile Facebook)
https://ai.facebook.com/blog/segment-anything-foundation-mod...
According to the article, the weights haven't been open sourced, so nobody can take this and build their own Photoshop plugin.
But I'm incredibly curious if they'll license this to Adobe, or if Adobe's already got its own team building the same thing that'll be just as good, or what.
It's not super-clear to me why Meta as opposed to Adobe has come out with this first, except maybe to be able to apply filters to just segments of photos in Insta?
The article seems to be misleading, since I can just go to the github and download the weights right now: https://github.com/facebookresearch/segment-anything#model-c...
However, for sake of fairness, competition, and reproducible science, I hope that if Adobe makes this a part of their commercial offering, they train their own so we can have two variants of the same model available to us.
I tried Meta's photo segmentation tool and I didn't see anything magical in it.
I recall my partner discovering it by accident whilst messing about with some photos. The people in the foreground were lifted off the background.
Meta is releasing it to everyone, so you can use it on your own apps/services. (and it is a great implementation/state of the art).
And totally with you that the Meta release is great for getting this kind of thing more widely adopted.
"magic wand tool that cuts an image from the background"
"like photoshop where you cut an object from the background"
"photo segmentation tool"
Google would be the de facto here, but its kind of worthless as its gamed by SEO.