@mozilla This is how it's done! Private, open-source AI models running locally.

Q: How much storage do the models take? (EDIT: 200MB according to the post - yeah, in this case, this better be a downloadable 'module' instead of being built-in) Could you make this feature optional, which would require the user to opt-in and download or delete the model(s) themselves? I don't want Firefox to go the Microsoft Edge route, where they shovel every feature under the sun, the user has no choice, and there is no way to reduce the storage occupied by the browser.

@mozilla Looks interesting at first glance.

Thanks for being this open and transparent about the process, used model etc.

@mozilla@mozilla.social Well I hope that does work fine.
I don't quite like AI features, but users with visual impairment* does need more love from software projects, especially browsers like Firefox.

* And other impairments, of course!

@mozilla Finally a good use of Machine Learning.
I hope it works well for those that need it.

Of course, it would be ideal if everyone wrote their own Alt Text and all platforms allowed users to do so, but this is a great solution for cases where people don't want to, have the spoons to or can't do it themself.

Assuming it works of course, but I am hopeful.

Edit: I posted second thoughts about it seperately. tl;dr: Might be useful if it admits failure instead of hallucinating.

@mozilla Nice work!!!

This would be particularly useful for postings to Mastodon, where alt-text is much socially desirable.

@mozilla@mozilla.social

Ah, so, reader-side rather than something that, say, a Fedi user c/would use when posting images?

@mozilla you do realize that incorrect alt-text can be worse than no alt-text? Sometimes critically so? I do hope this will be strictly opt-in and alt-texts generated by the model will be clearly marked, if this ever goes into stable Firefox

@mozilla alt text cannot be automatically generated without human input because the function of alt text is highly contextual. if you actually gave a shit about a freer more independent web you'd support projects like @hannah's distributed alt text database which is currently supported via browser extension. it looks like you're too late now to take it over but if you have any funding left for non-AI bullshit i'm sure the 501(c)(3) would absolutely love your support as well as built-in browser integration https://social.alt-text.org/@hannah/112522294652904233

@mozilla Good to know. Is there going to be a settings toggle to disable AI features altogether?

@mozilla hm, i think this can be useful, however the problem is when people will never look at the output and just accept it at face value.

Basically I hope you will add a warning box that says "Do note that the text generation is not perfect and you should make sure the text clearly fits the image" or something along those lines. Also when it generates the text, it should always add "This alt text was generated by Firefox language model." as the first sentence, so people who rely on alt text features will know that this may be inaccurate.

@mozilla What is it actually trained on?

@mozilla this is not how we push for better accessibility. PLEASE reconsider this "AI" shift

@mozilla

This is the first non dream stealing non tocic use of the clockwork parrots I have seen; good for firefox

@mozilla waiting for the hilarity that will ensue

@mozilla cool.
Is it time to invent webbrowser that actually dont spy on you?

@mozilla would be amazing if this offered an API for webapps to use. E.g. mastodons alt text field could detect it has that feature available and provide a suggested alt text to users.

On the other hand this might encourage lower quality alt texts, as that will always be quicker to do than writing down your own alt text.

Maybe keeping it "fallback" only for consumers of content that is missing alt text is best.

@mozilla AI haters in your replies in 3 ... 2 ... oh nevermind, they are already here 🫠

@mozilla Seems like a push in the right direction.

@mozilla Ok I'll admit this is pretty cool

@mozilla please ensure there's an option to turn it off. I don't want any AI in my system.

@mozilla I don't want 'AI' on my computer! How to get rid of this 200MB?

@mozilla cool. my visually impaired audience will be so happy i clicked the button to generate a bunch of gibberish, instead of spending 10 seconds to write a thoughtful description.

@mozilla no. get this garbage out of here

1. what the hell is it trained on? the same corpus of implicit bigotry and medical misinformation as every other "ai"?
2. it's just going to lie a bunch like every other "ai". wrong alt text is worse than no alt text.

@mozilla gippity

@mozilla Or, I don't know, people could type them? On the things they post? "This way, people don't have to think" - ?

@mozilla you really make it hard for decent people to not switch to other browsers for good. At this point there's no other explanation than your managers being bribed by competitors.

@mozilla please make sure that there is a opt out option for this feature, I do not want to use AI to generate alt text that can be potentially incorrect or misleading to the people that needed most, i’d say this feature is highly unnecessary while other features should be prioritized over adding more AI to a web browser. If I wanted AI in my web browser, I would be using Google Chrome and using glue to make sure the cheese doesn’t fall off my pizza.

@mozilla No. Bad Firefox.

@mozilla can it summarize?

@mozilla
Context is important though. AI (rather, ASI) will fail to understand the intent of a picutre. That's an easy way to render a person unable to understand the picture even with a physical explanation...

@mozilla So now alt text will be randomly generated gibberish. That sounds helpful. Hope AI bankrupts your company. Just stick to being a browser. It's all we've ever wanted from you. No crypto, no AI, just browser

@mozilla This is cool-ish (and love the private, on-device aspect), but seems like the wrong end of the pipeline. Does Firefox help people write their alt text when posting images (and prompt them to do it)? Now _that_ would be game-changing. No need for the cost of the power increase of hundreds or thousands of _readers_ to create it, do it when the _writer_ creates it.

@mozilla
Hard fucking pass. I don't want a tiny LLM using my device resources.

@mozilla Please stop with the "AI" bullshit.

@mozilla I would have preferred it if you had just focused on speed. I don't need AI built into every piece of software I use.

@mozilla no no no. No AI in Firefox, PLEASE. Y'all are the one viable alternative browser, please don't start following silly trends like this. If people want AI nonsense, there's plenty of places to get it. I would happily pay to avoid having features like these creep into Firefox.

@mozilla So, erm… how often will it phone home to teach a centralized model about how to do this? How much are you going to pay your laborers for their time to train your a.i. models?

@mozilla
What a shitty idea.

Wish I was surprised that Mozilla has jumped on the AI Highway to Hell.

But their priorities have not been user focused for years IMO.

https://www.youtube.com/watch?v=4hhlQU0zDpA

@mozilla The amount of kvetching in the comments here is hilarious. I guess it should be expected if you combine AI, open source, and alt-text into a single post on Mastodon.

@mozilla now you are talking ✔️

@mozilla using AI to fill in missing alt text is an excellent application, however that AI is trained on alt text, so in the long run there will be less and less training data and the AI will get wronger and wronger.

@mozilla@mozilla.social yayy :3

This is good

But please

Please, do things other than ai

@mozilla Automated alt-text isn't too bad. This is a fair use of AI that doesn't really step on any toes.

I could foresee this causing problems if the alt-text is very wrong, though.

@mozilla@mozilla.social why does my browser need AI at all? Can we stop wasting electricity on this snake oil?

@mozilla ITT all 7 firefox users upset that AI might be useful for some things

@mozilla "The first time the user adds an image, they’ll have to wait a bit for downloading the model (which can take up to a few minutes depending on your connection) but the subsequent uses will be much faster"

Hope you can disable this feature and 200 mb download at all
I understand it can be very useful for some users, but for me not at all
If I find porno images I don't understand I think I can skip them 🤪

@mozilla ITT all 7 firefox users upset that AI might be useful for some things

Not pictured: 2 billion people using Chrome

Alt text will absolutely still be required on websites and social media posts; this is just to patch over where people couldn't be bothered to be inclusive, basically :)

@mozilla
A clever way to make developers and site owners not forget to write proper alt text. "Write your images alt text, otherwise we'll generate gibberish and deface your websites!".
Smart move, Mozilla.

@mozilla How much RAM and CPU does this cost?

@mozilla I'm still waiting for the feature that obligatory speaks back the comment to the user before posting.

(see https://xkcd.com/481/)

@mozilla no AI, please!

@mozilla This is a good use of the technology.

@mozilla will I be able to turn it off? I don’t want AI everywhere and in everything.

@mozilla I asked ChatGPT for a short prompt for the candle picture:
“The image shows a birthday cake with lit candles in the foreground and a smiling woman in the background, likely in a room with several people.”
That's indeed longer than the Firefox text but not absurdly lengthy and detailed.
However, I'm impressed that Firefox does that locally.

@mozilla Sounds like a positive and non-intrusive integration of AI in an application for once!

@mozilla This is pretty cool; but I was unaware that Firefox has a built in PDF "editor" and not just viewer.

@mozilla

Way to fix bloat...

@mozilla BORING. Where's the focus on the actual browser standards, like PWA support!?

@mozilla how can I disable it? I dont have free gigabytes for this shit.

@mozilla Nice! Your PDF functionality was neat, so this looks promising!

@mozilla literally moved to Firefox to get AWAY from stoopid AI hype. WHYYYYYY????!!!!

@mozilla good i don't update…

@mozilla @podfeet

@mozilla awesome !!!

I don't actually Want that in my browser.

If I did, I suppose I'd run, I don't know, Fucking Windows?

@mozilla boooooooo

@mozilla Im ashamed to use this shitty browser that keeps getting worse. at this point the only way to save firefox is to fire all the devolpers who make these changes and never let them work on anything again except those little "square hole" baby toys.

@mozilla I've been working on tools around issues like this for years, including a large amount of communication with folks who depend on alt text. Should this technology be available to those writing alt text, rather than just voluntarily to screen reader users, it will be immensely harmful to the accessibility of social media everywhere. Please do not do this. I'm happy to dive into why but please do not make it easy for people posting images to use this AI generation.
- Founder, Alt-Text.org

@mozilla this looks like an awesome accessibility feature! thank you for including an easy-to-understand rundown in the article of how this tool differs from the ethically-dubious LLM stuff that's everywhere nowadays

@mozilla I've serious concerns about this. I get that most likely other browsers will do it, but I feel like if we let people generate alt captions via AI models the people who will benefit from those captions will most likely miss out of the details that only a human looking at an image will actually see and be able express? If anything the AI model will make things up and explain things in such detail that might not be relevant to the actual image/scene itself.

@mozilla This is good, but I feel like this shouldn't be a long term solution. Instead we should be pushing websites to serve alt text, especially since HTML already has support for it.

@mozilla I hate that you guys are doing this instead of giving us a proper tablet UI. I just want to browse the web, folks.

@mozilla This a good and well done use of AI. It may be particularly useful for posting images on mastodon and fediverse in general.

@mozilla this is exactly how software become increasingly bloated

@mozilla
I wish Mozilla the best with this AI experiment, but I will transition to LibreWolf and use that browser until the AI is removed from Firefox.

(Medien: 1)

@mozilla I just want an easy way to save my pinned tabs.


@mozilla if the point is to make more high-quality alt text available for people, can you imagine how much more effective it would be to ship a set of configurable UI hints, authorship guides, and a regular series of articles coaching authors which alt text writing patterns to cultivate and which to avoid? A metadata standard to allow embedding and attaching alt text alongside and inside images, with author attribution and strong multilingualization? This feels wasteful and ill-prioritized.

@mozilla Really wish you'd let us set our own passcode instead of the device code as a backup to the biometrics to view our passwords. If anyone figures out our device passcodes, they can get our passwords if we have them saved in Firefox mobile. Given that bad guys are already aware of this, you need to be on top of it.

@mozilla My photo collection search could use such a thing.

#digikam

@mozilla sorry I yelled at you about AI. This is good, actually. Still suspicious of the urge Mozilla has of throwing away a bunch of money chasing the latest fads.

@mozilla
Who in the screen reader community asked for this? Aspirations are great, but AI generated captions might miss the mark for accuracy and value added...

@mozilla So OCR is now renamed to AI?

@mozilla thank you for doing this. Seems like a very useful feature. I hope you get the AI right for it.

@mozilla

These modules need to be opt-in from the beginning ... not mandatory with a "maybe we'll let you get rid of them some day, teehee" that seems to be your current plan.

I REALLY don't want that functionality, and especially not the bloat or other resource use. Guess it's time to start looking for a new browser.

@mozilla this is awesome! An actually useful application of "AI", and on-device to boot. This and the on-device translations are the kinds of features I want more of!

@comicsansg

@mozilla

except blind people need an alt-text depending on context and what you wish to say (which IA won't have)

@mozilla

These kinds of inclusive quality of life features are one of the best use cases for AI, so good on Mozilla for creating this.

@mozilla wow, actually a good use of generative ai for once!

@mozilla I always hope that these kind of thinks will be some thing like Modules/plugins that needs to be installed as an extra

@mozilla - couldn't you teach this AI to mark the text of web links?

@mozilla Thats gonna be useful for web dev but i hope its privacy focusing since i would not use it