Share your feedback on the AI services experiment in Nightly

[asafko]

Hi folks, 

In the next few days, we will start the Nightly experiment which provides easy access to AI services from the sidebar. This functionality is entirely optional, and it’s there to see if it’s a helpful addition to Firefox. It is not built into any core functionality and needs to be turned on by you to see it. 

If you want to try the experiment, activate it via Nightly Settings > Nightly Experiments (please see full instructions here). 

We’d love to hear your feedback once you try out the feature, and we’re open to all your ideas and thoughts, whether it’s small tweaks to the current experience or big, creative suggestions that could boost your productivity and make accessing your favorite tools and services in Firefox even easier.

Thanks so much for helping us improve Firefox!

[DenJohn]

NO

[Mte90]

The announcement is not clear that is not started yet.

Also it isn't clear if it will require a personal token for this services and what settings are available.

[Mardak]

Try checking Nightly for updates. It should be available with 2024-06-25 builds with settings dropdown choices of 4 providers. No personal token is necessary as the chatbot will reuse your usual tab's cookies if any, e.g., ChatGPT doesn't require a login and can reuse your login or subscription for GPT-4o "omni."

 

If you're comfortable with changing about:config, you can set "browser.ml.chat.provider" to any URL to open it in the sidebar, and if that page accepts passing in prompts with ?q=, the context menu functionality will work too. You can create your own prompts by adding a new String pref, e.g., "browser.ml.chat.prompts.a" set to "Tell me more"

 

[Mte90]

I can confirm that works 🙂

[Mte90]

The first thing that I notice that there isn't a way to close the vertical window...

The only way I found is to press Customize Sidebar that has an X so I can close the AI sidebar.

[Mardak]

Thanks for the feedback. You should be able to close it in today's Nightly with the new AI chatbot icon.

[marianoguerra]

will this be exposed for js developers? chrome canary experiments with gemini seem to expose a window.ai object you can use to use the capabilities from code.

if so please try to talk to each other and standardize an MVP 🙂

[Mardak]

This current chatbot experiment does not make use of LLM / chat completion-type APIs, but there's a path to building a more integrated Firefox chatbot powered by local or remote inference APIs that could be exposed to developers both for the web and add-ons.

[magentapuppy]

yeah keep adding this useless and privacy invasive stuff that *all* the big corps are doing, rather than adding features that actually differentiate it, particularly as a less 'BS' browser option 🙄

[Mardak]

We included chatbot providers like HuggingChat which says: "We endorse Privacy by Design. As such, your conversations are private to you and will not be shared with anyone, including model authors, for any purpose, including for research or model training purposes."

Additionally, if you don't want to use a remotely hosted chatbot, you could use one that's on-device so data doesn't leave your computer while getting the benefits of optional AI.

[lackey]

AI has myriad ethical and environmental issues behind it. The first two options offered by Mozilla are created by some of the most invasive, unethical, environmentally destructive companies in the AI space. Google and Microsoft are not known for being private, not even a little bit.

The general issues behind AI, and the fact companies like HuFace need to scrape data without consent or permission, makes me question their promise to keep the data I offer them private.

These new changes appear antithetical to the Mozilla Manifesto.

I believe Mozilla should cease their attempts to integrate AI into the browser, and to reevaluate the ethical implications of promoting these products by these companies.

[magentapuppy]

Unfortunately I do not count scraping people's data without consent of any kind "private by design"

And wow, it must take an awful person to only consider the 'end user' as the one deserving of privacy. Like, just, wow.

[-0]

Why does this need to be shipped with Firefox? Can't this be an extension? I want less AI, not more. 😞

[Mardak]

This is currently a first integration of AI with plans for more optional functionality that don't require a sidebar interface, e.g., suggesting tabs that are relevant to your current activity or name for a tab group. Getting to more seamless integrations would likely be trickier as an extension and can be implemented with minimal impact for those who don't want it.

[duskord]

Can you also provide Claude?

[Mardak]

You can configure a custom provider to Anthropic Claude for easy access to chat while on another tab. Currently the context menu passing in a prompt doesn't work, but we'll work with them for better compatibility.

 

[TS25714]

This will absolutely need a second sidebar, the same way Edge already has (left side for tabs, right side for copilot etc, for example). Right now we would have to decide between Sidebery and similar (even the upcoming efforts to bring vertical tabs natively to FX) which is not worth it IMHO. Vertical tabs > shortcut to chatbot.

[Mardak]

These Nightly builds have the new sidebar experience with vertical tabs in development along with the chatbot experiment. We're still working on integrating these features, but you can have both active today.

 

[aminought]

Sidebery > native vertical tabs. Also:
1. Each sidebar extension should have their own width. It is very inconvenient to look at the result of the AI Chatbot in a small panel in which I usually have Sidebery.
2. Some sidebar extensions, like AI Chatbot, should be able to open in the overlay so that I can return to Sidebery without additional actions.
3. A setting is needed to select the language in which the chatbot should respond.

Thank you!

[Iron56716]

I honestly like the AI Chatbot integration. I'm hoping Google AI Studio will be added officially.

It'd be even better if the sidebar could get official web page integrations like Edge or Floorp, That feature is very handy.

[Mardak]

Curious, is there a particular functionality you like about Google AI Studio? The additional settings for Gemini model choice? Model tuning? Here's using it as a custom provider, but I had to manually copy/paste the prompt:

 

[Iron56716]

There's quite a lot of things I like about Google AI Studio. Firstly, the ability to completely turn off all the safety filters, it's much easier to chat with the AI without them. Secondly, like you mentioned - the Gemini Model choice. I can change the model to any different one right in the conversation without the need to open a new chat. And of course, there's some extra perks like Custom Instructions, Model tuning, etc. Last but not least, AI Studio allows the use of Gemini 1.5 Pro without the subscription (unlike the regular Gemini websites. Although, the AI Studio version of 1.5 Pro does have a 50 turn rate limit I believe).

[mcortt]

AI in sidebar is great. However, please give us some sort of keyboard shortcut to open it.

[Mardak]

Does it need to be a keyboard shortcut or would a dedicated button to click be good for now? There's a lot of existing keyboard shortcuts, so having one for each sidebar might be too many, but could a keyboard shortcut to toggle / reopen the last opened sidebar work too?

[mcortt]

The main goal of a web browser is to browse the web, not browse the browser UI, which is what I feel a specific button accomplishes. The button would work (the sidebar toggle button is what I currently use), but there is no keyboard shortcut to toggle it currently that I am aware of. Personally, I would prefer a keyboard shortcut for the AI chat specifically, but I am also of the thought that most things should have keyboard shortcuts, or at least be able to be assigned a keyboard shortcut (a la extension shortcuts). Though, I understand where that could be a lot of work.

My preferences in order:

• Assignable keyboard shortcut to AI, synced tabs, etc
• Keyboard shortcut for sidebar toggle
• Button to open it

For now I would settle and be happy with a keyboard shortcut to toggle the sidebar to its last state.

[BiORNADE]

You can put ChatGPT or any AI Chatting service into a sidebar, which makes the features useless and sort of a filler but... Actually, the only time that AI Chatbots in a browser was a big deal was when Microsoft Edge had BingAI, which was a free version of GPT 4.0 FE (with some improved enhancements), but it didn't last much until MS got Copilot (BingAI) in almost everywhere, so you can use it on Windows 7 now (sarcastically saying)

To test the waters with sidebar, using a known website (from time to time) is a nice idea, so the Firefox Team are able to detect the issue easier and deliver a fix earlier... But a native AI feature that works just like a sidebars is not that of an idea

But here's a nice thing; you can actually access everything just like watching a gecko video and commenting on this very forum using a simple technique.
Some more contrast in the lines, some more tests, some polish from the community, the final testing, then the final product. The technique could also extend to PWAs (the most Kudoed question still on review)

Looking ahead to an answer

Kind Regards,
BiORNADE

[Mardak]

I'm guessing you used Google Gemini to navigate the sidebar to another page such as this Connect discussion? You could more directly do that by setting a custom provider url, but that's not the main use case of this feature.

[Frisk]

Hello!

With introduction of even more AI services to Firefox I wanted to express that to me it does seem like Firefox is missing with development of features. This sentiment is echoed by a lot of people in my social bubble of technologists, ethicists and other people with same priorities as what one could think are values which Firefox was built on.

Those concerns are in my opinion very valid. The machine learning models have shown to be unreliable - just some of the recent examples from AI products made by large corporations: pointing users to eat pizza with glue, providing false information about just about anything. There are a number of ethical issues yet to be resolved with usage of AI, from its intense usage of computing resources that adds up to electric grid demands. Through privacy and possible copyright violations of datasets that power the models. To an entire bag of other issues monitored by excellent resources such as AIAAIC repository. 

AI/Machine learning is an amazing field with many likely applications, and yet, its recent rise to fame is characterized by failures and issues in many implementations. Personally I often don't even see whether the application of AI is truthfully necessary, in many cases a human would do a more trustworthy and fast task of information gathering than a large machine learning model.

When we compare current state of "AI", does it reflect what Firefox stands for? Does it reflect Mozilla's principals?

Let's compare.

• Principle 2

  The internet is a global public resource that must remain open and accessible.

LLMs are known for being a black box. Depending on our definition of "open and accessible", LLMs can be a very free resource or a completely inaccessible black box of math.

• Principle 4

  Individuals’ security and privacy on the internet are fundamental and must not be treated as optional.

It's clear that LLMs pose a privacy risk to Internet users. LLMs pose risk in at least two ways - because the data they are trained on sometimes contains private information due to negligent training process. In this case users of a learned model can possibly access private information. The second risk is of course usage of 3rd party services that may use information to infringe on privacy of users. While Mozilla in blog assures that "we are committed to following the principles of user choice, agency, and privacy as we bring AI-powered enhancements to Firefox", it's unclear how supporting such services as "ChatGPT, Google Gemini, HuggingChat, and Le Chat Mistral" helps protect Firefox user privacy. Giving users choice should not compromise their safety and privacy.

• Principle 10

  Magnifying the public benefit aspects of the internet is an important goal, worthy of time, attention and commitment.

In my opinion in the process of designing AI functionalities on top of Firefox there was no evaluation on how those functionalities can benefit the public. There are a number of issues as mentioned above with the LLMs, they can be dangerous and work in detriment to users. Investing and supporting in technology of this type may lead to terrible consequences with little actual benefit.

In addition Mozilla claims:

• We are committed to an internet that elevates critical thinking, reasoned argument, shared knowledge, and verifiable facts.

I argue that AI models are the opposite to that. AI output is not verifiable. They are working against sharing knowledge by making seemingly accurate information that turn out to be false.

I ask Mozilla to reevaluate impact of AI considering all of those points. I ask on behalf of myself as well as many users that I see on Fediverse being greatly worried and frustrated with AI changes added on top of Firefox. There is certainly a lot of potential greatness that could be done with AI, but those steps must be taken responsibly.

[narazamsa]

If its on device only and no data collection is going to happen. Also only complete opensource models. Then yes that would be good to have.

[Mardak]

Are there any particular open source models that you would recommend for running on device? You might be able to set it as a custom provider to chat directly from Firefox.

[ilikefirefox99]

I love the features being implemented, BUT... as many point out, a second sidebar  is necessary. The idea should be same as all other browsers. You have a vertical panel with your vertical tabs in the left, and you have the sidebar (history, bookmarks, etc, etc, etc) on the right.

Trying to create a single sidebar that also has your vertical tabs on it, it's a very messy concept and a terrible poor implementation in my view. I understand that it might be tricky to implement this feature, but honestly, it's very very very necessary.

No need to reinvent the wheel, Vivaldi, and especially Microsoft Edge, already show  you the best concept:

Thank you.

[0x4d6165]

if you're gonna use AI crap anyway why not just use edge lol

[Jon]

Thanks for the feedback! For those interested, there's another idea thread for a second sidebar here:

2nd Sidebar 

[ksetlak]

I have one observation: I'm logged in to Google on my work laptop and my work Google account is the primary one, while the private one is the secondary. However, I only use Gemini with my private account. I can't change the account I'm using in the side-docked window, because when I attempt to do that, a new tab is opened in my main Firefox window. Would be lovely if this was configurable and `authuser=1` could be added to the params in the URL used to open the side pane.

[Mardak]

Google does support multiple active accounts where in your case https://gemini.google.com/u/0/ is probably your primary work account and https://gemini.google.com/u/1/ is your private account. Currently, the sidebar defaults to the first account logged in, so if you could sign in first with your private account, that should get it to work.

Firefox also supports Container Tabs, which is useful for sites that don't normally support multiple accounts. If you logged in to your work Google account with a "Work" container while your private account is logged in to a regular tab (not container), this might work for you too.

Alternatively, you could specify a custom provider for the desired Gemini url to get that in the sidebar for simple chatting, but currently Gemini doesn't support passing in prompts when customized this way.

 

[0x4d6165]

Actually listen to your userbase for *once* when we say we do NOT want this! Be a good web browser stop trying to be edge.

[Reycko]

Would be nice to have a nice UI to add custom prompts if that's not in the works already

[aminought]

You can! Just edit "browser.ml.chat.prompt.prefix" and "browser.ml.chat.prompts.*" prefs. Also, you can add additional prompts like "Answer Questions" if you add "browser.ml.chat.prompts.3", "browser.ml.chat.prompts.4", etc.

[Mardak]

Any suggestions on what's a nice UI for adding custom prompts? The current way with about:config preferences is more for advanced testers, but it could work for you in the meantime. Or if not, what prompts were you wanting to add and perhaps we can add it to the default list.