Share your feedback on the AI services experiment in Nightly

[asafko]

Hi folks, 

In the next few days, we will start the Nightly experiment which provides easy access to AI services from the sidebar. This functionality is entirely optional, and it’s there to see if it’s a helpful addition to Firefox. It is not built into any core functionality and needs to be turned on by you to see it. 

If you want to try the experiment, activate it via Nightly Settings > Firefox Labs (please see full instructions here). 

We’d love to hear your feedback once you try out the feature, and we’re open to all your ideas and thoughts, whether it’s small tweaks to the current experience or big, creative suggestions that could boost your productivity and make accessing your favorite tools and services in Firefox even easier.

Thanks so much for helping us improve Firefox!

[MWIchigo]

This is a feature I would never use.

On the other hand, a feature I not only would use, but in fact have used numerous times per day for several years, is hitting enter in an empty search bar to go to the search engine. It's a novel feature that other browsers have and Firefox seems to be mysteriously missing.

[Lighthouse]

Please add DuckDuckGo AI Chat.
No need for account and more privacy.
(https://duckduckgo.com/aichat)

[ppm1337]

This is a nice addition (which always should remain optional in my opinion), but without a keyboard shortcut this is completely useless. It's faster to just open Edge next to firefox/having a chatbot in a separate tab than clicking 3 times to open the AI chatbot sidebar.

[Encrypter]

I appreciate that some users may have reservations about these features, but incorporating them into Firefox can help make the browser more accessible and appealing to a wider audience. Since these tools are entirely optional, users who have privacy concerns can easily disable them.

@asafko Regarding the implementation, I was wondering if it would be possible to enhance the chat-bot's Ask chatbot functionality. Specifically, when I click on 'Summarize' without selecting any text, could the chat-bot be provided with the URL of the page I'm on? Currently, it displays a message indicating that I'm on a page with no selection, which isn't very helpful when I'm trying to summarize the page's content. Thank you.

[Somebody1]

can we atleast have something similar to this

This is DuckDuckGos policy on their AI features. It does not even require login! While I cannot say that I have NO doubts on whether or not they follow this I am sure Mozilla can follow it.

[Jeeakro]

The feature seems useful, essentially a free and thus less taxing to use version of those AI extensions that do a similar thing. One very needed feature is editing the prompt templates; when studying a pdf and not understanding something, the explain option is not perfectly reliable and sometimes makes the AI regurgitate the selection bloatedly. Also the "quiz me" template is too specific to use most of the time, I'd like to select whether I want multiple choice or how many questions or options should be generated.

[phonics8226]

BUG REPORT:

When a custom model is added and then switched using the dropdown menu at the top of the sidebar, it becomes impossible to revert back to the custom model, even after closing and reopening the sidebar.

Additionally, is it possible to have multiple custom providers? I often compare and utilize different models for various tasks to achieve better results. Maybe with "browser.ml.chat.provider.1", "browser.ml.chat.provider.2", etc.

[Mardak]

That's an interesting idea to support multiple custom providers for easy switching. Are these all local providers or maybe we can add these to the list?

[phonics8226]

Well, one is duck.ai as you can see on the screenshots, and judging by another comment you made, there are no plans to include it in the list.

Anyway, any word on the bug? Or can we expect support for multiple custom providers in some future Nightly release?

[hazing3991]

It would be nice to include DuckDuckGo AI Chat as an option to the chatbot sidebar. It provides anonymous access to popular AI models.

[Mardak]

You can configure a custom provider by setting `browser.ml.chat.provider` to something like "https://duckduckgo.com/?q=chat&ia=chat" or now "https://duck.ai" (which redirects to a similar url), but it seems like DuckDuckGo AI Chat currently requires a ?q= value and ignores it, so passing in a prompt and selected text doesn't work currently.

This could still be useful to have anonymous chatbot access to AI models side-by-side with regular tabs.

[gmeneguzzo]

If I am reading an italian text (e.g.) and I select it , then I click the emoji to summarize it using AI , mozilla send to Claude this request

"Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy."

as result I receive a summarize in English which is useless (for me).
You should change the prompt as follow

Please summarize the selection using precise and concise language. Use headers and bulleted lists in the summary, to make it scannable. Maintain the meaning and factual accuracy. Summarize using the same language in the original text.

Or you should add in the emojy menu "Summarize in default language set in the lab section"

[Mardak]

Are you using a localized Firefox or language pack? The prompt passed in should be in Italian: "Riassumi la selezione utilizzando un linguaggio preciso e conciso. Usa intestazioni ed elenchi puntati nel riassunto per renderlo facile da leggere. Mantieni il significato e l’accuratezza fattuale."

Even with English prompts, we've noticed chatbots tend to respond in the language of the selected text. But you can also modify the prompts and/or prefix to include "in Italian."

[WoefKat]

This feature looks really nice but I would really love to be able to use my own ollama server (either through the ollama API or through the ChatGPT API that my OpenWebUI frontend offers).

It'll be pretty easy to implement as the API is the same as ChatGPT's but just with a different endpoint.

[Mardak]

The current implementation is rendering a webpage in the sidebar, so exposing ollama with open-webui or other web chatbots interfaces works for now. We have been looking into directly calling inference APIs such as chat/completions llamafile and ollama already expose, and this would allow for Firefox to build its own custom response interface not necessarily a chatbot.

[WoefKat]

Thanks! I already use openwebui so this worked well! One thing I did notice is that the webpage is not provided to the chatbot unless I select text first. In that case only the title is provided, not the URL (the URL would be handy so I could use a tool for the chatbot to receive it itself into its context window).

For example, if I don't select anything it would be nice if the whole page text would be provided to the AI, or at least a URL so it can retrieve it though an automation.

[SoliTheFox]

Awesome! Can we get an easier way to set up custom providers? And maybe a shortcut to summarize an entire page?

Using the about:config, i managed to get ollama with open-webui working, but noticed that every time i switch to other providers i need to set it up again on the settings.

Would also be lovely to be able to add custom prompts more easily. Finally, being able to provide the chatbot with the entire page instead of just a selection would be great too.

Awesome feature, specially now with small models like Gemma2 2b achieving higher user preference than GPT-3.5 on lmsys. Gemma2 2b and open-webui are a great pair with this feature.

[WoefKat]

Would you mind explaining how you got ollama + open-webui working? I don't really understand how to do this 🙂
I have this running here on a separate server but how to integrate this into firefox isn't clear to me.

Edit: Ah I found it elsewhere in the chat, it's just simply setting browser.ml.chat.provider to my server's URL. That was easy. It doesn't offer me a translate option though (which would normally work better than firefox translate) so that would be a great option to add 🙂 I'd much rather have that than "Quiz" or "Simplify".

[Mardak]

Do you always want it to translate to the same language? Here's an example of creating a new String pref `browser.ml.chat.prompts.translate` that is set to "Translate to zh-TW"

I did need to set ollama + open-webui to have a default model, but it seems to be working fine after configuring the provider.

[WoefKat]

Yes, always to English. Thank you!

I set browser.ml.chat.prompts.translate to "Translate to English". It now shows up in the context menu, thanks! But could you tell me where to define the prompt itself? (PS: Is there documentation about this perhaps? I couldn't find it).

[Mardak]

How do you think we should expose summarizing an entire page? We're looking into ways that don't require text selection such as getting article contents or comments section. Would you want it to always summarize this selection-less text or other prompts as well?

The current behavior of custom providers isn't great for frequent switching as you noticed it is forgotten when switching to a "known" provider. Do you expect to switch among multiple custom providers? Or maybe just saving the last custom provider could be sufficient to make it easy to switch back? Are you using these as 1-off chats with the provider or do you tend to switch provider and use it for multiple chats at a time?

[SoliTheFox]

I believe that making it available in the right click menu would be awesome. For my use, summarizing the article content would be enough, but the comment section would also be nice. I think that the summarization prompt + a "talk with the page" prompt would be enough for most users. Something like "Here is the page: {page}. In the following messages I will make questions to you about the content of the page. Wait for me to ask a question.". Would be awesome to be able to easily customize/add prompts too.

Yes, being able to add and switch from and to different custom providers would be awesome. With more and more different interfaces, chatbots, etc., I think that being able to add a list of custom providers would make the feature versatile for all users. Maybe even default to have no providers at all (or one), and let the users manage which providers they want.

For my personal use, I only need the latest custom provider. But again, ideally, it would allow to add multiple custom providers at the same time.

[user8]

Can't choose duck.ai

[Mardak]

You can set `browser.ml.chat.provider` to "https://duck.ai" to have it available in the sidebar, but it looks like passing in prompts and selected text does not work with this page right now.

 

[avakining]

All generative "AI" is bad. I don't want it in my browser, OS, or anything I own.

Generative "AI" is a fad that is yet to be useful for anything -- it provides falsehoods as if they were fact and scrapes private data to train.

Adding "ai" features to Firefox not only misunderstands what people want their browsers to be for, it also detracts time and money from improving Firefox itself.

If this is kept, I will have to assume that FF is a lost cause and move to another browser.

[Mardak]

Yes, depending on the model, hallucinations or misunderstanding can be quite frequent, but even then, some people find value in having an optional AI feature readily available for situations where this type of creativity is helpful.

This feature allows for people to configure any provider and model, so hopefully we can help increase interest in training without scraping private data.

[Doug7070]

This is a terrible standpoint to take. By adding this functionality despite openly admitting that the current available options have gross ethical problems you are not promoting the idea that any of these ethical issues should be resolved, you are signaling that you consider those ethical issues to be secondary to hopping on the bandwagon. Until they can be independently verified to address all basic ethical concerns regarding AI features, none of these things should come anywhere close to being integrated by default.

[nokko]

---

@Mardak wrote:

[...]

This feature allows for people to configure any provider and model, so hopefully we can help increase interest in training without scraping private data.

---

Yeah, but the ones you’ve specifically included as default options are models that do use irresponsibly scraped data. So, the message you’re sending is that you approve of rampant content theft.

This sucks.

[MageBunkshelf]

If this feature is just for "some" people who find it helpful, then this can be an extension, not a base feature. Why is your stated goal here to "increase interest in [model] training"? So this "feature" is just advertising for the plagiarism machine? Why are you adding adverts for unethical technology into the base version of the browser?

[November]

"Hallucinations" and misunderstandings" is in and of itself a misunderstanding of how LLMs work. They aren't factbots. They don't "know" things. They predict what text sounds like a plausible sentence. That's ALL they do. If they ever tell you the truth it's purely by coincidence.

[Liz]

real happy firefox is tanking its reputation for something that in your own words has a problem where "hallucinations or misunderstanding can be quite frequent".

i'm really hoping you get a good paycheck before this browser tanks for shattering user base trust.

[Dan3000]

Please allow the usage of containers in the Sidebar. I don't want to be logged in to one of the AI API providers without isolating it to some specific container.

[Mardak]

Currently the AI Chatbot in the sidebar reuses the same regular tab container, so if you want to also use a different account for the same provider, you can open that in a different container. Do you have a particular provider you're wanting to use with multiple accounts? I currently have a personal and work ChatGPT account that I use Firefox containers to access both at the same time, but I do currently need to keep in mind which one I log into in a container vs regular tab.

Or are you not using multiple accounts and dislike how the feature reuses your logged-in state from regular tabs? At least for ChatGPT in the US, this could be useful for using the logged-out experience.

[phonics8226]

I'm not sure if I understand, are you saying that the sidebar uses the container of the tab from which it was launched?

[marcobassetto]

I can't use it as it currently breaks Sideberry

[WoefKat]

Same here 😞 That's a bummer. It would be nice if we could have sideberry on the left and the AI on the right.

Or just to have native horizontal tabs again 😉

[Mardak]

What do you mean by native horizontal tabs? Turning on AI Chatbot with or without Sideberry still has the normal tabs at the top for me.

[WoefKat]

Oh yes I meant vertical tabs 😞

I thought of them as horizontal because it's still a horizontal bar 🙂

[Mardak]

How is Sideberry broken? Or more that you need to keep switching between AI Chatbot and Sideberry sidebar panels? It seems like Sideberry at least has a keyboard shortcut to switch back to it.

[marcobassetto]

I need to keep switching between the AI Chatbot and Sideberry sidebar. If there's a way to open/close the AI Chatbot with a button it would be nice