This is an automated archive made by the Lemmit Bot.

The original was posted on /r/opensource by /u/CuriousAustralianBoy on 2024-11-20 09:54:24+00:00.

Original Title: I Created an AI Research Assistant that actually DOES research! Feed it ANY topic, it searches the web, scrapes content, saves sources, and gives you a full research document + summary. Uses Ollama (FREE) - Just ask a question and let it work! No API costs, open source, runs locally!

Automated-AI-Web-Researcher: After months of work, I’ve made a python program that turns local LLMs running on Ollama into online researchers for you, Literally type a single question or topic and wait until you come back to a text document full of research content with links to the sources and a summary and ask it questions too! and more!

This automated researcher uses internet searching and web scraping to gather information, based on your topic or question of choice, it will generate focus areas relating to your topic designed to explore various aspects of your topic and investigate various related aspects of your topic or question to retrieve relevant information through online research to respond to your topic or question. The LLM breaks down your query into up to 5 specific research focuses, prioritising them based on relevance, then systematically investigates each one through targeted web searches and content analysis starting with the most relevant.

Then after gathering the content from those searching and exhausting all of the focus areas, it will then review the content and use the information within to generate new focus areas, and in the past it has often finding new, relevant focus areas based on findings in research content it has already gathered (like specific case studies which it then looks for specifically relating to your topic or question for example), previously this use of research content already gathered to develop new areas to investigate has ended up leading to interesting and novel research focuses in some cases that would never occur to humans although mileage may vary this program is still a prototype but shockingly it, it actually works!.

Key features:

  • Continuously generates new research focuses based on what it discovers
  • Saves every piece of content it finds in full, along with source URLs
  • Creates a comprehensive summary when you’re done of the research contents and uses it to respond to your original query/question
  • Enters conversation mode after providing the summary, where you can ask specific questions about its findings and research even things not mentioned in the summary should the research it found provide relevant information about said things.
  • You can run it as long as you want until the LLM’s context is at it’s max which will then automatically stop it’s research and still allow for summary and questions to be asked. Or stop it at anytime which will cause it to generate the summary.
  • But it also Includes pause feature to assess research progress to determine if enough has been gathered, allowing you the choice to unpause and continue or to terminate the research and receive the summary.
  • Works with popular Ollama local models (recommended phi3:3.8b-mini-128k-instruct or phi3:14b-medium-128k-instruct which are the ones I have so far tested and have worked)
  • Everything runs locally on your machine, and yet still gives you results from the internet with only a single query you can have a massive amount of actual research given back to you in a relatively short time.

The best part? You can let it run in the background while you do other things. Come back to find a detailed research document with dozens of relevant sources and extracted content, all organised and ready for review. Plus a summary of relevant findings AND able to ask the LLM questions about those findings. Perfect for research, hard to research and novel questions that you can’t be bothered to actually look into yourself, or just satisfying your curiosity about complex topics!

GitHub repo with full instructions:

(Built using Python, fully open source, and should work with any Ollama-compatible LLM, although only phi 3 has been tested by me)