I made a HN-style demo video at https://www.tella.tv/video/build-your-ai-second-brain-with-s... and there’s a marketing video at https://www.youtube.com/watch?v=c1jV6E9pyug.
I’ve been obsessed with this for a long time. I’ve been maintaining a “second brain” since 2020, in which I would store journals, handwritten notes, music I listen to, projects I'm working on, conversations I have with people, personal CRM etc. I experimented a lot of RAG in the early days with ParlAI, hundreds of fine-tuned GPT2 models, and GPT3 (https://forum.obsidian.md/t/fine-tuning-openai-api-gpt3-on-y...). Later I built Ava, the first Obsidian AI plugin, which grew to a few thousands of users quickly. It then became Embedbase, an API to make it easier to build AI apps powered by RAG.
What I learned from all this is how important it is for the models to have context about what you’re doing on your computer, in order to get them to do what you want.
In the early days there was fine tuning but it was too much pain, then there was tool calling so that AI can access software you use but still kinda not autonomous enough. needing micro management. Then MCP came, but it felt too static, and non technical users struggled to build and use MCP. Then we got skills. Most recently we’ve seen Karpathy’s LLM-maintained wiki, Garry's GBrain, etc., where an agent incrementally maintains a persistent collection of Markdown pages. New sources update entity pages, strengthen or contradict existing claims, and improve a synthesis that compounds over time. I like this pattern, but it still begins with someone selecting and importing the sources. There is still no way AI can know what you and your company are doing every day, across apps, not just inside of apps.
Of course, not everyone wants this. But I do! I want AI to know what I'm doing and never lose memory ever again, and I want it to use the same software that humans do, without painful context switches.
I started building Screenpipe for myself in 2024 - a CLI to record your screen and plug this context into AI. An HN user posted it in 2024 (https://news.ycombinator.com/item?id=41695840) and that discussion influenced the product. The most useful criticism concerned recording consent, local security, CPU usage, signal-to-noise, and whether agents could act on top of the data.
The naive implementation started from continuously recording video and running OCR over every frame. But that creates duplicate data, consumes substantial resources (it basically turns your computer into a space heater!), and discards structure the operating system already knows. Screenpipe now instead listens for events such as app switches, clicks, typing pauses, scrolling, and idle fallbacks. When something meaningful changes, it pairs a screenshot with the operating system’s accessibility tree at the same timestamp. OCR is used when structured accessibility data is unavailable. We also capture audio continuously, identify speakers and transcribe locally through Parakeet/Whisper or using cloud models.
Everything is indexed in a local SQLite database, mp4 files, and sometimes md files. An AI friendly API on port 3030 is open for agents, with authentication and a MCP and skills.
Once Screenpipe has been up and running for a while, you can use it through our built-in chat, Claude, ChatGPT, Hermes, Openclaw, or any agent, to do things like:
- adding context to your current chat, e.g. "gather all context about task X", then requiring less prompts to achieve your goal
- retrieve information, e.g. "retrieve the tasks i was working on from 8 am to 4 pm, make a list of what got done and what's left"
- create and maintain a personal wiki / second brain for your agents: "every 1h organize everything i do in projects, people, tasks, meetings in my Obsidian vault as markdown files and folders"
- create automations: whenever i visit someone's profile on linkedin, update my crm
- find automation opportunities: look at everything my team has done this week and turn it into a list of automation opportunities
Screenpipe data is stored locally, though we also offer an enterprise plan to discover automation opportunities and for that the company decides where the data lives. We built our own AI PII model to redact sensitive information, it runs locally on Apple MLX or Windows DirectML, we also support cloud confidential inference for low end devices, although our local models are meant to use <1% CPU and <400 mb RAM. Users can set apps, windows, and urls to filter, in addition to browser incognito mode.
We also support recording schedules and other privacy features.
Most of our codebase is written in Rust, MLX, Onnx, we like cidre or direct C call for Apple APIs and windows-rs for Windows API. We also experimentally support Linux.
We have a desktop app (https://screenpipe.com/how-to-install) and a CLI:
npx screenpipe record
Versions released before the license change remain available under MIT. We have a free tier, and other plans, including Enterprise which helps companies find automation opportunities.
Would love to hear any feedback, things you've done with screenpipe, or features you'd want
The systems that actually feel like a second brain flip that: they push relevant history, decisions, and prior context at you before you ask, and — this part matters more than people give it credit for — they show their work. Cite where a given answer came from (which note, which meeting, which thread), not just serve up a synthesized blob. Looking at the pushback in this thread (the harvesting claim, the "zero chance I trust cloud SaaS" comments), I don't think that's really an objection to always-on capture — it's an objection to opacity. People will tolerate a lot of passive collection if they can see the receipts on what it's doing with it and where an answer traces back to. Local-only storage helps with the trust problem, but provenance is what actually earns it over time.
I'm curious what does your AI buddies do more specifically and how do you use them?
Main concept of AI-buddies is to quickly spin up "buddies" who have varying personalities and feel like a real person. Just like real people buddies can talk to each other, share information and have very different functions in your life. For ex a Gym buddy is very different from a "mom" buddy or a Math prof buddy.
Main difference that they are easier to set up for non technical users, and that they can gather input from other real people (for example, if allowed, a Tax buddy can get input from your accountant).
Otherwise we usually collaborate with the company to allow screenpipe
Curious how it runs on more modern hardware? Is it pretty lightweight these days?
You can also configure settings like changing transcription engine or only record meetings or turning off some features to reduce resource usage
We capture everything and gives you and option for a flexible data retention policy, so that you can prune mp4 files for example or verbose accessibility trees, while our agents generate high level memories referencing low level data
It still not perfect, but ideally Screenpipe would record forever so that you have infinite memory of every tiny details, while having high level structured memories for humans and agents, individually or as a company
in my experience, even within a single project, the more agentic cycles, the more features and the more surface area the more cruft builds up. This is an obvious observation, but I suppose I’m at the point I think the move forward is to create a new project, keep only what worked from the preceding as modularized entities and intentionally wipe away all past of how we got there save for the API interface truths.
https://cal.com/team/screenpipe/chat
Feel free to configure local LLM like Ollama, btw
During normal usage, how often does it try to parse info from the screen capture? Once a minute?
Screenpipe captures accessibility tree and screenshot when you perform a meaningful action. It doesn't use AI at recording time (except PII removal and OCR infrequently). So it can be every few hundred milliseconds to few minutes or more (if screen idle or sleeping)
We usually benchmark CPU usage on $200 Windows/MacOS laptop
Also, I for one would never invest a second in tools I cannot freely modify and share the code of under OSI terms. I would strongly suggest a convenience tax model. Hackers will self host and maybe contribute but those with more money than time will put in a credit card. Maybe offer end to end encrypted memory and compute to secure enclaves where additional compute on the data can be done when the laptop is closed. (Shameless plug, this is what https://caution.co enables, and 100% FOSS)
However, with "source available" you are just begging for someone to AI launder your code into a FOSS clone you will have 0 recourse on. If you FOSS it yourself then you get to capture the FOSS community destined to form around this idea. Some of that community will have bosses that will pay you.
MIT may not be the right play though. AGPL is a good middle ground as it flips the script. Corpo lawyers are allergic to AGPL and will pay for an alternative license, but for community hackers that might want to improve and recommend your code it offers no restrictions.
We're open to more feedback, things could change in the future
Nearly everyone just pays them to run it (myself included - I self host GitLab and still pay them!).
That's not just specific to digital stuff. E.g. warehouse cameras that film anything else beyond securing entries/exits, like they are commonplace in e.g. the US don't exist for the same reason.
We also have early support for encryption at rest through our CLI
if this doesn't run fully local its a no go for enterprise let alone ordinary users
Ultimately, it’s up to you what you run and trust
Screenpipe is open source and local-first
You can inspect it and keep your data on-device, with encryption at rest
you're being unnecessarily hostile about a threat that almost certainly doesn't exist. And if it does, you can absolutely block those uploads before they even happen. In fact if you don't already block uploads to hosts you don't know about, then you're not really taking anything seriously, are you?
Source available *
https://screenpipe.com /how-to-install
I think that sucks, but it's certainly not what I thought of when I read your one-liner (which implies that they harvest and abuse your actual recorded data).
Have you ever founded a company and tried to find paying customers?
You'd probably be offended if I sent you a cute 90's era AOL CD with our software on it.
OP: I wouldn't be offended if you emailed me in this way. Ignore the naysayers. This is perfectly kosher.
In fact, I'm going to check out your company in depth just because of this pearl clutching.
Keep building and don't let naysayers tear you down over something not even slightly evil. (Venomous attacks are more evil than this, frankly.)
Please don't create accounts to break HN's rules with.
For one, even taking the ethos of local-only as something bulletproof, I can't think of a pro that would outweigh the con of recording everything I do. Part of that might be my lack of imagination, as I'm not a heavy user of ai besides basic Claude code.
Most important though, the concept is ripe for pivoting away from local-only. It will only take a juicy investment offer and I'm sure local-onpy will be quietly forgotten.
We're the only local-first, open source option
How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
>How much time do you spend explaining the context to your Claude? Or writing detailed prompts? Copy pasting documents into the prompt?
Prompts: not much. If I feel my prompt is becoming a paragraph-long text, that's when I know I probably haven't thought things through enough yet. Documents: I typically have a interpreter console open where I'll do something like File.write("foo.html", page.body), and I'll just tell Claude to "find the missing div in foo.html". Or if its a file already in the repo, I won't need to copy paste anything. Context: also not much. It has all the context by default, doesn't it?
The main difference is the product direction. Screenpipe seems focused on continuously giving agents context through APIs, MCP, and skills. Daydream is more narrowly built around answering "what did I do today?" through a timeline you can inspect, replay, search, and turn into a daily digest.
I'm also treating deletion as part of the data model. If you cut a sensitive span, its frames, audio, OCR, transcripts, embeddings, and summaries should be deleted or invalidated too.
Mine is still early and Linux-first. I'm open-sourcing it in case anyone wants to contribute, poke around, or use it as a starting point. It’s built with Tauri, a Rust backend, React/TypeScript, SQLite, GStreamer, Whisper, OCR, and VLM processing.
I genuinely didn’t know you were building this when I started. Apparently personal memory capture is becoming a SaaS category too lol.
Code is here: https://github.com/snackbit/daydream