On one project team, “approved” means the people and budget are ready. On another, it means funding is available and staffing comes later. An AI assistant working with both teams could collect plenty of accurate information and still give poor advice. Its usefulness depends on carrying each lesson into the right situation.
Thinking in terms of learning makes us specify what should change, where the lesson belongs, and how we will check whether it helped.
The ambition and the work we can do now
Rich Sutton’s 2019 essay The Bitter Lesson asks us to look at how progress in AI has happened. His argument is that general methods such as search and learning become especially powerful as more computation becomes available [1]. That gives us a reason to care about a system’s ability to discover better ways of working. For a project assistant, we can ask what would allow its advice to improve after it encounters something its designers did not anticipate.
David Silver and Sutton take that ambition further in Welcome to the Era of Experience. They describe agents that act in the world, observe the consequences, and adapt through a continuing stream of experience [2]. Imagine an assistant that follows a project for months, checks what happened after its recommendations, and uses those outcomes to improve later decisions. That is the kind of experiential learning relevant to our discussion.
When dependable systems of this breadth will be available remains uncertain. Learning from experience already works in bounded settings, and Silver and Sutton include adaptation through context among its possible mechanisms [2]. The larger ambition involves sustaining useful learning across changing situations and long periods. More computing power may support that ambition, but a project system would still need access to relevant events and a meaningful way to judge its results. The big world hypothesis also cautions us about the idea of eventually learning everything. Javed and Sutton describe problems where the world is larger than an agent can fully represent [3]. Even a capable agent must select what to learn about its particular circumstances.
We can begin working on that problem with the models available to us. A model’s weights are the numerical parameters established through training. Those parameters can remain fixed while the surrounding system gives the model different information on its next attempt. If that information includes a useful lesson from an earlier outcome, it can change how the model approaches the task.
Reflexion, introduced by Shinn and colleagues, provides a concrete example [4]. An agent first attempts a task. An evaluator checks the attempt and provides feedback. A language model then examines the attempt alongside that feedback and writes a short reflection about what went wrong and what to try next. The system stores that reflection and supplies it as context for a later attempt [4].
The stored reflection gives the agent a specific basis for changing its approach. In the paper, this process supported improvements on several coding, reasoning, and simulated decision tasks, although results depended on the task and the quality of evaluation [4]. A mistaken evaluation can produce a misleading lesson. The process therefore needs feedback that is worth learning from.
For our illustrative project assistant, the same design idea would mean reviewing a missed start forecast, identifying the assumption behind it, and retaining a proposed check for next time. Applying this approach to projects would require its own testing. Its immediate value is that it makes learning a concrete system design concern. We must decide how outcomes become feedback, how feedback becomes a lesson, and how that lesson reaches a later decision.
Give new evidence somewhere to go
Return to the two teams. Suppose the assistant forecasts a Monday start for Team B after a budget approval. Monday arrives without an assigned crew. A project manager confirms that staffing requires a separate decision. Processing this event into a useful lesson takes several steps.
First, preserve enough of the event to understand the mistake. Record the forecast, the approval it relied on, and the actual outcome. Add the project manager’s explanation with its source and date. Keeping the observation and its interpretation separately makes the lesson easier to review. A missed start alone leaves several possible explanations open, including a staffing problem, a delivery delay, or a changed priority.
Next, propose a change in how the assistant should handle a similar decision. “Be more careful with approvals” gives little guidance. “For Team B, confirm staffing before treating budget approval as a start commitment” identifies both the relevant situation and the action to take. The strength of the lesson should match its evidence. One event might justify a provisional check. Confirmation of the team’s formal process would provide a stronger basis for making that check routine.
Then decide where the lesson belongs. Someone responsible for the workflow should check it against existing guidance and decide whether to accept it, revise it, or seek more evidence. The accepted lesson can become part of the instructions used when assessing Team B’s readiness. Keeping it attached to that team prevents a local observation from silently becoming a rule for every project.
The system must also bring the lesson back at the right moment. If it stays buried in a meeting transcript, a later forecast may overlook it. Context curation means selecting the relevant instructions and supporting evidence when the assistant prepares its answer. Here, that means supplying Team B’s staffing check alongside the new approval.
Finally, test what changed. Give the assistant a new Team B approval and see whether it asks about staffing. Give it a Team A case as well to check whether the lesson has created an unnecessary obstacle there. Track subsequent forecasts against actual starts, then revisit the instruction when the evidence or team process changes. A lesson needs a path through review, use, testing, and revision throughout its useful life.
Let agents exchange lessons with their conditions
Separate agents could maintain the working knowledge of each team and send relevant lessons to a coordinating agent. Each report would carry the observation, the proposed lesson, its supporting evidence, and the circumstances in which it applies.
Their reports may appear contradictory. Team A treats approval as readiness to start. Team B requires a staffing check. Preserving both accounts lets the coordinating agent ask what each approval actually commits the team to. Adopting either account as a universal rule would lose that distinction.
Transfer can hurt performance when knowledge from one setting is poorly suited to another. Machine learning research calls this negative transfer [5]. For our proposed project system, this suggests testing a shared lesson in the receiving team’s setting before allowing it to guide decisions there.
Consider a more capable agent with a large computing budget. It could spend resources improving a shared capability or send specialist agents to learn about particular domains. My view is that the useful choice depends on what is missing. Better reasoning may help all teams, while knowledge of local approval practices requires access to those teams. Specialists also create work for the parent agent, which must evaluate what comes back and decide what should remain local.
Build the procedures that help the system learn
Our practical opportunity today is to use AI models within procedures that improve as we work.
Each recurring task and specialty can have its own procedural context, a maintained account of how to approach the work, what to check, which evidence to consult, and when to ask for help.
A useful procedure develops through contact with the work.
The assistant applies it, people review its recommendations, and subsequent events provide evidence about what helped or failed. We can use that evidence to revise the instructions, keep the reasons for each revision, and test whether the next decision improves. Related tasks can share a common method while retaining the local details that change how it should be applied.
Explain It to a Friend
What if an organization gave every AI agent everything its colleagues had learned, then found that its advice became worse? One agent’s reliable shortcut could become another’s mistake. Bringing their experience together might erase the conditions that made each lesson useful.
Perhaps part of intelligence lies in preserving differences long enough to understand them.
A shared system would then need a way to keep disagreement available, even while coordinating action. What would we lose if every agent had to settle on the same account of how the world works? How would you explain this to a friend?
Notes
This post is part of the PM Along AI series. See all episodes here.
Join the EPM Network to access insights, influence our research, and connect with a community shaping the industry’s future.
You are not receiving future PM Along AI posts simply by subscribing to EPM Research. Learn how to receive future PM Along AI posts.
Support us by sharing this article with your friends and colleagues, or over social media.
References
1. Sutton, R. S. (2019). The bitter lesson.
2. Silver, D., & Sutton, R. S. (2025). Welcome to the era of experience.
3. Javed, K., & Sutton, R. S. (2024). The big world hypothesis and its ramifications for artificial intelligence. Oak Lab.
4. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. arXiv.
5. Wang, Z., Dai, Z., Póczos, B., & Carbonell, J. (2019). Characterizing and avoiding negative transfer. arXiv. Published at CVPR 2019.