Status succeeded
Event issue:assigned on abrander/zoo#59
Agent greg greg

Log

Thinking
Let's start by understanding the project structure and finding the agent loop that needs the retry logic.
Tool πŸ”§ list_files {"path": ".", "recursive": true}
{"path": ".", "recursive": true}
/project/README.md
/project/cmd
/project/cmd/zoo
/project/cmd/zoo/main.go
/project/docker
/project/docker/Dockerfile
/project/docker/group
/project/docker/passwd
/project/docker/shadow
/project/docker/uid1000-sudo
/project/go.mod
/project/go.sum
/project/internal
/project/internal/agentrun
/project/internal/agentrun/context.go
/project/internal/agentrun/docker.go
/project/internal/agentrun/docker_integration_test.go
/project/internal/agentrun/forgejo.go
/project/internal/agentrun/loop.go
/project/internal/agentrun/roster.go
/project/internal/agentrun/roster_test.go
/project/internal/agentrun/run.go
/project/internal/agentrun/sandboxgit.go
/project/internal/agentrun/sandboxgit_test.go
/project/internal/agentrun/seed_test.go
/project/internal/agentrun/system.md
/project/internal/config
/project/internal/config/config.go
/project/internal/config/config_test.go
/project/internal/forgejo
/project/internal/forgejo/client.go
/project/internal/forgejo/client_test.go
/project/internal/forgejo/event.go
/project/internal/forgejo/poll.go
/project/internal/forgejo/poll_test.go
/project/internal/forgejo/repo_filter.go
/project/internal/forgejo/repo_filter_test.go
/project/internal/forgejo/watcher.go
/project/internal/forgejo/watcher_test.go
/project/internal/forgejo/webhook.go
/project/internal/forgejo/webhook_test.go
/project/internal/livelog
/project/internal/livelog/hub.go
/project/internal/livelog/hub_test.go
/project/internal/llm
/project/internal/llm/client.go
/project/internal/llm/streamer.go
/project/internal/llm/streamer_test.go
/project/internal/llm/types.go
/project/internal/scheduler
/project/internal/scheduler/scheduler.go
/project/internal/scheduler/scheduler_test.go
/project/internal/store
/project/internal/store/schema.sql
/project/internal/store/store.go
/project/internal/store/store_test.go
/project/internal/tools
/project/internal/tools/bash.go
/project/internal/tools/context.go
/project/internal/tools/edit-file.go
/project/internal/tools/forgejo.go
/project/internal/tools/grep_search.go
/project/internal/tools/list_files.go
/project/internal/tools/move_file.go
/project/internal/tools/read_file.go
/project/internal/tools/registry.go
/project/internal/tools/remove_file.go
/project/internal/tools/shell.go
/project/internal/tools/tools_test.go
/project/internal/tools/write_file.go
/project/internal/web
/project/internal/web/templates.go
/project/internal/web/web.go
/project/internal/web/web_test.go
/project/zoo.hcl.example
Tool πŸ”§ bash {"command": "cd /project && git log --oneline -5 && git status"}
{"command": "cd /project && git log --oneline -5 && git status"}
98298f6 Merge pull request 'Fix excessive vertical space in job view blocks (#53)' (#55) from issue-53-greg into main
b75b64d Fix excessive vertical space in job view blocks
10c3dce Better Docker/git inter.
315f993 Add docker image build.
5717258 Merge pull request 'forgejo: repository allowlist with wildcard patterns' (#40) from issue-39-greg into main
On branch issue-59-greg
nothing to commit, working tree clean
Thinking
Let's take a look at the agent loop and the LLM client to understand where the retry logic should be placed.
Tool πŸ”§ read_file {"end_line": 400, "path": "internal/agentrun/loop.go", "start_line": 1}
{"end_line": 400, "path": "internal/agentrun/loop.go", "start_line": 1}
{"path":"internal/agentrun/loop.go","content":"package agentrun\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/abrander/zoo/internal/llm\"\n\t\"github.com/abrander/zoo/internal/tools\"\n)\n\n// Hooks lets a caller observe a runLoop invocation live, as the model\n// streams and tool calls execute, in addition to the final []llm.Message\n// it returns. Any of these may be nil.\ntype Hooks struct {\n\t// OnReasoningDelta and OnContentDelta fire with just the newly\n\t// streamed text for the current turn, not the accumulated total.\n\tOnReasoningDelta func(delta string)\n\tOnContentDelta   func(delta string)\n\n\t// OnTurnEnd fires once per completed streamer round-trip, after the\n\t// model's message for that turn is fully received and before any of\n\t// its tool calls run.\n\tOnTurnEnd func()\n\n\t// OnTool fires once per tool call, after it has run.\n\tOnTool func(name, arguments, result string, toolErr bool)\n}\n\n// runLoop is a headless port of ../a's App.generate(): send messages +\n// tool defs, get a completion, run any tool_calls and append their\n// results, repeat until a plain finish or ctx is done.\nfunc runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) {\n\tfor {\n\t\tif err := ctx.Err(); err != nil {\n\t\t\treturn messages, err\n\t\t}\n\n\t\tstreamer, err := client.StreamChatCompletion(ctx, \u0026llm.ChatCompletionRequest{\n\t\t\tMessages: messages,\n\t\t\tStream:   true,\n\t\t\tTools:    tools.All(),\n\t\t})\n\t\tif err != nil {\n\t\t\treturn messages, fmt.Errorf(\"chat completion: %w\", err)\n\t\t}\n\n\t\tvar completion *llm.ChatCompletion\n\n\t\tvar prevContent, prevReasoning string\n\n\t\tfor {\n\t\t\tc, err := streamer.Get()\n\t\t\tif err == io.EOF {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tif err != nil {\n\t\t\t\treturn messages, fmt.Errorf(\"stream completion: %w\", err)\n\t\t\t}\n\n\t\t\tcompletion = c\n\n\t\t\tif len(c.Choices) \u003e 0 {\n\t\t\t\tmsg := c.Choices[0].Message\n\n\t\t\t\tif hooks.OnReasoningDelta != nil \u0026\u0026 len(msg.ReasoningContent) \u003e len(prevReasoning) {\n\t\t\t\t\thooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):])\n\t\t\t\t}\n\t\t\t\tprevReasoning = msg.ReasoningContent\n\n\t\t\t\tif hooks.OnContentDelta != nil \u0026\u0026 len(msg.Content) \u003e len(prevContent) {\n\t\t\t\t\thooks.OnContentDelta(msg.Content[len(prevContent):])\n\t\t\t\t}\n\t\t\t\tprevContent = msg.Content\n\t\t\t}\n\t\t}\n\n\t\tif hooks.OnTurnEnd != nil {\n\t\t\thooks.OnTurnEnd()\n\t\t}\n\n\t\tif completion == nil || len(completion.Choices) == 0 {\n\t\t\treturn messages, fmt.Errorf(\"model returned an empty completion\")\n\t\t}\n\n\t\tchoice := completion.Choices[0]\n\t\tmessages = append(messages, choice.Message)\n\n\t\tif choice.FinishReason != \"tool_calls\" || len(choice.Message.ToolCalls) == 0 {\n\t\t\treturn messages, nil\n\t\t}\n\n\t\tfor _, tc := range choice.Message.ToolCalls {\n\t\t\tif err := ctx.Err(); err != nil {\n\t\t\t\treturn messages, err\n\t\t\t}\n\n\t\t\tresult, err := tools.Call(toolsCtx, tc.FunctionCall)\n\n\t\t\ttoolErr := err != nil\n\t\t\tif toolErr {\n\t\t\t\tresult = fmt.Sprintf(\"ERROR CALLING TOOL: %s\", err.Error())\n\t\t\t}\n\n\t\t\tif hooks.OnTool != nil {\n\t\t\t\thooks.OnTool(tc.FunctionCall.Name, tc.FunctionCall.Arguments, result, toolErr)\n\t\t\t}\n\n\t\t\tmessages = append(messages, llm.Message{\n\t\t\t\tRole:       \"tool\",\n\t\t\t\tContent:    result,\n\t\t\t\tToolCallID: tc.Id,\n\t\t\t\tName:       tc.FunctionCall.Name,\n\t\t\t})\n\t\t}\n\t}\n}","start_line":1,"end_line":118,"total_lines":118,"truncated":false}
Tool πŸ”§ read_file {"end_line": 200, "path": "internal/llm/client.go", "start_line": 1}
{"end_line": 200, "path": "internal/llm/client.go", "start_line": 1}
{"path":"internal/llm/client.go","content":"package llm\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"net/http\"\n\n\t\"github.com/abrander/zoo/internal/config\"\n)\n\ntype Client struct {\n\tendpoint string\n\ttoken    string\n\tmodel    string\n\n\thttpClient *http.Client\n}\n\n// NewClient builds a Client from a configured llm block.\nfunc NewClient(cfg config.LLM) *Client {\n\treturn \u0026Client{\n\t\tendpoint:   cfg.OpenAI,\n\t\ttoken:      cfg.Token,\n\t\tmodel:      cfg.Model,\n\t\thttpClient: http.DefaultClient,\n\t}\n}\n\nfunc (c *Client) StreamChatCompletion(ctx context.Context, req *ChatCompletionRequest) (*Streamer, error) {\n\treq2 := *req\n\n\tif req2.Model == \"\" {\n\t\treq2.Model = c.model\n\t}\n\n\treq2.Stream = true\n\treq2.Seed = 5\n\treq2.Temperature = 0.6\n\treq2.TopP = 0.95\n\n\tbody, err := json.Marshal(req2)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Bound to ctx so a stalled connection or silent server (headers or\n\t// stream body never arriving) is unblocked by the run's deadline\n\t// instead of hanging the goroutine forever β€” http.DefaultClient has\n\t// no timeout of its own.\n\tr, err := http.NewRequestWithContext(ctx, http.MethodPost, c.endpoint+\"/chat/completions\", bytes.NewBuffer(body))\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tr.Header.Set(\"Authorization\", \"Bearer \"+c.token)\n\tr.Header.Set(\"Content-Type\", \"application/json\")\n\n\tresp, err := c.httpClient.Do(r)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn NewStreamer(resp)\n}","start_line":1,"end_line":65,"total_lines":65,"truncated":false}
Thinking
Let's take a look at the streamer and types, and also the run.go that calls runLoop.
Tool πŸ”§ read_file {"end_line": 200, "path": "internal/llm/streamer.go", "start_line": 1}
{"end_line": 200, "path": "internal/llm/streamer.go", "start_line": 1}
{"path":"internal/llm/streamer.go","content":"package llm\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n)\n\ntype Streamer struct {\n\tcompletion ChatCompletion\n\tresp       *http.Response\n\tscanner    *bufio.Scanner\n\teof        bool\n}\n\nfunc NewStreamer(resp *http.Response) (*Streamer, error) {\n\tif resp.StatusCode != http.StatusOK {\n\t\tbody, _ := io.ReadAll(resp.Body)\n\t\tresp.Body.Close()\n\t\treturn nil, fmt.Errorf(\"status: %d: %s\", resp.StatusCode, body)\n\t}\n\n\treturn \u0026Streamer{\n\t\tresp:    resp,\n\t\tscanner: bufio.NewScanner(resp.Body),\n\t}, nil\n}\n\n// Get returns the accumulated completion after merging in the next SSE\n// chunk. Callers should keep calling Get until it returns io.EOF.\nfunc (s *Streamer) Get() (*ChatCompletion, error) {\n\tif s.eof {\n\t\treturn nil, io.EOF\n\t}\n\n\tprefix := []byte(\"data: \")\n\n\tvar buf []byte\n\n\tfor s.scanner.Scan() {\n\t\tif err := s.scanner.Err(); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\tbuf = bytes.TrimSpace(s.scanner.Bytes())\n\n\t\tif len(buf) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif bytes.HasPrefix(buf, prefix) {\n\t\t\tbreak\n\t\t}\n\t}\n\n\tbuf = bytes.TrimPrefix(buf, prefix)\n\n\tif bytes.Equal(buf, []byte(\"[DONE]\")) {\n\t\ts.resp.Body.Close()\n\t\ts.eof = true\n\t\treturn nil, io.EOF\n\t}\n\n\tvar completion ChatCompletion\n\n\t// Unknown fields are tolerated on purpose: servers (e.g. newer\n\t// llama.cpp builds) may add fields we don't know about yet, and\n\t// failing on them would break compatibility.\n\tif err := json.Unmarshal(buf, \u0026completion); err != nil {\n\t\treturn nil, fmt.Errorf(\"decode completion chunk: %w: %s\", err, buf)\n\t}\n\n\ts.completion.Merge(\u0026completion)\n\n\treturn \u0026s.completion, nil\n}","start_line":1,"end_line":79,"total_lines":79,"truncated":false}
Tool πŸ”§ read_file {"end_line": 250, "path": "internal/agentrun/run.go", "start_line": 1}
{"end_line": 250, "path": "internal/agentrun/run.go", "start_line": 1}
{"path":"internal/agentrun/run.go","content":"// Package agentrun creates the throwaway per-job environment an agent\n// runs in: a Docker container with the agent's git working tree cloned\n// inside it (authenticated with the agent's own Forgejo token), the\n// triggering event written to /event, and the tool-calling loop\n// (internal/llm + internal/tools) driven against it.\npackage agentrun\n\nimport (\n\t\"context\"\n\t_ \"embed\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"log/slog\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/abrander/zoo/internal/config\"\n\t\"github.com/abrander/zoo/internal/forgejo\"\n\t\"github.com/abrander/zoo/internal/livelog\"\n\t\"github.com/abrander/zoo/internal/llm\"\n\t\"github.com/abrander/zoo/internal/store\"\n)\n\n//go:embed system.md\nvar defaultSystemPrompt string\n\n// DefaultTimeout bounds a single agent run's wall-clock time if the\n// caller doesn't override it.\nconst DefaultTimeout = 120 * time.Minute\n\ntype Runner struct {\n\tdocker        *dockerRuntime\n\tforgejo       *forgejo.Client\n\tstore         *store.Store\n\thub           *livelog.Hub\n\tcfg           *config.Config\n\tlogger        *slog.Logger\n\ttimeout       time.Duration\n\tkeepOnFailure bool\n\n\tagentClientsMu sync.Mutex\n\tagentClients   map[string]*forgejo.Client\n}\n\nfunc NewRunner(cfg *config.Config, fg *forgejo.Client, st *store.Store, hub *livelog.Hub, logger *slog.Logger, timeout time.Duration, keepOnFailure bool) (*Runner, error) {\n\tdocker, err := newDockerRuntime()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif timeout \u003c= 0 {\n\t\ttimeout = DefaultTimeout\n\t}\n\n\treturn \u0026Runner{\n\t\tdocker:        docker,\n\t\tforgejo:       fg,\n\t\tstore:         st,\n\t\thub:           hub,\n\t\tcfg:           cfg,\n\t\tlogger:        logger,\n\t\ttimeout:       timeout,\n\t\tkeepOnFailure: keepOnFailure,\n\t\tagentClients:  make(map[string]*forgejo.Client),\n\t}, nil\n}\n\n// forgejoAs returns a Forgejo client that authenticates as the given\n// agent (using the agent's own token from config). This lets each agent\n// act as themselves on Forgejo without needing a global token with sudo\n// privileges. Clients are built once per agent and cached, since\n// constructing one costs an extra API round trip.\n//\n// If the agent has no token configured, falls back to the shared zoo\n// identity so existing deployments without per-agent tokens still work.\nfunc (r *Runner) forgejoAs(agentName, token string) *forgejo.Client {\n\tr.agentClientsMu.Lock()\n\tdefer r.agentClientsMu.Unlock()\n\n\tif c, ok := r.agentClients[agentName]; ok {\n\t\treturn c\n\t}\n\n\tvar c *forgejo.Client\n\tif token != \"\" {\n\t\tc = r.forgejo.As(token)\n\t} else {\n\t\t// Fallback: use shared identity. Optionally log a warning\n\t\t// if we ever want to enforce per-agent tokens.\n\t\tc = r.forgejo\n\t}\n\n\tr.agentClients[agentName] = c\n\n\treturn c\n}\n\n// Run implements scheduler.Runner.\nfunc (r *Runner) Run(ctx context.Context, jobID string, agent config.AgentConfig, llmCfg config.LLM, dockerImage string, ev forgejo.Event) error {\n\tctx, cancel := context.WithTimeout(ctx, r.timeout)\n\tdefer cancel()\n\n\tlogger := r.logger.With(\"job\", jobID, \"agent\", agent.Name)\n\n\trepoInfo, err := r.forgejo.RepositoryInfo(ev.Owner, ev.Repo)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"look up repository: %w\", err)\n\t}\n\n\tworkDir, err := os.MkdirTemp(\"\", \"zoo-run-*\")\n\tif err != nil {\n\t\treturn fmt.Errorf(\"create work dir: %w\", err)\n\t}\n\n\tsucceeded := false\n\n\tdefer func() {\n\t\tif succeeded || !r.keepOnFailure {\n\t\t\tos.RemoveAll(workDir)\n\t\t} else {\n\t\t\tlogger.Warn(\"keeping work dir after failure\", \"dir\", workDir)\n\t\t}\n\t}()\n\n\t// The container bind-mounts projectDir as /project and does the\n\t// initial clone into it, so the (empty) directory must exist on the\n\t// host before the container is created β€” otherwise Docker would\n\t// create it itself, root-owned.\n\tprojectDir := filepath.Join(workDir, \"project\")\n\n\tif err := os.MkdirAll(projectDir, 0o755); err != nil {\n\t\treturn fmt.Errorf(\"create project dir: %w\", err)\n\t}\n\n\t// A pr:review run works on the PR's own head branch, so the agent's\n\t// commits push straight to the PR. Every other event kind branches\n\t// off the default branch as usual.\n\tvar review *forgejo.ReviewDetail\n\tbranch := fmt.Sprintf(\"issue-%d-%s\", ev.Index, agent.Name)\n\n\tif ev.Kind == forgejo.EventPRReview {\n\t\t// Always fetch the current head ref, not just when the event\n\t\t// lacks one (the polling path doesn't carry it): the webhook's\n\t\t// copy could be stale if the PR's head branch was renamed since\n\t\t// the review, and the push target depends on it.\n\t\theadRef := ev.HeadRef\n\n\t\tif prInfo, err := r.forgejo.PullRequestInfo(ev.Owner, ev.Repo, ev.Index); err != nil {\n\t\t\tlogger.Warn(\"fetch pull request head failed; falling back to the event's head ref\", \"error\", err)\n\t\t} else if prInfo.HeadRef != \"\" {\n\t\t\theadRef = prInfo.HeadRef\n\t\t}\n\n\t\tif headRef == \"\" {\n\t\t\treturn fmt.Errorf(\"pr:review event has no pull request head branch to check out\")\n\t\t}\n\n\t\tbranch = headRef\n\n\t\t// Fetch the full review (verdict, body, inline comments) so the\n\t\t// agent sees all the feedback, not just the triggering event. A\n\t\t// failure degrades to no review detail rather than failing the\n\t\t// run: the agent can still do its job, just without the inline\n\t\t// comments.\n\t\treview, err = r.forgejo.ReviewDetail(ev.Owner, ev.Repo, ev.Index, ev.ReviewID)\n\t\tif err != nil {\n\t\t\tlogger.Warn(\"fetch review detail failed; agent will not see inline review comments\", \"error\", err)\n\t\t\treview = nil\n\t\t}\n\t}\n\n\troster := buildRoster(r.forgejo, r.cfg.Agents, logger)\n\tgitName, gitEmail := gitIdentity(agent.Name, roster)\n\n\t// The credential the sandbox's git uses for remote operations: the\n\t// agent's own Forgejo token when configured, so its git activity is\n\t// attributed to its own account, falling back to the shared zoo\n\t// identity for deployments without per-agent tokens (mirroring\n\t// forgejoAs).\n\tgitUser, gitToken := \"zoo\", r.forgejo.Token()\n\n\tif agent.Token != \"\" {\n\t\tgitUser, gitToken = agent.Name, agent.Token\n\t}\n\n\teventPath := filepath.Join(workDir, \"event.json\")\n\tif err := os.WriteFile(eventPath, ev.Raw, 0o644); err != nil {\n\t\treturn fmt.Errorf(\"write event file: %w\", err)\n\t}\n\n\tcontainerID, err := r.docker.createContainer(ctx, dockerImage, []string{\n\t\tprojectDir + \":/project\",\n\t\teventPath + \":/event:ro\",\n\t}, fmt.Sprintf(\"zoo-issue-%d-%s\", ev.Index, agent.Name))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"start container: %w\", err)\n\t}\n\n\tdefer func() {\n\t\tcleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 30*time.Second)\n\t\tdefer cleanupCancel()\n\t\tif err := r.docker.remove(cleanupCtx, containerID); err != nil {\n\t\t\tlogger.Warn(\"failed to remove container\", \"container\", containerID, \"error\", err)\n\t\t}\n\t}()\n\n\t// Git must simply work inside the sandbox: safe.directory, commit\n\t// identity, and the remote credential all go into the container's\n\t// system gitconfig (see configureSandboxGit).\n\tif err := configureSandboxGit(ctx, r.docker, containerID, repoInfo.CloneURL, gitUser, gitToken, gitName, gitEmail); err != nil {\n\t\treturn fmt.Errorf(\"configure git in container: %w\", err)\n\t}\n\n\t// The initial clone happens inside the sandbox, so the working tree\n\t// is owned by the container's user and git never runs on the host.\n\tif ev.Kind == forgejo.EventPRReview {\n\t\tif err := clonePRHead(ctx, r.docker, containerID, repoInfo.CloneURL, repoInfo.DefaultBranch, branch, ev.Index); err != nil {\n\t\t\treturn fmt.Errorf(\"prepare git working tree: %w\", err)\n\t\t}\n\t} else {\n\t\tif err := cloneAndBranch(ctx, r.docker, containerID, repoInfo.CloneURL, repoInfo.DefaultBranch, branch); err != nil {\n\t\t\treturn fmt.Errorf(\"prepare git working tree: %w\", err)\n\t\t}\n\t}\n\n\tlogAppend := func(stream, line string) {\n\t\tif err := r.store.AppendLog(context.Background(), jobID, stream, line); err != nil {\n\t\t\tlogger.Warn(\"failed to append log\", \"error\", err)\n\t\t}\n\t}\n\n\trunCtx := \u0026runContext{\n\t\tdocker:      r.docker,\n\t\tcontainerID: containerID,\n\t\tforgejo: \u0026runForgejoActions{\n\t\t\tclient: r.forgejoAs(agent.Name, agent.Token),\n\t\t\towner:  ev.Owner,\n\t\t\trepo:   ev.Repo,\n\t\t\tindex:  ev.Index,\n\t\t\tlogger: logger,\n\t\t},\n\t}\n\n\tllmClient := llm.NewClient(llmCfg)\n\n\tsystemPrompt := defaultSystemPrompt + identitySection(agent.Name, roster)\n","start_line":1,"end_line":250,"total_lines":437,"truncated":true}
Thinking
Let's take a look at the rest of run.go, the llm types, and also the config to understand how the LLM configuration is structured.
Tool πŸ”§ read_file {"end_line": 437, "path": "internal/agentrun/run.go", "start_line": 250}
{"end_line": 437, "path": "internal/agentrun/run.go", "start_line": 250}
{"path":"internal/agentrun/run.go","content":"\n\tinstructions := r.cfg.EventInstructions(ev.Kind)\n\n\t// Fetch the full comment thread so the agent sees everything that's\n\t// been said on the issue/PR, not just the triggering event (which\n\t// only carries the latest comment, if any). A failure degrades to\n\t// no comments rather than failing the run: the agent can still do\n\t// its job, just without prior context.\n\tcomments, err := r.forgejo.ListIssueComments(ev.Owner, ev.Repo, ev.Index)\n\tif err != nil {\n\t\tlogger.Warn(\"fetch issue comments failed; agent will not see prior comments\", \"error\", err)\n\t\tcomments = nil\n\t}\n\n\tmessages := []llm.Message{\n\t\t{Role: \"system\", Content: systemPrompt},\n\t\t{Role: \"user\", Content: seedMessage(ev, branch, repoInfo.DefaultBranch, instructions, comments, review)},\n\t}\n\n\thooks := r.streamHooks(jobID, logAppend)\n\n\t_, err = runLoop(ctx, llmClient, runCtx, messages, hooks)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"agent loop: %w\", err)\n\t}\n\n\tsucceeded = true\n\n\treturn nil\n}\n\n// streamHooks builds the Hooks a single Run passes to runLoop: every\n// delta is published live to the hub for connected dashboard viewers,\n// and once a reasoning/content block or tool call is complete, it's\n// persisted to the store as one row and the hub's replay buffer for\n// jobID is checkpointed β€” so a viewer connecting from this point on\n// sees it via the persisted history instead of a live replay, and is\n// never shown it twice.\nfunc (r *Runner) streamHooks(jobID string, logAppend func(stream, line string)) Hooks {\n\tvar reasoningBuf, contentBuf strings.Builder\n\n\treasoningOpen, contentOpen := false, false\n\n\treturn Hooks{\n\t\tOnReasoningDelta: func(delta string) {\n\t\t\tif !reasoningOpen {\n\t\t\t\treasoningOpen = true\n\t\t\t\treasoningBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningStart})\n\t\t\t}\n\n\t\t\treasoningBuf.WriteString(delta)\n\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningDelta, Text: delta})\n\t\t},\n\t\tOnContentDelta: func(delta string) {\n\t\t\tif !contentOpen {\n\t\t\t\tcontentOpen = true\n\t\t\t\tcontentBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentStart})\n\t\t\t}\n\n\t\t\tcontentBuf.WriteString(delta)\n\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentDelta, Text: delta})\n\t\t},\n\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},\n\t\tOnTool: func(name, arguments, result string, toolErr bool) {\n\t\t\tr.hub.Publish(jobID, livelog.Event{\n\t\t\t\tType:      livelog.Tool,\n\t\t\t\tName:      name,\n\t\t\t\tArguments: arguments,\n\t\t\t\tResult:    result,\n\t\t\t\tError:     toolErr,\n\t\t\t})\n\n\t\t\tline, err := json.Marshal(store.ToolLogEntry{Name: name, Arguments: arguments, Result: result, Error: toolErr})\n\t\t\tif err != nil {\n\t\t\t\tr.logger.Warn(\"failed to marshal tool log entry\", \"job\", jobID, \"error\", err)\n\t\t\t} else {\n\t\t\t\tlogAppend(\"tool\", string(line))\n\t\t\t}\n\n\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}\n\nfunc seedMessage(ev forgejo.Event, branch, defaultBranch, instructions string, comments []forgejo.IssueComment, review *forgejo.ReviewDetail) string {\n\traw, _ := json.MarshalIndent(json.RawMessage(ev.Raw), \"\", \"  \")\n\n\tvar instructionsSection string\n\tif instructions != \"\" {\n\t\tinstructionsSection = fmt.Sprintf(\"Instructions for this event, from zoo.hcl:\\n%s\\n\\n\", instructions)\n\t}\n\n\t// A pr:review run works on the PR's own head branch, not a fresh\n\t// branch off the default branch.\n\tbranchLine := fmt.Sprintf(\"Your working branch is %q, checked out from the default branch %q.\\n\\n\", branch, defaultBranch)\n\tif ev.Kind == forgejo.EventPRReview {\n\t\tbranchLine = fmt.Sprintf(\"Your working branch is %q, the pull request's head branch β€” commits you push here update the pull request directly.\\n\\n\", branch)\n\t}\n\n\tvar reviewSection string\n\tif review != nil {\n\t\treviewSection = renderReviewSection(review)\n\t}\n\n\tvar commentsSection string\n\tif len(comments) \u003e 0 {\n\t\tvar b strings.Builder\n\t\tfmt.Fprintf(\u0026b, \"Comments (%d):\\n\\n\", len(comments))\n\n\t\tfor i, c := range comments {\n\t\t\tfmt.Fprintf(\u0026b, \"%d. %s (%s):\\n%s\\n\\n\", i+1, c.Author, c.Created.Format(time.RFC3339), c.Body)\n\t\t}\n\n\t\tcommentsSection = b.String()\n\t}\n\n\treturn fmt.Sprintf(\n\t\t\"You were triggered by a %q event on %s/%s.\\n\\n\"+\n\t\t\t\"%s%s\"+\n\t\t\t\"%sTitle: %s\\n\\nBody:\\n%s\\n\\n%sFull event payload:\\n```json\\n%s\\n```\",\n\t\tev.Kind, ev.Owner, ev.Repo, instructionsSection, branchLine, reviewSection, ev.Title, ev.Body, commentsSection, raw)\n}\n\n// renderReviewSection renders the submitted review as a briefing\n// section: the verdict, the review body, and each inline comment with\n// its location and id (the id lets the agent refer to a specific\n// comment in its reply).\nfunc renderReviewSection(review *forgejo.ReviewDetail) string {\n\tvar b strings.Builder\n\n\tfmt.Fprintf(\u0026b, \"Review submitted by %s:\\n\", review.Reviewer)\n\tfmt.Fprintf(\u0026b, \"  Verdict: %s\\n\", humanReviewState(review.State))\n\n\tif review.Body != \"\" {\n\t\tb.WriteString(\"  Review body:\\n\")\n\t\tfor _, line := range strings.Split(review.Body, \"\\n\") {\n\t\t\tfmt.Fprintf(\u0026b, \"  %s\\n\", line)\n\t\t}\n\t}\n\n\tif len(review.Comments) \u003e 0 {\n\t\tfmt.Fprintf(\u0026b, \"  Inline comments (%d):\\n\", len(review.Comments))\n\n\t\tfor i, c := range review.Comments {\n\t\t\twhere := c.Path\n\t\t\tif c.Line \u003e 0 {\n\t\t\t\twhere = fmt.Sprintf(\"%s:%d\", c.Path, c.Line)\n\t\t\t}\n\n\t\t\tfmt.Fprintf(\u0026b, \"  %d. [%s] %s (comment id %d):\\n\", i+1, where, c.Author, c.ID)\n\t\t\tfor _, line := range strings.Split(c.Body, \"\\n\") {\n\t\t\t\tfmt.Fprintf(\u0026b, \"     %s\\n\", line)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn b.String() + \"\\n\"\n}\n\n// humanReviewState maps a Forgejo review state to a readable verdict.\nfunc humanReviewState(state string) string {\n\tswitch state {\n\tcase \"APPROVED\":\n\t\treturn \"approved\"\n\tcase \"REQUEST_CHANGES\":\n\t\treturn \"changes requested\"\n\tcase \"COMMENT\":\n\t\treturn \"comment\"\n\tdefault:\n\t\treturn state\n\t}\n}","start_line":250,"end_line":437,"total_lines":437,"truncated":false}
Tool πŸ”§ read_file {"end_line": 200, "path": "internal/llm/types.go", "start_line": 1}
{"end_line": 200, "path": "internal/llm/types.go", "start_line": 1}
{"path":"internal/llm/types.go","content":"// Package llm is an OpenAI-compatible chat completions client with\n// streaming and tool-call support, ported from ../a's llamacpp package.\npackage llm\n\nimport \"fmt\"\n\ntype Message struct {\n\tContent          string     `json:\"content\"`\n\tRole             string     `json:\"role\"`\n\tReasoningContent string     `json:\"reasoning_content,omitempty\"`\n\tToolCalls        []ToolCall `json:\"tool_calls,omitempty\"`\n\tToolCallID       string     `json:\"tool_call_id,omitempty\"`\n\tName             string     `json:\"name,omitempty\"`\n}\n\nfunc (m *Message) String() string {\n\tstr := fmt.Sprintf(\"Role: %s, Content: %s ReasoningContent: %s\", m.Role, m.Content, m.ReasoningContent)\n\n\tfor _, call := range m.ToolCalls {\n\t\tstr += fmt.Sprintf(\"\\n  ToolCall: %s\", call.String())\n\t}\n\n\treturn str\n}\n\n// Merge folds a streamed delta into the accumulated message.\nfunc (m *Message) Merge(next *Message) {\n\tm.Content += next.Content\n\n\tif next.Role != \"\" {\n\t\tm.Role = next.Role\n\t}\n\n\tm.ReasoningContent += next.ReasoningContent\n\n\tminLength := 0\n\tfor _, choice := range next.ToolCalls {\n\t\tif choice.Index+1 \u003e minLength {\n\t\t\tminLength = choice.Index + 1\n\t\t}\n\t}\n\n\tif len(m.ToolCalls) \u003c minLength {\n\t\tnewToolCalls := make([]ToolCall, minLength)\n\t\tcopy(newToolCalls, m.ToolCalls)\n\t\tm.ToolCalls = newToolCalls\n\t}\n\n\tfor _, nextChoice := range next.ToolCalls {\n\t\tm.ToolCalls[nextChoice.Index].Merge(\u0026nextChoice)\n\t}\n}\n\ntype FunctionCall struct {\n\tName      string `json:\"name\"`\n\tArguments string `json:\"arguments\"`\n}\n\ntype ToolCall struct {\n\tType         string       `json:\"type\"`\n\tId           string       `json:\"id\"`\n\tIndex        int          `json:\"index\"`\n\tName         string       `json:\"name\"`\n\tFunctionCall FunctionCall `json:\"function\"`\n}\n\nfunc (t *ToolCall) String() string {\n\treturn fmt.Sprintf(\"Type: %s, Id: %s, Index: %d, Name: %s, FunctionCall: {Name: %s, Arguments: %s}\",\n\t\tt.Type, t.Id, t.Index, t.Name, t.FunctionCall.Name, t.FunctionCall.Arguments)\n}\n\nfunc (t *ToolCall) Merge(next *ToolCall) {\n\tif next.Type != \"\" {\n\t\tt.Type = next.Type\n\t}\n\n\tif next.Id != \"\" {\n\t\tt.Id = next.Id\n\t}\n\n\tif next.Name != \"\" {\n\t\tt.Name = next.Name\n\t}\n\n\tt.Index = next.Index\n\n\tif next.FunctionCall.Name != \"\" {\n\t\tt.FunctionCall.Name = next.FunctionCall.Name\n\t}\n\n\tt.FunctionCall.Arguments += next.FunctionCall.Arguments\n}\n\ntype Property struct {\n\tType        string    `json:\"type\"`\n\tDescription string    `json:\"description,omitempty\"`\n\tEnum        []string  `json:\"enum,omitempty\"`\n\tItems       *Property `json:\"items,omitempty\"`\n}\n\ntype Parameters struct {\n\tType       string              `json:\"type\"` // object\n\tProperties map[string]Property `json:\"properties,omitempty\"`\n\tRequired   []string            `json:\"required,omitempty\"`\n}\n\ntype Function struct {\n\tName        string     `json:\"name\"`\n\tDescription string     `json:\"description,omitempty\"`\n\tParameters  Parameters `json:\"parameters,omitempty\"`\n\tStrict      bool       `json:\"strict\"`\n}\n\ntype Tool struct {\n\tType     string   `json:\"type\"`\n\tFunction Function `json:\"function\"`\n}\n\nfunc NewTool(name, description string) Tool {\n\treturn Tool{\n\t\tType: \"function\",\n\t\tFunction: Function{\n\t\t\tName:        name,\n\t\t\tDescription: description,\n\t\t\tStrict:      true,\n\t\t},\n\t}\n}\n\nfunc (t *Tool) AddProperty(name string, property Property, required bool) *Tool {\n\tif t.Function.Parameters.Properties == nil {\n\t\tt.Function.Parameters.Properties = make(map[string]Property)\n\t} else if _, exists := t.Function.Parameters.Properties[name]; exists {\n\t\tpanic(\"property already exists: \" + name)\n\t}\n\n\tt.Function.Parameters.Properties[name] = property\n\n\tif required {\n\t\tt.Function.Parameters.Required = append(t.Function.Parameters.Required, name)\n\t}\n\n\treturn t\n}\n\nfunc (t *Tool) AddStringProperty(name, description string, required bool) *Tool {\n\treturn t.AddProperty(name, Property{Type: \"string\", Description: description}, required)\n}\n\nfunc (t *Tool) AddBooleanProperty(name, description string, required bool) *Tool {\n\treturn t.AddProperty(name, Property{Type: \"boolean\", Description: description}, required)\n}\n\nfunc (t *Tool) AddIntegerProperty(name, description string, required bool) *Tool {\n\treturn t.AddProperty(name, Property{Type: \"integer\", Description: description}, required)\n}\n\nfunc (t *Tool) AddEnumProperty(name, description string, values []string, required bool) *Tool {\n\treturn t.AddProperty(name, Property{Type: \"string\", Description: description, Enum: values}, required)\n}\n\nfunc (t *Tool) AddArrayProperty(name, itemType, description string, required bool) *Tool {\n\treturn t.AddProperty(name, Property{\n\t\tType:        \"array\",\n\t\tDescription: description,\n\t\tItems:       \u0026Property{Type: itemType},\n\t}, required)\n}\n\nfunc (t *Tool) AddStringArrayProperty(name, description string, required bool) *Tool {\n\treturn t.AddArrayProperty(name, \"string\", description, required)\n}\n\ntype ChatCompletionRequest struct {\n\tModel       string    `json:\"model\"`\n\tMessages    []Message `json:\"messages,omitempty\"`\n\tStream      bool      `json:\"stream\"`\n\tTools       []Tool    `json:\"tools,omitempty\"`\n\tSeed        int       `json:\"seed,omitempty\"`\n\tTemperature float64   `json:\"temperature,omitempty\"`\n\tTopP        float64   `json:\"top_p,omitempty\"`\n}\n\ntype Choice struct {\n\tDelta        Message `json:\"delta\"`\n\tFinishReason string  `json:\"finish_reason,omitempty\"`\n\tIndex        int     `json:\"index\"`\n\tMessage      Message `json:\"message\"`\n\tRole         string  `json:\"role\"`\n}\n\nfunc (c *Choice) String() string {\n\treturn fmt.Sprintf(\"Index: %d, Role: %s, FinishReason: %v, Delta: %s, Message: %s\",\n\t\tc.Index, c.Role, c.FinishReason, c.Delta.String(), c.Message.String())\n}\n\nfunc (c *Choice) Merge(next *Choice) {\n\tc.Message.Merge(\u0026next.Delta)\n\n\tif next.FinishReason != \"\" {","start_line":1,"end_line":200,"total_lines":267,"truncated":true}
Thinking
Alright, let's check the LLM configuration and the existing tests to understand the conventions.
Tool πŸ”§ read_file {"end_line": 200, "path": "internal/config/config.go", "start_line": 1}
{"end_line": 200, "path": "internal/config/config.go", "start_line": 1}
{"path":"internal/config/config.go","content":"// Package config loads and validates zoo's HCL configuration file.\npackage config\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/hashicorp/hcl/v2/hclsimple\"\n)\n\n// Known event kinds. issue:assigned and pr:review are resolved\n// dynamically (agent name must match the Forgejo assignee's username,\n// or the pull request author's username, respectively) so they never\n// carry an `agent` attribute; the rest map statically to one\n// configured agent.\nconst (\n\tEventIssueNew      = \"issue:new\"\n\tEventIssueComment  = \"issue:comment\"\n\tEventIssueAssigned = \"issue:assigned\"\n\tEventPRNew         = \"pr:new\"\n\tEventPRReview      = \"pr:review\"\n)\n\nvar staticEventKinds = map[string]bool{\n\tEventIssueNew:     true,\n\tEventIssueComment: true,\n\tEventPRNew:        true,\n}\n\ntype Config struct {\n\tLLMs        []LLM       `hcl:\"llm,block\"`\n\tForgejo     Forgejo     `hcl:\"forgejo,block\"`\n\tEnvironment Environment `hcl:\"environment,block\"`\n\tAgents      []Agent     `hcl:\"agent,block\"`\n\tEvents      []Event     `hcl:\"event,block\"`\n\tWeb         *Web        `hcl:\"web,block\"`\n}\n\n// Web configures the dashboard's optional bearer-token gate. Leave the\n// block out of zoo.hcl entirely to run without one (fine on localhost;\n// put a real gate or a proxy in front for anything else).\ntype Web struct {\n\tToken string `hcl:\"token,optional\"`\n}\n\ntype LLM struct {\n\tName   string `hcl:\"name,label\"`\n\tOpenAI string `hcl:\"openai\"`\n\tToken  string `hcl:\"token\"`\n\tModel  string `hcl:\"model\"`\n}\n\ntype Forgejo struct {\n\tURL           string `hcl:\"url\"`\n\tToken         string `hcl:\"token\"`\n\tWebhookSecret string `hcl:\"webhook_secret,optional\"`\n\n\t// Repos is the allowlist of repository patterns to watch, e.g.\n\t// [\"acme/*\", \"acme/widgets\"]. Patterns are \"owner/repo\" pairs with\n\t// glob wildcards; \"*\" watches everything on the instance. An empty\n\t// list keeps the historical behavior of watching every repository\n\t// the token can see.\n\tRepos []string `hcl:\"repos,optional\"`\n}\n\ntype Environment struct {\n\tDockerImage string `hcl:\"docker_image\"`\n\tMaxLive     int    `hcl:\"max_live_agents\"`\n}\n\ntype Agent struct {\n\tName  string `hcl:\"name,label\"`\n\tLLM   string `hcl:\"llm\"`\n\tToken string `hcl:\"token,optional\"`\n}\n\ntype Event struct {\n\tKind         string `hcl:\"name,label\"`\n\tAgent        string `hcl:\"agent,optional\"`\n\tInstructions string `hcl:\"instructions,optional\"`\n}\n\n// Load reads and validates the config file at path.\nfunc Load(path string) (*Config, error) {\n\tvar cfg Config\n\n\tif err := hclsimple.DecodeFile(path, nil, \u0026cfg); err != nil {\n\t\treturn nil, fmt.Errorf(\"parse config: %w\", err)\n\t}\n\n\tif err := cfg.Validate(); err != nil {\n\t\treturn nil, fmt.Errorf(\"invalid config: %w\", err)\n\t}\n\n\treturn \u0026cfg, nil\n}\n\n// Validate checks that the config is internally consistent: every\n// reference between blocks resolves, and required values are set.\nfunc (c *Config) Validate() error {\n\tllmNames := make(map[string]bool, len(c.LLMs))\n\tfor _, l := range c.LLMs {\n\t\tif l.OpenAI == \"\" || l.Token == \"\" || l.Model == \"\" {\n\t\t\treturn fmt.Errorf(\"llm %q: openai, token, and model are required\", l.Name)\n\t\t}\n\t\tllmNames[l.Name] = true\n\t}\n\n\tif c.Forgejo.URL == \"\" || c.Forgejo.Token == \"\" {\n\t\treturn fmt.Errorf(\"forgejo: url and token are required\")\n\t}\n\n\tfor _, p := range c.Forgejo.Repos {\n\t\tif err := validRepoPattern(p); err != nil {\n\t\t\treturn fmt.Errorf(\"forgejo: %w\", err)\n\t\t}\n\t}\n\n\tif c.Environment.MaxLive \u003c 1 {\n\t\treturn fmt.Errorf(\"environment: max_live_agents must be \u003e= 1, got %d\", c.Environment.MaxLive)\n\t}\n\n\tif c.Environment.DockerImage == \"\" {\n\t\treturn fmt.Errorf(\"environment: docker_image is required\")\n\t}\n\n\tagentNames := make(map[string]bool, len(c.Agents))\n\tfor _, a := range c.Agents {\n\t\tif !llmNames[a.LLM] {\n\t\t\treturn fmt.Errorf(\"agent %q: references undeclared llm %q\", a.Name, a.LLM)\n\t\t}\n\t\tagentNames[a.Name] = true\n\t}\n\n\tseenEventKinds := make(map[string]bool, len(c.Events))\n\tfor _, e := range c.Events {\n\t\tif seenEventKinds[e.Kind] {\n\t\t\treturn fmt.Errorf(\"event %q: declared more than once\", e.Kind)\n\t\t}\n\t\tseenEventKinds[e.Kind] = true\n\n\t\tswitch {\n\t\tcase e.Kind == EventIssueAssigned:\n\t\t\tif e.Agent != \"\" {\n\t\t\t\treturn fmt.Errorf(\"event %q: must not set agent; the agent whose name matches the Forgejo assignee's username is triggered dynamically\", e.Kind)\n\t\t\t}\n\n\t\tcase e.Kind == EventPRReview:\n\t\t\tif e.Agent != \"\" {\n\t\t\t\treturn fmt.Errorf(\"event %q: must not set agent; the agent whose name matches the pull request author's username is triggered dynamically\", e.Kind)\n\t\t\t}\n\n\t\tcase staticEventKinds[e.Kind]:\n\t\t\tif e.Agent == \"\" {\n\t\t\t\treturn fmt.Errorf(\"event %q: agent is required\", e.Kind)\n\t\t\t}\n\t\t\tif !agentNames[e.Agent] {\n\t\t\t\treturn fmt.Errorf(\"event %q: references undeclared agent %q\", e.Kind, e.Agent)\n\t\t\t}\n\n\t\tdefault:\n\t\t\treturn fmt.Errorf(\"event %q: unknown event kind\", e.Kind)\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// validRepoPattern checks the shape of a forgejo `repos` entry. It must\n// be a non-empty \"owner/repo\" pair (or a bare owner name, or \"*\") β€” at\n// most one slash, none at either end. Glob syntax within the segments\n// is validated later by forgejo.NewRepoFilter.\nfunc validRepoPattern(p string) error {\n\tif p == \"\" {\n\t\treturn fmt.Errorf(\"repos: pattern must not be empty\")\n\t}\n\n\tif strings.Count(p, \"/\") \u003e 1 {\n\t\treturn fmt.Errorf(\"repos: pattern %q must be \\\"owner/repo\\\" (at most one \\\"/\\\")\", p)\n\t}\n\n\tif strings.HasPrefix(p, \"/\") || strings.HasSuffix(p, \"/\") {\n\t\treturn fmt.Errorf(\"repos: pattern %q must not start or end with \\\"/\\\"\", p)\n\t}\n\n\treturn nil\n}\n\n// AgentConfig holds an agent's full configuration including its\n// Forgejo token (if configured).\ntype AgentConfig struct {\n\tName  string\n\tLLM   string\n\tToken string\n}\n\n// AgentByName returns the agent config with the given name, if declared.\nfunc (c *Config) AgentByName(name string) (AgentConfig, bool) {\n\tfor _, a := range c.Agents {\n\t\tif a.Name == name {","start_line":1,"end_line":200,"total_lines":246,"truncated":true}
Tool πŸ”§ read_file {"end_line": 267, "path": "internal/llm/types.go", "start_line": 200}
{"end_line": 267, "path": "internal/llm/types.go", "start_line": 200}
{"path":"internal/llm/types.go","content":"\tif next.FinishReason != \"\" {\n\t\tc.FinishReason = next.FinishReason\n\t}\n\n\tif next.Role != \"\" {\n\t\tc.Role = next.Role\n\t}\n}\n\n// Timings mirrors llama.cpp's /chat/completions timing block. The\n// streamer tolerates unknown fields, so extra fields from the server\n// are simply ignored.\ntype Timings struct {\n\tCacheN              int     `json:\"cache_n\"`\n\tDraftN              int     `json:\"draft_n\"`\n\tDraftNAccepted      int     `json:\"draft_n_accepted\"`\n\tPromptN             int     `json:\"prompt_n\"`\n\tPromptMs            float64 `json:\"prompt_ms\"`\n\tPromptPerTokenMs    float64 `json:\"prompt_per_token_ms\"`\n\tPromptPerSecond     float64 `json:\"prompt_per_second\"`\n\tPredictedN          int     `json:\"predicted_n\"`\n\tPredictedMs         float64 `json:\"predicted_ms\"`\n\tPredictedPerTokenMs float64 `json:\"predicted_per_token_ms\"`\n\tPredictedPerSecond  float64 `json:\"predicted_per_second\"`\n}\n\ntype ChatCompletion struct {\n\tChoices           []Choice `json:\"choices\"`\n\tCreated           int64    `json:\"created\"`\n\tId                string   `json:\"id\"`\n\tModel             string   `json:\"model\"`\n\tSystemFingerprint string   `json:\"system_fingerprint\"`\n\tObject            string   `json:\"object\"`\n\tTimings           Timings  `json:\"timings\"`\n}\n\nfunc (c *ChatCompletion) String() string {\n\ts := fmt.Sprintf(\"Id: %s, Model: %s, Created: %d, SystemFingerprint: %s, Object: %s\\n\",\n\t\tc.Id, c.Model, c.Created, c.SystemFingerprint, c.Object)\n\n\tfor _, choice := range c.Choices {\n\t\ts += fmt.Sprintf(\"  Choice: %s\\n\", choice.String())\n\t}\n\n\treturn s\n}\n\n// Merge folds a streamed chunk into the accumulated completion.\nfunc (c *ChatCompletion) Merge(next *ChatCompletion) {\n\tminLength := 0\n\tfor _, choice := range next.Choices {\n\t\tif choice.Index+1 \u003e minLength {\n\t\t\tminLength = choice.Index + 1\n\t\t}\n\t}\n\n\tif len(c.Choices) \u003c minLength {\n\t\tnewChoices := make([]Choice, minLength)\n\t\tcopy(newChoices, c.Choices)\n\t\tc.Choices = newChoices\n\t}\n\n\tfor _, nextChoice := range next.Choices {\n\t\tc.Choices[nextChoice.Index].Merge(\u0026nextChoice)\n\t}\n\n\tc.Timings = next.Timings\n}","start_line":200,"end_line":267,"total_lines":267,"truncated":false}
Thinking
Now I understand the structure. Let me think about the design. The issue: "We need some kind of retry logic in the agent loop. In case the LLM endpoint is down." The agent loop is `runLoop` in `internal/agentrun/loop.go`. It calls `client.StreamChatCompletion(ctx, req)` to get a streamer, then repeatedly calls `streamer.Get()` until EOF. Where can failures happen? 1. `client.StreamChatCompletion` β€” HTTP request fails (connection refused, DNS, timeout), or non-200 status (5xx, 429, etc.). 2. `streamer.Get()` β€” stream body read fails mid-stream (connection reset, timeout, malformed chunk). Retry strategy design: - Retry transient failures: connection errors, 5xx, 429 (rate limit), mid-stream interruptions. - Don't retry non-transient errors: 400/401/403/404 etc. (bad request, auth failure, model not found). - Exponential backoff with jitter, bounded by max attempts and/or max total time. - Important: the agent loop is long-running (up to 120 minutes), so retries should be bounded per turn (per stream round-trip) rather than per run. - Also important: when retrying a turn, the partial content that was streamed before the failure has already been published to the hub via hooks (OnReasoningDelta/OnContentDelta). Retrying the whole turn means the model re-generates that turn β€” the partial delta is already in the live log, but the turn is only persisted on OnTurnEnd. Hmm, this is a subtle UX issue: if a turn fails mid-stream and we retry, the viewer will see the partial text, then see the new text. But the persisted log only gets the final version (OnTurnEnd fires after the full completion). Actually wait β€” OnTurnEnd fires after the full completion. If the stream fails mid-way, OnTurnEnd never fires for that partial content, so the partial content is not persisted. But it is published live to the hub. On retry, the new turn's deltas get published. The viewer will see duplicated content (partial + full). How to handle this? Options: 1. Retry only on failures that occur before any content has been streamed (i.e., request-level failures and failures before the first chunk). This is the cleanest: if we've already streamed some content, retrying means re-running the whole turn and the viewer sees the partial content again. 2. Retry the whole turn regardless, and accept the duplication in the live view. 3. Add a hook to "reset" the turn's live view on retry. Hmm. Actually, let me think about what's cleanest architecturally. The issue says "in the agent loop, in case the LLM endpoint is down." The most common failure mode: endpoint is down (connection refused) β†’ `StreamChatCompletion` fails immediately, nothing has been streamed. Retry with backoff is clearly the right answer. For mid-stream failures (endpoint dies during generation), the partial content has been published. Retrying the whole turn is still correct from a correctness standpoint (the message is only appended to `messages` after the full completion, and tool calls only run after that), but the live view shows the partial content. I think a good design is: - Put the retry logic in `runLoop` (the agent loop) as the issue requests. Wrap the "one turn" (request + full stream read) in a retry loop. - Retry on transient errors: network errors, 5xx, 429, and mid-stream interruptions. - Don't retry on: ctx cancellation, 4xx (except 429), or after content has already been streamed? Hmm. Let me think about whether to retry after partial content. If we've already streamed partial content to the viewer and retry, the viewer will see the partial content, then the full new content. That's a bit confusing but not wrong β€” it's like the model "started over." Alternatively, we could add a `OnTurnReset` hook to clear the live buffer. That's more invasive. Actually, let me reconsider. The simplest, most robust approach that matches "in case the LLM endpoint is down": Retry at the level of a single LLM call (request + stream read) inside `runLoop`. Only retry if no content has been streamed yet for that turn (i.e., the failure happened before the first non-empty delta, or the failure is at the request level). If content has been streamed and then the stream breaks, retrying would duplicate content in the live view β€” but the persisted log is fine. Hmm, but actually, is that really so bad? Let me think about what happens in the live view: - Turn N starts: OnReasoningDelta fires multiple times (partial reasoning), OnContentDelta fires multiple times (partial content). - Stream breaks. - Retry: new stream. OnReasoningDelta fires again (new reasoning). The hub just appends deltas. The viewer sees: partial reasoning, partial content, then new reasoning, new content. The persisted log only gets the final version (OnTurnEnd). So the dashboard history is clean; only the live view during the failure window shows the duplication. I think this is acceptable, and arguably worth it: mid-stream failure with a long generation is exactly when you want a retry. But the duplication in the live view is a real UX wart. Alternative: only retry if nothing has been streamed yet. That's simpler and avoids the wart. But then a mid-stream break (very common with local llama.cpp servers under load, or network hiccups) kills the entire 120-minute run. That seems worse. Hmm, let me think about what a senior architect would do. I think the right call is: 1. Retry the whole turn on transient failure, regardless of how much was streamed. 2. Add an `OnTurnReset` hook (or reuse OnTurnEnd semantics) so the live view can be reset. Actually, the hub publishes deltas; there's no "reset" event. Let me look at the livelog hub to see what events exist. Let me check the livelog hub to understand the event types. Also, let me check how `runLoop` errors surface to the job status β€” the scheduler probably marks the job as failed, and the issue has a "zoo:failed" label (interesting β€” the issue itself is labeled zoo:failed, meaning a previous run of this issue failed! That's probably why the issue exists β€” the agent failed to handle it, probably because the LLM endpoint was down. A nice meta detail.) Let me look at the livelog hub and the scheduler to understand the failure path.
Tool πŸ”§ read_file {"end_line": 150, "path": "internal/livelog/hub.go", "start_line": 1}
{"end_line": 150, "path": "internal/livelog/hub.go", "start_line": 1}
{"path":"internal/livelog/hub.go","content":"// Package livelog is an in-memory pub/sub hub that lets agentrun publish\n// fine-grained, not-yet-persisted events (reasoning/content deltas, tool\n// calls) for a running job, and lets the web dashboard subscribe to them\n// live over SSE. It carries only the tail of a run that hasn't made it\n// into SQLite yet β€” see Hub.Checkpoint.\npackage livelog\n\nimport \"sync\"\n\ntype Type string\n\nconst (\n\tReasoningStart Type = \"reasoning_start\"\n\tReasoningDelta Type = \"reasoning_delta\"\n\tReasoningEnd   Type = \"reasoning_end\"\n\tContentStart   Type = \"content_start\"\n\tContentDelta   Type = \"content_delta\"\n\tContentEnd     Type = \"content_end\"\n\tTool           Type = \"tool\"\n\tStatus         Type = \"status\"\n)\n\ntype Event struct {\n\tType      Type   `json:\"type\"`\n\tText      string `json:\"text,omitempty\"`\n\tName      string `json:\"name,omitempty\"`\n\tArguments string `json:\"arguments,omitempty\"`\n\tResult    string `json:\"result,omitempty\"`\n\tError     bool   `json:\"error,omitempty\"`\n\tStatus    string `json:\"status,omitempty\"`\n}\n\n// subChanBuffer bounds how many events a slow subscriber can lag behind\n// by before new events are dropped for it. Generous for a single-user\n// local dashboard; a dropped event just means a subscriber's browser\n// tab misses a chunk and catches up on the next one, never a hang.\nconst subChanBuffer = 256\n\n// maxBufferedEvents caps the per-job replay buffer as a safety net\n// against unbounded growth if a caller forgets to Checkpoint. Ordinary\n// operation clears the buffer far below this on every persisted block.\nconst maxBufferedEvents = 10000\n\ntype stream struct {\n\tmu   sync.Mutex\n\tbuf  []Event\n\tsubs map[chan Event]struct{}\n}\n\n// Hub fans out per-job Events to any number of live subscribers.\ntype Hub struct {\n\tmu   sync.Mutex\n\tjobs map[string]*stream\n}\n\nfunc NewHub() *Hub {\n\treturn \u0026Hub{jobs: make(map[string]*stream)}\n}\n\nfunc (h *Hub) get(jobID string) *stream {\n\th.mu.Lock()\n\tdefer h.mu.Unlock()\n\n\ts, ok := h.jobs[jobID]\n\tif !ok {\n\t\ts = \u0026stream{subs: make(map[chan Event]struct{})}\n\t\th.jobs[jobID] = s\n\t}\n\n\treturn s\n}\n\n// Publish appends ev to jobID's replay buffer and fans it out to every\n// current subscriber. Never blocks: a subscriber whose channel is full\n// simply misses this event.\nfunc (h *Hub) Publish(jobID string, ev Event) {\n\ts := h.get(jobID)\n\n\ts.mu.Lock()\n\tdefer s.mu.Unlock()\n\n\ts.buf = append(s.buf, ev)\n\tif len(s.buf) \u003e maxBufferedEvents {\n\t\ts.buf = s.buf[len(s.buf)-maxBufferedEvents:]\n\t}\n\n\tfor ch := range s.subs {\n\t\tselect {\n\t\tcase ch \u003c- ev:\n\t\tdefault:\n\t\t}\n\t}\n}\n\n// Checkpoint clears jobID's replay buffer. Call this right after a\n// block derived from buffered events has been durably written to\n// storage, so a subscriber that connects afterward β€” and therefore\n// already sees that block via the persisted history β€” is never replayed\n// it a second time over the live channel.\nfunc (h *Hub) Checkpoint(jobID string) {\n\ts := h.get(jobID)\n\n\ts.mu.Lock()\n\tdefer s.mu.Unlock()\n\n\ts.buf = nil\n}\n\n// Subscribe returns the events currently buffered for jobID (i.e. not\n// yet checkpointed) plus a channel of everything published from this\n// point on. The returned cancel func must be called when the caller is\n// done reading, to unregister and close the channel.\nfunc (h *Hub) Subscribe(jobID string) (buffered []Event, ch \u003c-chan Event, cancel func()) {\n\ts := h.get(jobID)\n\n\ts.mu.Lock()\n\tdefer s.mu.Unlock()\n\n\tbuffered = append([]Event(nil), s.buf...)\n\n\tc := make(chan Event, subChanBuffer)\n\ts.subs[c] = struct{}{}\n\n\tvar once sync.Once\n\n\tcancelFn := func() {\n\t\tonce.Do(func() {\n\t\t\ts.mu.Lock()\n\t\t\tdefer s.mu.Unlock()\n\n\t\t\tif _, ok := s.subs[c]; ok {\n\t\t\t\tdelete(s.subs, c)\n\t\t\t\tclose(c)\n\t\t\t}\n\t\t})\n\t}\n\n\treturn buffered, c, cancelFn\n}\n\n// Close publishes a terminal Status event, disconnects every current\n// subscriber, and drops jobID's state. Call this exactly once, when the\n// job's run has finished.\nfunc (h *Hub) Close(jobID string) {\n\th.Publish(jobID, Event{Type: Status, Status: \"done\"})\n\n\th.mu.Lock()\n\ts, ok := h.jobs[jobID]\n\tdelete(h.jobs, jobID)\n\th.mu.Unlock()","start_line":1,"end_line":150,"total_lines":164,"truncated":true}
Tool πŸ”§ read_file {"end_line": 120, "path": "internal/scheduler/scheduler.go", "start_line": 1}
{"end_line": 120, "path": "internal/scheduler/scheduler.go", "start_line": 1}
{"path":"internal/scheduler/scheduler.go","content":"// Package scheduler resolves incoming Forgejo events to configured\n// agents and runs them, bounded by max_live_agents.\npackage scheduler\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\t\"log/slog\"\n\t\"sync\"\n\n\t\"github.com/google/uuid\"\n\n\t\"github.com/abrander/zoo/internal/config\"\n\t\"github.com/abrander/zoo/internal/forgejo\"\n\t\"github.com/abrander/zoo/internal/livelog\"\n\t\"github.com/abrander/zoo/internal/store\"\n)\n\n// forgejoActions is the narrow slice of Client the scheduler needs for\n// its own failure-reporting side effects (defined here, not in\n// internal/forgejo, so tests can inject a fake).\ntype forgejoActions interface {\n\tCreateIssueComment(owner, repo string, index int64, body string) error\n\tAddLabel(owner, repo string, index int64, name string) error\n}\n\n// FailureLabel is applied to the triggering issue/PR, alongside a\n// comment, whenever an agent run fails or times out.\nconst FailureLabel = \"zoo:failed\"\n\n// Runner runs a single agent invocation to completion. Implemented by\n// internal/agentrun.Run; a narrow interface here so the scheduler is\n// testable without Docker.\ntype Runner interface {\n\tRun(ctx context.Context, jobID string, agent config.AgentConfig, llm config.LLM, dockerImage string, ev forgejo.Event) error\n}\n\ntype Scheduler struct {\n\tcfg     *config.Config\n\tstore   *store.Store\n\tforgejo forgejoActions\n\trunner  Runner\n\thub     *livelog.Hub\n\tlogger  *slog.Logger\n\n\tsem chan struct{}\n\twg  sync.WaitGroup\n}\n\nfunc New(cfg *config.Config, st *store.Store, fg forgejoActions, runner Runner, hub *livelog.Hub, logger *slog.Logger) *Scheduler {\n\treturn \u0026Scheduler{\n\t\tcfg:     cfg,\n\t\tstore:   st,\n\t\tforgejo: fg,\n\t\trunner:  runner,\n\t\thub:     hub,\n\t\tlogger:  logger,\n\t\tsem:     make(chan struct{}, cfg.Environment.MaxLive),\n\t}\n}\n\n// resolveAgent returns the name of the agent that should handle ev, if\n// any. Two kinds resolve dynamically: issue:assigned to the agent whose\n// config label matches the Forgejo assignee's username, and pr:review to\n// the agent whose config label matches the pull request author's\n// username (the agent that opened the PR reacts to the review of it).\n// Every other event kind uses the static event-\u003eagent mapping from\n// config.\nfunc resolveAgent(cfg *config.Config, ev forgejo.Event) (string, bool) {\n\tswitch ev.Kind {\n\tcase config.EventIssueAssigned:\n\t\tif _, ok := cfg.AgentByName(ev.Assignee); ok {\n\t\t\treturn ev.Assignee, true\n\t\t}\n\n\t\treturn \"\", false\n\n\tcase config.EventPRReview:\n\t\tif _, ok := cfg.AgentByName(ev.PRAuthor); ok {\n\t\t\treturn ev.PRAuthor, true\n\t\t}\n\n\t\treturn \"\", false\n\t}\n\n\treturn cfg.EventAgent(ev.Kind)\n}\n\n// Run consumes events until ctx is canceled or the channel closes,\n// dispatching each to its resolved agent and blocking on the\n// max_live_agents semaphore before starting a run.\nfunc (s *Scheduler) Run(ctx context.Context, events \u003c-chan forgejo.Event) {\n\tfor {\n\t\tselect {\n\t\tcase \u003c-ctx.Done():\n\t\t\treturn\n\n\t\tcase ev, ok := \u003c-events:\n\t\t\tif !ok {\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\ts.handle(ctx, ev)\n\t\t}\n\t}\n}\n\nfunc (s *Scheduler) handle(ctx context.Context, ev forgejo.Event) {\n\tagentName, ok := resolveAgent(s.cfg, ev)\n\tif !ok {\n\t\ts.logger.Debug(\"no agent resolved for event, dropping\", \"kind\", ev.Kind, \"owner\", ev.Owner, \"repo\", ev.Repo, \"index\", ev.Index)\n\t\treturn\n\t}\n\n\t// An agent's own actions (e.g. a comment posted via the `comment`\n\t// tool, authenticated with its own per-agent token) can themselves\n\t// show up as new events. Don't let an agent trigger itself off its\n\t// own activity β€” that's a self-reinforcing loop, not new work.\n\tif ev.Author != \"\" \u0026\u0026 ev.Author == agentName {","start_line":1,"end_line":120,"total_lines":252,"truncated":true}
Thinking
Now I understand the whole picture. Let me also check the existing tests for the llm package and loop to get a grasp of the testing conventions.
Tool πŸ”§ read_file {"end_line": 120, "path": "internal/llm/streamer_test.go", "start_line": 1}
{"end_line": 120, "path": "internal/llm/streamer_test.go", "start_line": 1}
{"path":"internal/llm/streamer_test.go","content":"package llm\n\nimport (\n\t\"encoding/json\"\n\t\"io\"\n\t\"net/http\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc sseChunk(t *testing.T, delta Message, finish string) string {\n\tt.Helper()\n\n\tbody, err := json.Marshal(ChatCompletion{\n\t\tChoices: []Choice{{Delta: delta, FinishReason: finish}},\n\t})\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\n\treturn \"data: \" + string(body) + \"\\n\\n\"\n}\n\nfunc newTestResponse(body string) *http.Response {\n\treturn \u0026http.Response{\n\t\tStatusCode: http.StatusOK,\n\t\tBody:       io.NopCloser(strings.NewReader(body)),\n\t}\n}\n\nfunc TestStreamerMergesChunksAndStops(t *testing.T) {\n\tsse := sseChunk(t, Message{Content: \"Hel\", Role: \"assistant\"}, \"\") +\n\t\tsseChunk(t, Message{Content: \"lo\"}, \"stop\") +\n\t\t\"data: [DONE]\\n\\n\"\n\n\ts, err := NewStreamer(newTestResponse(sse))\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\n\tvar last *ChatCompletion\n\tfor {\n\t\tc, err := s.Get()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\tt.Fatal(err)\n\t\t}\n\t\tlast = c\n\t}\n\n\tif last == nil {\n\t\tt.Fatal(\"expected at least one completion\")\n\t}\n\tif len(last.Choices) != 1 {\n\t\tt.Fatalf(\"expected 1 choice, got %d\", len(last.Choices))\n\t}\n\tif last.Choices[0].Message.Content != \"Hello\" {\n\t\tt.Fatalf(\"expected merged content %q, got %q\", \"Hello\", last.Choices[0].Message.Content)\n\t}\n\tif last.Choices[0].FinishReason != \"stop\" {\n\t\tt.Fatalf(\"expected finish_reason stop, got %q\", last.Choices[0].FinishReason)\n\t}\n\tif last.Choices[0].Message.Role != \"assistant\" {\n\t\tt.Fatalf(\"expected role to stick from first chunk, got %q\", last.Choices[0].Message.Role)\n\t}\n}\n\nfunc TestStreamerNonOKStatus(t *testing.T) {\n\tresp := newTestResponse(\"boom\")\n\tresp.StatusCode = http.StatusInternalServerError\n\n\t_, err := NewStreamer(resp)\n\tif err == nil {\n\t\tt.Fatal(\"expected error for non-200 response\")\n\t}\n}\n\nfunc TestStreamerToleratesUnknownFields(t *testing.T) {\n\t// Servers may add fields we don't know about; decoding must not fail.\n\tchunk := `data: {\"id\":\"x\",\"model\":\"m\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"hi\",\"new_field\":\"boom\"},\"finish_reason\":null,\"extra\":\"y\"}],\"timings\":{\"prompt_n\":1,\"brand_new_timing\":42},\"surprise\":true}\n\n`\n\n\ts, err := NewStreamer(newTestResponse(chunk))\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\n\tc, err := s.Get()\n\tif err != nil {\n\t\tt.Fatalf(\"expected unknown fields to be tolerated, got error: %v\", err)\n\t}\n\tif c.Choices[0].Message.Content != \"hi\" {\n\t\tt.Fatalf(\"expected content %q, got %q\", \"hi\", c.Choices[0].Message.Content)\n\t}\n}\n\nfunc TestToolCallMerge(t *testing.T) {\n\tsse := sseChunk(t, Message{ToolCalls: []ToolCall{{Index: 0, Id: \"call1\", FunctionCall: FunctionCall{Name: \"bash\", Arguments: `{\"command\":`}}}}, \"\") +\n\t\tsseChunk(t, Message{ToolCalls: []ToolCall{{Index: 0, FunctionCall: FunctionCall{Arguments: `\"ls\"}`}}}}, \"tool_calls\") +\n\t\t\"data: [DONE]\\n\\n\"\n\n\ts, err := NewStreamer(newTestResponse(sse))\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\n\tvar last *ChatCompletion\n\tfor {\n\t\tc, err := s.Get()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tif err != nil {\n\t\t\tt.Fatal(err)\n\t\t}\n\t\tlast = c\n\t}","start_line":1,"end_line":120,"total_lines":132,"truncated":true}
Tool πŸ”§ grep_search {"context_after": 3, "query": "func Test"}
{"context_after": 3, "query": "func Test"}
/project/internal/config/config_test.go:64:func TestLoadValid(t *testing.T) {
/project/internal/config/config_test.go-65-	cfg, err := load(t, validConfig)
/project/internal/config/config_test.go-66-	if err != nil {
/project/internal/config/config_test.go-67-		t.Fatalf("unexpected error: %v", err)
--
/project/internal/config/config_test.go:100:func TestValidatePRReviewRejectsAgent(t *testing.T) {
/project/internal/config/config_test.go-101-	_, err := load(t, `
/project/internal/config/config_test.go-102-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-103-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/config/config_test.go:113:func TestValidateUndeclaredLLM(t *testing.T) {
/project/internal/config/config_test.go-114-	_, err := load(t, `
/project/internal/config/config_test.go-115-forgejo { url = "https://example.com" token = "tok" }
/project/internal/config/config_test.go-116-environment { docker_image = "debian:unstable" max_live_agents = 1 }
--
/project/internal/config/config_test.go:124:func TestValidateUndeclaredEventAgent(t *testing.T) {
/project/internal/config/config_test.go-125-	_, err := load(t, `
/project/internal/config/config_test.go-126-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-127-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/config/config_test.go:137:func TestValidateIssueAssignedRejectsAgent(t *testing.T) {
/project/internal/config/config_test.go-138-	_, err := load(t, `
/project/internal/config/config_test.go-139-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-140-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/config/config_test.go:150:func TestValidateMaxLiveAgents(t *testing.T) {
/project/internal/config/config_test.go-151-	_, err := load(t, `
/project/internal/config/config_test.go-152-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-153-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/config/config_test.go:161:func TestValidateMissingDockerImage(t *testing.T) {
/project/internal/config/config_test.go-162-	_, err := load(t, `
/project/internal/config/config_test.go-163-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-164-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/config/config_test.go:199:func TestLoadRepos(t *testing.T) {
/project/internal/config/config_test.go-200-	cfg, err := load(t, fmt.Sprintf(reposConfigTemplate, `["acme/*", "bob/widgets"]`))
/project/internal/config/config_test.go-201-	if err != nil {
/project/internal/config/config_test.go-202-		t.Fatalf("unexpected error: %v", err)
--
/project/internal/config/config_test.go:210:func TestValidateReposEmptyList(t *testing.T) {
/project/internal/config/config_test.go-211-	cfg, err := load(t, fmt.Sprintf(reposConfigTemplate, `[]`))
/project/internal/config/config_test.go-212-	if err != nil {
/project/internal/config/config_test.go-213-		t.Fatalf("unexpected error: %v", err)
--
/project/internal/config/config_test.go:221:func TestValidateReposRejectsMalformedPatterns(t *testing.T) {
/project/internal/config/config_test.go-222-	for _, bad := range []string{`""`, `"acme/widgets/extra"`, `"/acme"`, `"acme/"`, `"acme//widgets"`} {
/project/internal/config/config_test.go-223-		_, err := load(t, fmt.Sprintf(reposConfigTemplate, `[`+bad+`]`))
/project/internal/config/config_test.go-224-		if err == nil {
--
/project/internal/config/config_test.go:230:func TestValidateUnknownEventKind(t *testing.T) {
/project/internal/config/config_test.go-231-	_, err := load(t, `
/project/internal/config/config_test.go-232-llm "qwen" { openai = "https://example.com" token = "tok" model = "qwen3" }
/project/internal/config/config_test.go-233-forgejo { url = "https://example.com" token = "tok" }
--
/project/internal/web/web_test.go:19:func testServer(t *testing.T) (*Server, *store.Store) {
/project/internal/web/web_test.go-20-	t.Helper()
/project/internal/web/web_test.go-21-
/project/internal/web/web_test.go-22-	st, err := store.Open(filepath.Join(t.TempDir(), "zoo.db"))
--
/project/internal/web/web_test.go:39:func TestIndexRenders(t *testing.T) {
/project/internal/web/web_test.go-40-	s, _ := testServer(t)
/project/internal/web/web_test.go-41-
/project/internal/web/web_test.go-42-	rr := httptest.NewRecorder()
--
/project/internal/web/web_test.go:53:func TestJobsAndDetail(t *testing.T) {
/project/internal/web/web_test.go-54-	s, st := testServer(t)
/project/internal/web/web_test.go-55-	ctx := context.Background()
/project/internal/web/web_test.go-56-
--
/project/internal/web/web_test.go:112:func TestJobDetailTrimsBlockWhitespace(t *testing.T) {
/project/internal/web/web_test.go-113-	s, st := testServer(t)
/project/internal/web/web_test.go-114-	ctx := context.Background()
/project/internal/web/web_test.go-115-
--
/project/internal/web/web_test.go:154:func TestAuthGate(t *testing.T) {
/project/internal/web/web_test.go-155-	st, err := store.Open(filepath.Join(t.TempDir(), "zoo.db"))
/project/internal/web/web_test.go-156-	if err != nil {
/project/internal/web/web_test.go-157-		t.Fatal(err)
--
/project/internal/web/web_test.go:182:func TestJobsShowAgentAvatars(t *testing.T) {
/project/internal/web/web_test.go-183-	// Minimal in-test Forgejo API: the version probe the SDK makes when
/project/internal/web/web_test.go-184-	// the client is constructed, plus per-user profile lookups.
/project/internal/web/web_test.go-185-	avatars := map[string]string{
--
/project/internal/livelog/hub_test.go:25:func TestSubscribeReplaysBufferedEvents(t *testing.T) {
/project/internal/livelog/hub_test.go-26-	h := NewHub()
/project/internal/livelog/hub_test.go-27-
/project/internal/livelog/hub_test.go-28-	h.Publish("job-1", Event{Type: ContentStart})
--
/project/internal/livelog/hub_test.go:49:func TestCheckpointClearsBuffer(t *testing.T) {
/project/internal/livelog/hub_test.go-50-	h := NewHub()
/project/internal/livelog/hub_test.go-51-
/project/internal/livelog/hub_test.go-52-	h.Publish("job-1", Event{Type: ContentStart})
--
/project/internal/livelog/hub_test.go:64:func TestCloseTerminatesSubscribers(t *testing.T) {
/project/internal/livelog/hub_test.go-65-	h := NewHub()
/project/internal/livelog/hub_test.go-66-
/project/internal/livelog/hub_test.go-67-	_, ch, cancel := h.Subscribe("job-1")
--
Thinking
Alright, I've got a solid understanding of the codebase now. Let's design the retry logic. ## Design **Where to put the retry:** The issue says "inside the agent loop." The agent loop is `runLoop` in `internal/agentrun/loop.go`. The unit of work that can fail is a single "turn": `client.StreamChatCompletion` (HTTP request) + reading the stream until EOF. **Retry policy:** - Retry transient failures: - Network/transport errors from `httpClient.Do` (connection refused, reset, DNS, timeout) - HTTP 5xx (server error) and 429 (rate limited) - Mid-stream interruptions (stream body read error after the request succeeded) - Do NOT retry: - Context cancellation/deadline (the run is being stopped β€” don't fight it) - Other 4xx (400 bad request, 401/403 auth, 404 model not found) β€” retrying won't fix these - Malformed response (decode error) β€” hmm, this is tricky. A decode error might be a transient glitch (truncated chunk) or it might be a server bug. I'd say decode errors are non-retryable, because if the server is consistently sending malformed chunks, retrying just wastes time. Actually... a truncated final chunk is a classic transient failure. But "decode completion chunk" errors come from malformed JSON. Let me think β€” if the stream is cut off mid-JSON, the scanner will return the partial line... actually, if the connection drops mid-chunk, `scanner.Scan()` returns false and `scanner.Err()` returns the read error, so we get a read error, not a decode error. A decode error means we got a complete line that wasn't valid JSON β€” that's a server-side protocol issue, unlikely to be fixed by retrying. Treat as non-retryable. Hmm, but wait β€” actually, let me reconsider. Some servers send non-JSON "data:" lines (e.g., keep-alive comments or error messages). Retrying on those could loop forever. Non-retryable is the safer choice. - "model returned an empty completion" β€” the stream completed cleanly but with no choices. This is a server-side weirdness. Retrying might help (could be a transient empty response). Hmm. I want to say retryable β€” an empty completion from a healthy server is weird, and a retry is cheap. Actually, let me be careful: if the model genuinely returns empty (e.g., content filter), the retry will loop until we exhaust attempts. With a bounded attempt count, that's fine β€” we'll eventually fail with a clear error. Let's make it retryable. **Backoff:** Exponential backoff with full jitter, capped. E.g., base 1s, factor 2, cap 30s, max 5 attempts (i.e., 1 initial + 4 retries)? For a 120-minute agent run, we can afford a bit more. Let's think about total worst-case wait: 1+2+4+8+16 = 31s (capped at 30s each: 1+2+4+8+16=31s for 5 retries). That's fine. Actually, let me think about what's reasonable: - `maxAttempts = 5` (1 initial + 4 retries) - Base delay 1s, doubling: 1s, 2s, 4s, 8s β†’ worst case total backoff 15s + jitter. - Jitter: full jitter (random between 0 and delay) or "equal jitter". Full jitter is the standard AWS recommendation. But for tests, we want determinism β€” make the jitter function injectable, or use a seedable source. Better: make the retry policy a struct with fields, and have a `sleep` function field that tests can stub. **Architecture:** Option A: Put the retry in `runLoop` (agentrun package). Option B: Put the retry in the `llm.Client` (llm package) β€” e.g., a `StreamChatCompletionWithRetry`. The issue says "in the agent loop," so Option A is what's being asked for. But there's a subtlety: mid-stream failures. The `Streamer` reads from the response body. If the stream breaks mid-way, `streamer.Get()` returns an error. To retry, we need to re-issue the whole request. That's exactly what wrapping the turn in `runLoop` does. But there's a problem: the `Hooks`. If we've already streamed partial content via `OnReasoningDelta`/`OnContentDelta` and then retry the turn, the viewer sees the partial content followed by the new content. And the `prevContent`/`prevReasoning` tracking in `runLoop` β€” on retry, we need to reset these (since it's a new stream, content starts from scratch). How to handle the live-view duplication? Options: 1. Only retry if nothing has been streamed yet (no deltas published). Clean, but misses the mid-stream recovery case. 2. Retry anyway, and add a new hook `OnTurnReset` that the dashboard can use to clear the current turn's display. Let me look at how the web dashboard consumes these events and see if a reset event would be easy to support. Actually, let me reconsider. What does the dashboard do with reasoning_delta / content_delta? Let me check the web templates / JS.
Tool πŸ”§ grep_search {"context_after": 15, "context_before": 5, "query": "reasoning_delta|content_delta"}
{"context_after": 15, "context_before": 5, "query": "reasoning_delta|content_delta"}
/project/internal/web/templates.go-962-
/project/internal/web/templates.go-963-        switch (ev.type) {
/project/internal/web/templates.go-964-          case "reasoning_start":
/project/internal/web/templates.go-965-            reasoningBody = newBlock("reasoning", "Thinking");
/project/internal/web/templates.go-966-            break;
/project/internal/web/templates.go:967:          case "reasoning_delta":
/project/internal/web/templates.go-968-            if (reasoningBody) reasoningBody.textContent += ev.text;
/project/internal/web/templates.go-969-            break;
/project/internal/web/templates.go-970-          case "reasoning_end":
/project/internal/web/templates.go-971-            // Model output routinely starts/ends with newlines; drop them
/project/internal/web/templates.go-972-            // so the finished block matches the static (trimmed) view.
/project/internal/web/templates.go-973-            if (reasoningBody) reasoningBody.textContent = reasoningBody.textContent.trim();
/project/internal/web/templates.go-974-            reasoningBody = null;
/project/internal/web/templates.go-975-            break;
/project/internal/web/templates.go-976-          case "content_start":
/project/internal/web/templates.go-977-            contentBody = newBlock("content", null);
/project/internal/web/templates.go-978-            break;
/project/internal/web/templates.go:979:          case "content_delta":
/project/internal/web/templates.go-980-            if (contentBody) contentBody.textContent += ev.text;
/project/internal/web/templates.go-981-            break;
/project/internal/web/templates.go-982-          case "content_end":
/project/internal/web/templates.go-983-            if (contentBody) contentBody.textContent = contentBody.textContent.trim();
/project/internal/web/templates.go-984-            contentBody = null;
/project/internal/web/templates.go-985-            break;
/project/internal/web/templates.go-986-          case "tool":
/project/internal/web/templates.go-987-            newToolBlock(ev);
/project/internal/web/templates.go-988-            break;
/project/internal/web/templates.go-989-          case "status":
/project/internal/web/templates.go-990-            es.close();
/project/internal/web/templates.go-991-            location.reload();
/project/internal/web/templates.go-992-            return;
/project/internal/web/templates.go-993-        }
/project/internal/web/templates.go-994-
--
/project/internal/livelog/hub.go-9-
/project/internal/livelog/hub.go-10-type Type string
/project/internal/livelog/hub.go-11-
/project/internal/livelog/hub.go-12-const (
/project/internal/livelog/hub.go-13-	ReasoningStart Type = "reasoning_start"
/project/internal/livelog/hub.go:14:	ReasoningDelta Type = "reasoning_delta"
/project/internal/livelog/hub.go-15-	ReasoningEnd   Type = "reasoning_end"
/project/internal/livelog/hub.go-16-	ContentStart   Type = "content_start"
/project/internal/livelog/hub.go:17:	ContentDelta   Type = "content_delta"
/project/internal/livelog/hub.go-18-	ContentEnd     Type = "content_end"
/project/internal/livelog/hub.go-19-	Tool           Type = "tool"
/project/internal/livelog/hub.go-20-	Status         Type = "status"
/project/internal/livelog/hub.go-21-)
/project/internal/livelog/hub.go-22-
/project/internal/livelog/hub.go-23-type Event struct {
/project/internal/livelog/hub.go-24-	Type      Type   `json:"type"`
/project/internal/livelog/hub.go-25-	Text      string `json:"text,omitempty"`
/project/internal/livelog/hub.go-26-	Name      string `json:"name,omitempty"`
/project/internal/livelog/hub.go-27-	Arguments string `json:"arguments,omitempty"`
/project/internal/livelog/hub.go-28-	Result    string `json:"result,omitempty"`
/project/internal/livelog/hub.go-29-	Error     bool   `json:"error,omitempty"`
/project/internal/livelog/hub.go-30-	Status    string `json:"status,omitempty"`
/project/internal/livelog/hub.go-31-}
/project/internal/livelog/hub.go-32-
Thinking
OK, so the dashboard appends deltas to a block. If a turn is retried after partial streaming, the new deltas will be appended to the same block (since `reasoningBody`/`contentBody` remain set β€” no `reasoning_end` was emitted), so the viewer will see partial text + full new text concatenated. That's ugly. Design decision: I'll add a `TurnReset` livelog event type, and a corresponding `OnTurnReset` hook. When a turn is retried after partial streaming, the hook fires, the dashboard clears the current reasoning/content blocks, and the streamHooks in run.go reset its own buffer. This gives us clean UX. Wait, but there's a subtlety in streamHooks: `OnTurnEnd` persists the buffer to the store and checkpoints the hub. If the turn is retried, we should NOT persist the partial buffer. So on reset, streamHooks should discard the buffer without persisting. Let me design: - `Hooks.OnTurnReset func()` β€” fires when a turn's stream is being restarted after a transient failure. - In `streamHooks`: on reset, if reasoningOpen, publish `ReasoningEnd`? No wait β€” that would make the dashboard think the block is complete and trim it. Then a new `ReasoningStart` would create a new block. Hmm, that would actually work for the dashboard: `reasoning_end` trims and nulls the block, then `reasoning_start` creates a new block. But semantically "end" is wrong, and the persisted log isn't touched (good β€” OnTurnEnd is what persists, and OnTurnReset shouldn't persist). Hmm, wait, let me actually reconsider. If we emit `reasoning_end` (which the dashboard treats as block completion + trim) and then a new `reasoning_start`, the viewer sees: partial text (trimmed, looks complete), then a new block with the full new text. That's actually not bad UX! And no new event type is needed. But it's semantically misleading β€” the block wasn't complete, it was interrupted. Alternatively, a dedicated `turn_reset` event: the dashboard clears the current block and shows a "retrying…" status. Cleaner semantics, slightly more code (new event type + dashboard case). Hmm, let me think about scope. The issue is specifically about retry logic in the agent loop. The live-view duplication is a secondary concern. But as a senior architect, I should handle it properly β€” a half-baked retry that shows garbage in the dashboard is worse than no retry. Let me go with a dedicated approach, but keep it minimal: 1. **`internal/llm`**: Add error classification. `NewStreamer` already returns an error for non-200. I'll add an `Error` type or helper `IsTransient(err error) bool` that classifies: - `*url.Error` (transport errors) β†’ transient - HTTP status 5xx or 429 β†’ transient - Mid-stream read errors β†’ transient (these surface from `streamer.Get()` as raw read errors β€” hmm, how do I distinguish a mid-stream read error from a decode error? Decode errors are wrapped with "decode completion chunk". Read errors are raw from the scanner. I could also wrap scanner errors… let me look at streamer.Get again: `if err := s.scanner.Err(); err != nil { return nil, err }` β€” raw error. And decode: `fmt.Errorf("decode completion chunk: %w: %s", err, buf)`.) Actually, cleaner: define the classification inside the llm package where the errors originate: - `NewStreamer` non-OK status β†’ wrap in a typed error `StatusError{StatusCode int}` so classification is easy. - Transport errors from `httpClient.Do` β†’ `*url.Error` wrapping `*net.OpError` etc. - Mid-stream read errors from the scanner β†’ wrap in a typed error `StreamError`? Hmm. Let me design a clean API: ```go // in llm package type StatusError struct { StatusCode int Body string } func (e *StatusError) Error() string { ... } ``` And `NewStreamer` returns `*StatusError` for non-200. For transport: `StreamChatCompletion` returns the raw error from `httpClient.Do`. These are `*url.Error`. For mid-stream: `streamer.Get()` returns raw scanner errors (io errors from reading the body). Then a classifier in llm: ```go // IsTransient reports whether err is likely to succeed if the same // request is retried: transport failures, 5xx/429 responses, and // mid-stream interruptions. 4xx (other than 429) and malformed // responses are not transient. func IsTransient(err error) bool ``` Implementation: - `var se *StatusError; errors.As(err, &se)` β†’ transient if 5xx or 429. - `var ue *url.Error; errors.As(err, &ue)` β†’ transient (connection-level). Note: url.Error can also wrap non-OK… no, url.Error is for transport. But careful: `url.Error` wraps any error from the transport including "context canceled" β€” so we need to exclude ctx errors first. - Mid-stream read errors: hard to classify generically. `*net.OpError`, `io.ErrUnexpectedEOF`, `bufio.Scanner` errors… scanner.Err() returns the underlying read error. Hmm, this is getting complex. A simpler, more robust approach: **default to retryable unless we know it's permanent**. I.e., classify as permanent only: - `ctx.Err()` (context.Canceled / DeadlineExceeded) - `*StatusError` with 4xx (other than 429) - Decode errors (malformed response) β€” wrapped with a marker. Everything else (transport, 5xx, 429, mid-stream read) β†’ retryable. This is a defensible policy: if we don't know why it failed, assume the endpoint is flaky and retry with backoff (bounded attempts). If we know it's a permanent client error (bad auth, bad request), fail fast. To make decode errors identifiable, I'll define a `DecodeError` type in llm that streamer.Get wraps with. 2. **`internal/agentrun/loop.go`**: Wrap the turn in a retry loop: ```go for { // attempt the turn ... } ``` Structure: ```go func runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) { for { completion, err := runTurn(ctx, client, messages, hooks) if err == nil { // append message, handle tool calls or return } // on error: if transient and attempts remain, backoff and retry the turn } } ``` Where `runTurn` does the request + stream read, firing hooks. Retry state: attempt count, backoff delay. Where do the retry parameters come from? Options: - Constants in agentrun (simple). - Config (zoo.hcl `llm` block: `retry_attempts`, etc.). Config is nicer but adds surface area. The issue says "some kind of retry logic" β€” I think reasonable defaults as constants, possibly exposed via config. Hmm. Let me think about what's most surgical. The LLM config block is per-LLM (name, openai, token, model). Adding retry knobs there is plausible: `retry = 5` or similar. But HCL validation changes, config tests, etc. Let me keep it surgical: constants in agentrun with clear names and a comment. Actually β€” wait. Let me reconsider. The backoff total is bounded (~15-30s) and per-turn, so it's fine for a 120-minute run. Constants are fine. I'll note in a comment that they could be made configurable later. 3. **Hooks and live view**: Add `OnTurnReset func()` to Hooks. In `runTurn`, when we're about to retry after partial streaming, call `hooks.OnTurnReset()`. In `streamHooks` (run.go), implement it: - Discard the current reasoning/content buffer (don't persist). - Publish a `TurnReset` livelog event so the dashboard can clear the partial block. And in the dashboard JS (templates.go), handle the `turn_reset` event: clear the current reasoning/content block (remove it from the DOM) so the retried turn starts fresh. Wait, but there's a subtlety: the hub's replay buffer. The partial deltas are in the hub's buffer (not checkpointed). When a new subscriber connects, they get the partial deltas replayed, then the new deltas β€” duplication in the replay too. On reset, we should `hub.Checkpoint(jobID)` to clear the buffer (the partial content is discarded, so it's correct to clear β€” it's not persisted, so a reconnecting viewer shouldn't see it). So `OnTurnReset` in streamHooks: ```go OnTurnReset: func() { reasoningOpen, contentOpen = false, false reasoningBuf.Reset() contentBuf.Reset() r.hub.Checkpoint(jobID) // discard un-persisted partial deltas from the replay buffer }, ``` And publish a `TurnReset` event… wait, order matters: publish TurnReset first (so live subscribers clear their blocks), then Checkpoint (clears the buffer). Actually, if we Checkpoint first and then publish TurnReset, a subscriber that connects in between gets an empty buffer + TurnReset. If we publish TurnReset first and then Checkpoint, the buffer still contains the partial deltas + TurnReset, and a subscriber connecting in between gets partial deltas replayed (which they'll clear via TurnReset β€” since the JS processes events in order, that works: append deltas, then clear on reset). Either order works for in-order processing. Let me do: publish TurnReset, then Checkpoint. Hmm wait, actually, should the dashboard show something like "retrying"? A `Status` event with a status text? The dashboard's `status` case closes the SSE and reloads the page! So we can't use Status for a transient notice. Let me just clear the blocks. Optionally, we could show a small "connection interrupted, retrying…" note, but that adds UI complexity. Let me keep it minimal: clear the partial block. The user sees the text disappear and new text stream in. That's clear enough. Actually, hold on. Let me reconsider whether to fire OnTurnReset only when content was actually streamed, or always on retry. If the request failed before any delta (e.g., connection refused), no hooks fired, nothing to reset. Firing OnTurnReset unconditionally is harmless (checkpointing an empty buffer is a no-op, publishing turn_reset with no open block is a no-op in JS). But it's cleaner to only fire when something was streamed. Hmm β€” but "something was streamed" is tracked in streamHooks (reasoningOpen/contentOpen), not in runLoop. In runLoop, we can track whether any delta fired during this turn attempt (prevContent/prevReasoning non-empty, or… actually, tool calls can't start mid-stream). Simplest: in runTurn, track `streamed := false`, set to true when a delta hook fires. On retry, if streamed, call OnTurnReset. Hmm, wait. Actually, there's an even simpler consideration: should we even retry after partial streaming? Let me reconsider the tradeoff: - Retry after partial streaming: recovers from mid-stream failures (common with local LLM servers: OOM kill, GPU reset, network blip). Cost: viewer sees the partial text disappear and a new generation start. Token cost: the partial generation is wasted. - Don't retry after partial streaming: the run dies on any mid-stream failure. For long generations (this is a coding agent, turns can be long), that's a big loss. I'll retry regardless of how much was streamed. The whole point of the issue is resilience against "the LLM endpoint is failing." 4. **Backoff details**: - Max attempts: 5 total (1 + 4 retries). - Base: 1s, factor 2, cap 15s? Delays: 1, 2, 4, 8 β†’ total 15s. With jitter: uniform [0, delay] (full jitter) or [delay/2, delay] (equal jitter)? Full jitter can be very short (near 0), which for a downed endpoint means rapid retries β€” fine, they're bounded. AWS recommends full jitter. Let me use full jitter: `time.Duration(rand.Float64() * float64(delay))`. Hmm, but a 1ms retry against a downed endpoint is a bit silly. Equal jitter (delay/2 + rand*delay/2) guarantees at least half the backoff. Let me use equal jitter β€” guarantees meaningful spacing while avoiding thundering herd. Actually, for a single client (zoo runs one agent at a time per job, few jobs), thundering herd is not a concern. But equal jitter is still better than full jitter here. Let me go with equal jitter. - The sleep must respect ctx: `select { case <-time.After(d): case <-ctx.Done(): return ctx.Err() }`. - Make the RNG injectable for tests? For testability, I'll structure the retry as a small helper with a `sleep func(ctx, time.Duration) error` or a jitter function. Let me design: ```go // retryPolicy bounds how hard runLoop hammers a failing LLM endpoint. type retryPolicy struct { maxAttempts int baseDelay time.Duration maxDelay time.Duration } ``` And in runLoop: ```go const ( llmMaxAttempts = 5 llmBaseDelay = time.Second llmMaxDelay = 15 * time.Second ) ``` For tests, I want to verify: transient errors are retried, permanent errors are not, attempts are bounded, backoff is respected (with a fake clock/sleep). Let me make the sleep function a variable or pass it through. Since runLoop is a package-level function called from Runner.Run, I could add an unexported package variable `var sleep = func(ctx context.Context, d time.Duration) error {...}` that tests override. That's a common Go pattern. Or better: extract the retry into a testable function: ```go // withRetry retries fn while it returns a transient error... func withRetry(ctx context.Context, policy retryPolicy, sleep func(ctx context.Context, d time.Duration) error, fn func() error) error ``` Hmm, let me think about how to structure runLoop cleanly. Current structure: ```go for { // turn loop streamer, err := client.StreamChatCompletion(...) ... for { c, err := streamer.Get(); ... } // stream read hooks.OnTurnEnd() // process completion, tool calls } ``` New structure: ```go for { // turn loop completion, streamed, err := requestTurn(ctx, client, messages, hooks) // retry loop around requestTurn: ... } ``` Let me write it out: ```go func runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) { for { if err := ctx.Err(); err != nil { return messages, err } completion, err := requestTurn(ctx, client, messages, hooks) if err != nil { return messages, err } choice := completion.Choices[0] messages = append(messages, choice.Message) if choice.FinishReason != "tool_calls" || len(choice.Message.ToolCalls) == 0 { return messages, nil } for _, tc := range choice.Message.ToolCalls { ... (unchanged) } } } ``` And: ```go // requestTurn performs one LLM round trip (request + full stream read), // retrying transient endpoint failures with exponential backoff. It // returns the completed turn's completion. func requestTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) { var lastErr error for attempt := 0; ; attempt++ { completion, streamed, err := streamTurn(ctx, client, messages, hooks) if err == nil { return completion, nil } lastErr = err if ctxErr := ctx.Err(); ctxErr != nil { return nil, ctxErr // hmm, or the original err? } if !llm.IsTransient(err) || attempt == maxAttempts-1 { return nil, err } if streamed && hooks.OnTurnReset != nil { hooks.OnTurnReset() } if err := sleepCtx(ctx, backoff(attempt)); err != nil { return nil, err } } } ``` Wait, subtle point: if ctx is done, we should return ctx's error (the run was canceled/timed out) rather than the LLM error. Actually, if the ctx deadline fired, the LLM error is probably also a deadline error. Returning ctx.Err() is cleaner. But careful: the `ctx.Err()` check β€” if the error is a transient network error and ctx is still alive, we retry. If ctx is done, we stop. Hmm, another subtlety: what if `streamTurn` returns an error AND ctx is done? E.g., the deadline fired during a stream read. The error might be a `*url.Error` wrapping "context deadline exceeded". `llm.IsTransient` would classify `*url.Error` as transient β€” but we must not retry after ctx is done. The `ctx.Err()` check handles that: return ctx.Err() immediately. Good. And `streamTurn`: ```go // streamTurn performs one attempt: issue the request and read the // stream to completion, firing hooks as content arrives. It reports // whether any content was streamed (so callers can reset the live // view before retrying). func streamTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, bool, error) { streamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{ Messages: messages, Stream: true, Tools: tools.All(), }) if err != nil { return nil, false, fmt.Errorf("chat completion: %w", err) } var completion *llm.ChatCompletion streamed := false var prevContent, prevReasoning string for { c, err := streamer.Get() if err == io.EOF { break } if err != nil { return nil, streamed, fmt.Errorf("stream completion: %w", err) } completion = c if len(c.Choices) > 0 { msg := c.Choices[0].Message if hooks.OnReasoningDelta != nil && len(msg.ReasoningContent) > len(prevReasoning) { hooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):]) streamed = true } prevReasoning = msg.ReasoningContent if hooks.OnContentDelta != nil && len(msg.Content) > len(prevContent) { hooks.OnContentDelta(msg.Content[len(prevContent):]) streamed = true } prevContent = msg.Content } } if hooks.OnTurnEnd != nil { hooks.OnTurnEnd() } if completion == nil || len(completion.Choices) == 0 { return nil, streamed, fmt.Errorf("model returned an empty completion") } return completion, streamed, nil } ``` Wait β€” the `streamed` flag: should it be set when a delta fires, or when content arrives? If the hook is nil (headless test), streamed won't be set. Better: track it based on content arrival, not hook invocation: ```go if len(msg.ReasoningContent) > len(prevReasoning) || len(msg.Content) > len(prevContent) { streamed = true } ``` That's more accurate: "the endpoint sent us some of this turn." Hmm, also: should `streamed` consider tool call deltas? Tool call arguments also stream. If a tool call was partially streamed and then the stream broke… the turn is retried, and the partial tool call is discarded (it was never executed). The live view doesn't show tool calls until they execute (OnTool fires after execution). So no live-view concern for tool calls. But for the "was anything streamed" decision, tool call deltas also count as streamed content. Let me keep it simple: check content/reasoning growth, and also tool call argument growth. Hmm, actually β€” even simpler and more robust: `streamed = completion != nil` after the first Get returns a completion? No wait β€” the first Get returns the first chunk, which might be an empty role-only delta. Hmm. Let me just track it: any chunk with non-empty delta content (content, reasoning, or tool call arguments) β†’ streamed. Actually, you know what? Let me simplify: `streamed := false`, and set it to true when `c` (the accumulated completion) has any non-empty choice message content. Let me write a small helper: ```go if len(c.Choices) > 0 && (c.Choices[0].Message.Content != "" || c.Choices[0].Message.ReasoningContent != "" || len(c.Choices[0].Message.ToolCalls) > 0) { streamed = true } ``` Since `c` is the accumulated completion, once anything arrives it stays true. Clean. Now, the "empty completion" error: the stream ended cleanly (EOF) but no choices. Is that transient? I said yes earlier. With my "default retryable unless known permanent" policy, it's retryable (not a StatusError, not a decode error, not a ctx error). Good β€” it'll retry up to max attempts. Now, `llm.IsTransient` classification. Let me define in the llm package: ```go // StatusError is returned when the endpoint answers with a non-200 // status. type StatusError struct { StatusCode int Body string } func (e *StatusError) Error() string { return fmt.Sprintf("llm endpoint returned status %d: %s", e.StatusCode, e.Body) } // DecodeError is returned when the endpoint sends a chunk that is // not valid JSON... type DecodeError struct{ ... } ``` Hmm, do I need a DecodeError type? The current decode error is `fmt.Errorf("decode completion chunk: %w: %s", err, buf)`. I could classify by type: wrap in `DecodeError{Err error, Chunk string}`. Let me do it properly: ```go type DecodeError struct { Err error Chunk string } func (e *DecodeError) Error() string { return fmt.Sprintf("decode completion chunk: %v: %s", e.Err, e.Chunk) } func (e *DecodeError) Unwrap() error { return e.Err } ``` And `IsTransient`: ```go // IsTransient reports whether a failed LLM round trip is worth // retrying. Transport failures (connection refused/reset, DNS, // timeouts), 5xx and 429 responses, and mid-stream interruptions are // transient: the endpoint may be down or overloaded and the same // request may succeed later. Client errors (other 4xx) and malformed // responses are not: retrying the same request cannot fix them. // Context cancellation is never transient β€” the caller gave up. func IsTransient(err error) bool { if err == nil { return false } if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return false } var se *StatusError if errors.As(err, &se) { return se.StatusCode == http.StatusTooManyRequests || se.StatusCode >= 500 } var de *DecodeError if errors.As(err, &de) { return false } return true } ``` Wait, but mid-stream read errors: what do they look like? `scanner.Err()` returns the raw error from reading the body β€” e.g., `*net.OpError` (connection reset), `io.ErrUnexpectedEOF`, `*os.File`… For an http response body, typically `*net.OpError` or `http2` errors. These aren't StatusError or DecodeError β†’ default to transient. Good. But wait β€” one more: `bufio.Scanner`'s "token too long" error (`bufio.ErrTooLong`) β€” that's a protocol issue, not transient. Edge case, ignore (default transient, bounded attempts). Hmm, also `NewStreamer` currently: `fmt.Errorf("status: %d: %s", resp.StatusCode, body)`. I'll change this to return `&StatusError{StatusCode: resp.StatusCode, Body: string(body)}`. The body could be huge; should I truncate it? The current code reads the whole body. Let me keep it, but maybe cap at a few KB for the error message. Let me cap at 1KB to avoid pathological error strings. Actually, the current code doesn't cap; changing to a cap is a small improvement, but let me keep the behavior close to the original β€” I'll cap at 512 bytes with a note. Hmm, let me keep it simple: read up to 1KB. Now, `StreamChatCompletion`'s transport error: `c.httpClient.Do(r)` returns a `*url.Error` on connection failure. That's not StatusError/DecodeError β†’ transient. Good. But note: if the ctx is canceled during Do, the error is a `*url.Error` wrapping `context.Canceled` β€” the `errors.Is` check catches it. Good. Now, where does the retry live β€” agentrun or llm? I'll put the policy (attempts, backoff, sleep) in agentrun's loop.go, and the classification in llm (since llm knows its own error types). The issue says "in the agent loop" β€” so agentrun/loop.go is the right place. Backoff function: ```go // llmRetry bounds how hard the agent loop hammers a failing LLM // endpoint. A run can last hours, so the budget is per turn (one // request + stream round trip), not per run. const ( llmMaxAttempts = 5 llmBaseDelay = time.Second llmMaxDelay = 15 * time.Second ) // llmBackoff returns the delay before retry attempt n (0-based): // exponential growth from llmBaseDelay, capped at llmMaxDelay, with // equal jitter (half the delay fixed, half random) so concurrent // agents don't retry in lockstep. func llmBackoff(attempt int, rnd *rand.Rand) time.Duration { d := llmBaseDelay << attempt if d > llmMaxDelay || d <= 0 { d = llmMaxDelay } return d/2 + time.Duration(rnd.Int63n(int64(d/2)+1)) } ``` Hmm, `llmBaseDelay << attempt`: 1s<<0=1s, <<1=2s, <<2=4s, <<3=8s, <<4=16sβ†’capped at 15s. With 5 attempts, we sleep after attempts 0..3 (4 sleeps): 1, 2, 4, 8 β†’ total 15s worst case. Good. For testability, I'll make the sleep injectable. Options: - Package variable `var llmSleep = func(ctx context.Context, d time.Duration) error` β€” tests override it. Simple, common. - Pass through a struct. runLoop's signature is called from run.go; I don't want to change the public signature (it's unexported, but still). A package variable for sleep + a package variable for the RNG? The RNG only matters for the jitter bounds; in tests, we can use a fixed seed. Let me do: ```go var ( llmSleep = func(ctx context.Context, d time.Duration) error { select { case <-time.After(d): return nil case <-ctx.Done(): return ctx.Err() } } ) ``` And in requestTurn, use a `rand.New(rand.NewSource(time.Now().UnixNano()))` β€” or math/rand's global (in Go 1.20+ it's auto-seeded and safe for concurrency). Let me check the Go version in go.mod. Let me check go.mod. Also β€” one more design question: should the retry also apply to tool calls? No β€” tools run locally in the sandbox; the issue is specifically about the LLM endpoint. And what about `Run` in run.go β€” the forgejo calls there (RepositoryInfo, ListIssueComments, etc.)? Out of scope; the issue says "in the agent loop, when the LLM endpoint is failing." Now, the dashboard: add a `TurnReset Type = "turn_reset"` event. In streamHooks: ```go OnTurnReset: func() { reasoningOpen, contentOpen = false, false reasoningBuf.Reset() contentBuf.Reset() r.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset}) r.hub.Checkpoint(jobID) }, ``` Wait, order: publish then checkpoint. A subscriber that connects after the checkpoint gets an empty buffer β€” they miss the turn_reset, but they also miss the partial deltas (which is correct, since they were discarded). A live subscriber gets turn_reset and clears the block. Good. In the dashboard JS: ```js case "turn_reset": // A turn's stream broke and is being retried; drop the partial // block so the retried turn starts clean. if (reasoningBody) { reasoningBody.closest(".block")?.remove(); reasoningBody = null; } // hmm, need to check the DOM structure if (contentBody) { ... } break; ``` I need to look at the JS more carefully to see how blocks are created/removed. Let me read the relevant part of templates.go. Also, is `OnTurnReset` needed in the Hooks struct? Yes: ```go // OnTurnReset fires when a turn's stream fails transiently and is // about to be retried after some of it was already streamed, so the // live view can discard the partial output. OnTurnReset func() ``` Now β€” tests. What should I test? 1. `llm.IsTransient` classification: StatusError 500/503/429 β†’ true; 400/401/404 β†’ false; url.Error β†’ true; DecodeError β†’ false; context.Canceled β†’ false; wrapped errors (fmt.Errorf %w) β†’ still classified. 2. `NewStreamer` returns *StatusError (errors.As works). 3. runLoop retry behavior (in agentrun): I need a fake LLM client… but runLoop takes a concrete `*llm.Client`, not an interface. Hmm. That's a testability problem. Options: - Change runLoop to take an interface for the streaming call. - Test via a real httptest server (integration-ish): stand up an httptest.Server that fails N times then succeeds. That's actually a great test β€” it exercises the real client, real streamer, real retry. With a short backoff… but the backoff is a constant (1s base). A test with 2 retries would take ~1.5s+. I can override `llmSleep` in the test (same package) to no-op. An httptest-based test is the most end-to-end and doesn't require refactoring runLoop's signature. Let me do that: - Test A: server returns 500 twice, then a valid SSE stream β†’ runLoop succeeds, and the server saw 3 requests. - Test B: server returns 400 β†’ runLoop fails immediately, 1 request. - Test C: server always 500 β†’ runLoop fails after 5 requests. - Test D: server streams a partial SSE then cuts the connection (close the body early / hijack) β†’ retry succeeds. - Test E: ctx canceled during backoff sleep β†’ returns promptly. - Also verify OnTurnReset fires when a partial stream is retried. For the partial stream test: an httptest handler that writes a few SSE chunks, then `hijack`s and closes the TCP connection. Or more simply: write chunks and return without [DONE]… no, returning from the handler cleanly closes the body β†’ the scanner gets EOF, not an error β†’ the stream "completes" without [DONE]… let me check: if the body ends without [DONE], `scanner.Scan()` returns false, `scanner.Err()` is nil (clean EOF), the loop breaks… wait: ```go for s.scanner.Scan() { if err := s.scanner.Err(); err != nil { return nil, err } ... } ``` If Scan returns false with no error (clean EOF), the for loop exits, `buf` is whatever it was (nil or the last non-"data:" line), then `buf = bytes.TrimPrefix(buf, prefix)` β†’ empty, not [DONE], `json.Unmarshal("")` β†’ error "decode completion chunk: unexpected end of JSON input". So a clean early close surfaces as a DecodeError β†’ non-transient β†’ no retry! Hmm! That's a problem. A server that drops the connection cleanly (FIN) mid-stream produces a decode error in the current code. Is that "malformed response" or "stream interruption"? It's an interruption β€” the stream ended without [DONE]. Retrying is the right call. So I need to distinguish: - Body ended cleanly (EOF) without [DONE] β†’ stream was cut β†’ transient (StreamError: "stream ended before [DONE]"). - A complete data line failed to decode β†’ malformed β†’ non-transient (DecodeError). Let me restructure streamer.Get to handle this: ```go func (s *Streamer) Get() (*ChatCompletion, error) { if s.eof { return nil, io.EOF } prefix := []byte("data: ") var buf []byte for s.scanner.Scan() { if err := s.scanner.Err(); err != nil { return nil, err } buf = bytes.TrimSpace(s.scanner.Bytes()) if len(buf) == 0 { continue } if bytes.HasPrefix(buf, prefix) { break } } if err := s.scanner.Err(); err != nil { return nil, err // mid-stream read failure } if !s.scanner... // hmm, need to know if Scan stopped due to EOF } ``` Wait, the current code has a bug-ish pattern: `if err := s.scanner.Err(); err != nil` is checked inside the loop, after Scan returns true β€” at that point Err() is always nil (Scan only returns true on a successful token). The check is dead code inside the loop. The real error check should be after the loop: if Scan returns false, check Err() β€” non-nil means a read error, nil means clean EOF. So: ```go for s.scanner.Scan() { buf = bytes.TrimSpace(s.scanner.Bytes()) if len(buf) == 0 { continue } if bytes.HasPrefix(buf, prefix) { break } } if err := s.scanner.Err(); err != nil { return nil, err } buf = bytes.TrimPrefix(buf, prefix) if buf == nil/empty && !foundDataLine { // clean EOF without a data line: stream ended s.resp.Body.Close() s.eof = true return nil, &StreamError{...} // hmm, or io.ErrUnexpectedEOF? } ``` Hmm wait, but note: the current behavior when the body ends cleanly without [DONE] is a decode error (non-transient in my scheme). If I change it to a StreamError (transient), that changes the behavior for a genuinely malformed but complete stream… no wait β€” a complete stream always ends with [DONE] (the OpenAI protocol). A stream that ends without [DONE] is, by definition, interrupted. So: - Scanner hits a data line β†’ process it (decode error possible β†’ DecodeError, non-transient). - Scanner ends (EOF) without finding a data line and without [DONE] β†’ the stream was cut β†’ return an error that's classified as transient. What error? I could return `io.ErrUnexpectedEOF` β€” semantically perfect ("stream ended before [DONE]") and it's not StatusError/DecodeError β†’ transient by default. Hmm, but a raw io.ErrUnexpectedEOF with no context is cryptic. Let me define: ```go // StreamError is returned when the response body ends (or breaks) // before the stream's [DONE] marker, i.e. the connection dropped // mid-turn. type StreamError struct{ Detail string } func (e *StreamError) Error() string { return "llm stream interrupted: " + e.Detail } ``` Hmm, or simpler: just return `io.ErrUnexpectedEOF` with a wrap: `fmt.Errorf("stream ended before [DONE]: %w", io.ErrUnexpectedEOF)`. IsTransient: not StatusError/DecodeError/ctx β†’ transient. That works and reuses the standard library. But a named type is more explicit and self-documenting for future classification. Let me go with a named type β€” it makes the error taxonomy explicit: StatusError (HTTP), DecodeError (protocol), StreamError (interruption), plus raw transport errors. Wait, actually, hmm: what about scanner errors (mid-stream read failure)? Those are raw (e.g., *net.OpError). Should I wrap them in StreamError too for consistency? `return nil, &StreamError{Detail: err.Error()}`… but then I lose the original error for errors.Is/As. Let me do: ```go if err := s.scanner.Err(); err != nil { return nil, &StreamError{Err: err} } ``` with `func (e *StreamError) Unwrap() error`. Hmm, OK wait, I'm going back and forth. Let me settle the llm package's error taxonomy concretely: ```go // StatusError: non-200 HTTP response. type StatusError struct { StatusCode int Body string } func (e *StatusError) Error() string { return fmt.Sprintf("llm endpoint returned status %d: %s", e.StatusCode, e.Body) } // StreamError: the response body broke or ended before the // stream's [DONE] marker. type StreamError struct { Err error // underlying read error, if any } func (e *StreamError) Error() string { if e.Err != nil { return fmt.Sprintf("llm stream interrupted: %v", e.Err) } return "llm stream ended before [DONE]" } func (e *StreamError) Unwrap() error { return e.Err } // DecodeError: a data line that is not valid JSON. type DecodeError struct { Err error Chunk string } func (e *DecodeError) Error() string { return fmt.Sprintf("decode completion chunk: %v: %s", e.Err, e.Chunk) } func (e *DecodeError) Unwrap() error { return e.Err } ``` And IsTransient: ```go func IsTransient(err error) bool { if err == nil { return false } if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return false } var se *StatusError if errors.As(err, &se) { return se.StatusCode == http.StatusTooManyRequests || se.StatusCode >= 500 } var de *DecodeError if errors.As(err, &de) { return false } return true } ``` Wait, one more consideration: should 408 (Request Timeout) be transient? Yes, definitely. 408 is a client-side timeout from the server's perspective β€” retryable. Let me add: `se.StatusCode == 408 || se.StatusCode == 429 || se.StatusCode >= 500`. Also 529 (some providers use it for "overloaded")? That's >= 500, covered. Now, the existing test `TestStreamerNonOKStatus` expects an error for non-200 β€” still passes with StatusError. Are there other places that construct/parse these errors? Let me grep for "status: %d" and "decode completion chunk" elsewhere (e.g., web display, logs). The error strings flow into job failure messages (the scheduler comments on the issue with the failure). Changing the format slightly is fine. Now, let me also double-check `NewStreamer`'s body read: `body, _ := io.ReadAll(resp.Body)`. I'll cap it: `io.LimitReader(resp.Body, 1024)`. Now, the agentrun loop changes. Let me also think: `requestTurn` needs `tools.All()` β€” that's fine, it's already imported. One more thing: the `Hooks.OnTurnEnd` semantics β€” "fires once per completed streamer round trip, after the model's message for that turn is fully received." On a failed attempt, OnTurnEnd must NOT fire (the turn didn't complete). In my streamTurn, OnTurnEnd fires only after a clean EOF with a valid completion. Good. And the streamHooks' OnTurnEnd persists the buffer β€” correct only for completed turns. But wait, there's an issue with the streamHooks' buffer state across retries: if attempt 1 streams partial reasoning (reasoningOpen=true, buffer has text), then attempt 2 streams new reasoning. OnTurnReset fires between them: resets the buffer and reasoningOpen=false. Then attempt 2's OnReasoningDelta reopens the block. Correct. But what if attempt 1 fails at the request level (no streaming)? OnTurnReset isn't called (streamed=false), the buffer is untouched (empty anyway). No problem. Now, the dashboard JS. Let me read the whole JS section to integrate turn_reset cleanly. Also, I need to double-check: are there other callers of runLoop or Hooks? Let me grep. Let me also check go.mod for the Go version (for math/rand's global functions). Let me also look at the rest of the scheduler (the failure comment) to see how the error surfaces β€” for the final error message quality. After exhausting retries, the error will be something like "agent loop: chat completion: llm endpoint returned status 500: ..." β€” good. Should I wrap the final error with attempt count context? E.g., "after 5 attempts: <last err>". That's useful for debugging. Let me do it in requestTurn: ```go return nil, fmt.Errorf("llm endpoint failed after %d attempts: %w", llmMaxAttempts, lastErr) ``` Hmm, but for a non-transient error (400), it fails on attempt 1 β€” the message "after 1 attempts" is awkward. Let me track the attempt count and only add the wrapper if attempt > 1: ```go if attempt > 0 { return nil, fmt.Errorf("llm endpoint failed after %d attempts: %w", attempt+1, lastErr) } return nil, lastErr ``` OK. Now, let me also think about whether the retry should live in `llm.Client` instead, so future callers benefit. The issue explicitly says "in the agent loop." There's currently only one caller of StreamChatCompletion. I'll put the policy in agentrun (as requested) and the classification in llm (where the error types live). That's a clean split. Now let me check the remaining unknowns: 1. go.mod (Go version) 2. The full dashboard JS (block creation/removal) 3. Other callers of runLoop/Hooks/StreamChatCompletion 4. The rest of the scheduler's failure path (for context)
Now let me look at the remaining pieces: the dashboard JS, go.mod, and any other callers of the LLM client/loop.
Tool πŸ”§ read_file {"end_line": 1000, "path": "internal/web/templates.go", "start_line": 880}
{"end_line": 1000, "path": "internal/web/templates.go", "start_line": 880}
{"path":"internal/web/templates.go","content":"  \u003cscript\u003e\n    (function() {\n      var jobID = {{.Job.ID}};\n      var log = document.getElementById(\"log\");\n      var reasoningBody = null;\n      var contentBody = null;\n\n      function nearBottom() {\n        return (window.innerHeight + window.scrollY) \u003e= (document.body.offsetHeight - 80);\n      }\n\n      function newBlock(kind, label) {\n        var div = document.createElement(\"div\");\n        div.className = \"block block-\" + kind;\n        if (label) {\n          var l = document.createElement(\"div\");\n          l.className = \"block-label\";\n          l.textContent = label;\n          div.appendChild(l);\n        }\n        var body = document.createElement(\"div\");\n        body.className = \"block-body\";\n        div.appendChild(body);\n        log.appendChild(div);\n        return body;\n      }\n\n      function newToolBlock(ev) {\n        var details = document.createElement(\"details\");\n        details.className = \"block block-tool\" + (ev.error ? \" block-tool-error\" : \"\");\n\n        var summary = document.createElement(\"summary\");\n\n        var badge = document.createElement(\"span\");\n        badge.className = \"tool-badge\";\n        badge.textContent = \"Tool\";\n\n        var text = document.createElement(\"span\");\n        text.className = \"tool-summary-text\";\n\n        var name = document.createElement(\"span\");\n        name.className = \"tool-name\";\n        name.textContent = \"πŸ”§ \" + ev.name;\n\n        var preview = document.createElement(\"span\");\n        preview.className = \"tool-args-preview\";\n        preview.textContent = ev.arguments;\n\n        text.appendChild(name);\n        text.appendChild(preview);\n        summary.appendChild(badge);\n        summary.appendChild(text);\n        details.appendChild(summary);\n\n        var body = document.createElement(\"div\");\n        body.className = \"block-body\";\n\n        var argsLabel = document.createElement(\"div\");\n        argsLabel.className = \"tool-section-label\";\n        argsLabel.textContent = \"Arguments\";\n        var argsPre = document.createElement(\"pre\");\n        argsPre.textContent = ev.arguments;\n\n        var resultLabel = document.createElement(\"div\");\n        resultLabel.className = \"tool-section-label\";\n        resultLabel.textContent = \"Result\";\n        var resultPre = document.createElement(\"pre\");\n        resultPre.textContent = ev.result;\n\n        body.appendChild(argsLabel);\n        body.appendChild(argsPre);\n        body.appendChild(resultLabel);\n        body.appendChild(resultPre);\n        details.appendChild(body);\n        log.appendChild(details);\n      }\n\n      var es = new EventSource(\"/jobs/\" + jobID + \"/events\");\n\n      es.onmessage = function(e) {\n        var ev = JSON.parse(e.data);\n        var shouldScroll = nearBottom();\n\n        switch (ev.type) {\n          case \"reasoning_start\":\n            reasoningBody = newBlock(\"reasoning\", \"Thinking\");\n            break;\n          case \"reasoning_delta\":\n            if (reasoningBody) reasoningBody.textContent += ev.text;\n            break;\n          case \"reasoning_end\":\n            // Model output routinely starts/ends with newlines; drop them\n            // so the finished block matches the static (trimmed) view.\n            if (reasoningBody) reasoningBody.textContent = reasoningBody.textContent.trim();\n            reasoningBody = null;\n            break;\n          case \"content_start\":\n            contentBody = newBlock(\"content\", null);\n            break;\n          case \"content_delta\":\n            if (contentBody) contentBody.textContent += ev.text;\n            break;\n          case \"content_end\":\n            if (contentBody) contentBody.textContent = contentBody.textContent.trim();\n            contentBody = null;\n            break;\n          case \"tool\":\n            newToolBlock(ev);\n            break;\n          case \"status\":\n            es.close();\n            location.reload();\n            return;\n        }\n\n        if (shouldScroll) window.scrollTo(0, document.body.scrollHeight);\n      };\n    })();\n  \u003c/script\u003e\n  {{end}}\n\u003c/div\u003e","start_line":880,"end_line":1000,"total_lines":1004,"truncated":true}
Tool πŸ”§ read_file {"end_line": 30, "path": "go.mod", "start_line": 1}
{"end_line": 30, "path": "go.mod", "start_line": 1}
{"path":"go.mod","content":"module github.com/abrander/zoo\n\ngo 1.26.4\n\nrequire (\n\tcodeberg.org/mvdkleijn/forgejo-sdk/forgejo/v2 v2.2.0\n\tgithub.com/docker/docker v28.5.2+incompatible\n\tgithub.com/google/uuid v1.6.0\n\tgithub.com/hashicorp/hcl/v2 v2.24.0\n\tmodernc.org/sqlite v1.56.0\n)\n\nrequire (\n\tgithub.com/42wim/httpsig v1.2.3 // indirect\n\tgithub.com/Microsoft/go-winio v0.6.2 // indirect\n\tgithub.com/agext/levenshtein v1.2.1 // indirect\n\tgithub.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect\n\tgithub.com/cespare/xxhash/v2 v2.3.0 // indirect\n\tgithub.com/containerd/errdefs v1.0.0 // indirect\n\tgithub.com/containerd/errdefs/pkg v0.3.0 // indirect\n\tgithub.com/containerd/log v0.1.0 // indirect\n\tgithub.com/davidmz/go-pageant v1.0.2 // indirect\n\tgithub.com/distribution/reference v0.6.0 // indirect\n\tgithub.com/docker/go-connections v0.8.1 // indirect\n\tgithub.com/docker/go-units v0.5.0 // indirect\n\tgithub.com/dustin/go-humanize v1.0.1 // indirect\n\tgithub.com/felixge/httpsnoop v1.1.0 // indirect\n\tgithub.com/go-fed/httpsig v1.1.0 // indirect\n\tgithub.com/go-logr/logr v1.4.4 // indirect\n\tgithub.com/go-logr/stdr v1.2.2 // indirect","start_line":1,"end_line":30,"total_lines":62,"truncated":true}
Tool πŸ”§ grep_search {"query": "runLoop|StreamChatCompletion|Hooks\\{"}
{"query": "runLoop|StreamChatCompletion|Hooks\\{"}
/project/internal/llm/client.go:30:func (c *Client) StreamChatCompletion(ctx context.Context, req *ChatCompletionRequest) (*Streamer, error) {
/project/internal/agentrun/loop.go:12:// Hooks lets a caller observe a runLoop invocation live, as the model
/project/internal/agentrun/loop.go:30:// runLoop is a headless port of ../a's App.generate(): send messages +
/project/internal/agentrun/loop.go:33:func runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) {
/project/internal/agentrun/loop.go:39:		streamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{
/project/internal/agentrun/run.go:271:	_, err = runLoop(ctx, llmClient, runCtx, messages, hooks)
/project/internal/agentrun/run.go:281:// streamHooks builds the Hooks a single Run passes to runLoop: every
/project/internal/agentrun/run.go:293:	return Hooks{
Thinking
Good. Go 1.26 β€” the global functions of math/rand are auto-seeded and concurrency-safe. There's only one caller of runLoop (run.go), and one caller of StreamChatCompletion (loop.go). Alright, let me lay out the full implementation plan: ### 1. `internal/llm/errors.go` (new file) - `StatusError{StatusCode, Body}` β€” non-200 responses. - `StreamError{Err}` β€” stream interrupted before [DONE] (either a read error or a clean EOF without [DONE]). - `DecodeError{Err, Chunk}` β€” data line that isn't valid JSON. - `IsTransient(err) bool` β€” classification. ### 2. `internal/llm/client.go` - `NewStreamer` returns `*StatusError` for non-200 (body capped at 1KB). ### 3. `internal/llm/streamer.go` - Fix the scanner error check (currently dead code inside the loop; check after the loop). - Clean EOF without a data line β†’ `&StreamError{}` (stream ended before [DONE]). - Scanner read error β†’ `&StreamError{Err: err}`. - Decode failure β†’ `&DecodeError{Err: err, Chunk: string(buf)}`. Wait, I need to be careful with the scanner loop restructure. Current code: ```go var buf []byte for s.scanner.Scan() { if err := s.scanner.Err(); err != nil { return nil, err } buf = bytes.TrimSpace(s.scanner.Bytes()) if len(buf) == 0 { continue } if bytes.HasPrefix(buf, prefix) { break } } buf = bytes.TrimPrefix(buf, prefix) if bytes.Equal(buf, []byte("[DONE]")) { s.resp.Body.Close() s.eof = true return nil, io.EOF } ``` Note: if the loop exits via `break` (found a data line), buf holds the data line. If the loop exits via Scan()==false, buf holds the last non-empty non-data line (or nil). In the current code, in that case it falls through to the decode β†’ error. New code: ```go var buf []byte for s.scanner.Scan() { buf = bytes.TrimSpace(s.scanner.Bytes()) if len(buf) == 0 { continue } if bytes.HasPrefix(buf, prefix) { break } } if err := s.scanner.Err(); err != nil { return nil, &StreamError{Err: err} } buf = bytes.TrimPrefix(buf, prefix) if len(buf) == 0 { // The body ended (or broke) before a data line arrived β€” the // stream was interrupted before [DONE]. s.resp.Body.Close() s.eof = true return nil, &StreamError{} } if bytes.Equal(buf, []byte("[DONE]")) { s.resp.Body.Close() s.eof = true return nil, io.EOF } var completion ChatCompletion if err := json.Unmarshal(buf, &completion); err != nil { return nil, &DecodeError{Err: err, Chunk: string(buf)} } ``` Hmm wait, one edge case: what if the body contains a non-empty non-data line (e.g. `: keepalive` comment or `event: x`)? Then the loop exits via EOF with buf = that line, and we'd return StreamError even though… no wait, that's correct: the stream ended without [DONE] β†’ interrupted. OK. Another edge: what if a data line is found but the scanner had an error before that? Impossible β€” if Scan() returned true, there's no error. Edge: what if the first data line is found but it's an empty `data:` (buf after trim is "data:" with nothing after)? TrimPrefix β†’ empty β†’ len(buf)==0 β†’ StreamError. Hmm, `data:` with an empty payload… per the SSE spec, `data:` with no value is an empty data field β€” a legitimate keepalive-ish thing? In OpenAI streams, data lines always have JSON. An empty data line at the end without [DONE] β†’ interrupted. An empty data line in the middle… my code would treat it as an interrupted stream. Edge case within an edge case; acceptable. Actually wait β€” no. Let me re-check: `buf = bytes.TrimSpace(s.scanner.Bytes())` β€” the line "data:" stays as "data:" (no trailing space to trim). HasPrefix("data: ") is false (needs a space after the colon)! So "data:" is treated as a non-data line and the scan continues. Then if [DONE] comes later, fine. OK, no problem. Now, `s.eof = true` on StreamError: should subsequent Get() calls return io.EOF? If the stream is interrupted, the caller (runLoop) will retry the entire turn β€” a new streamer. So the old streamer won't be used again. Setting eof=true is just defensive. Fine. ### 4. `internal/llm/errors.go` IsTransient ```go // IsTransient reports whether a failed LLM round trip is worth retrying. // ... func IsTransient(err error) bool { if err == nil { return false } // The caller gave up; no retry can help. if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return false } var se *StatusError if errors.As(err, &se) { // 5xx and 429 (overloaded) are server-side and may clear; // 408 (request timeout) likewise. Every other 4xx is a client // error that retrying the same request cannot fix. return se.StatusCode == http.StatusRequestTimeout || se.StatusCode == http.StatusTooManyRequests || se.StatusCode >= 500 } var de *DecodeError if errors.As(err, &de) { return false } // Transport failures (connection refused/reset, DNS, timeouts), // mid-stream interruptions (StreamError), and anything else we // don't recognize: assume the endpoint is flaky and let the // caller's bounded retry budget decide. return true } ``` ### 5. `internal/agentrun/loop.go` Restructure with retry. Full new file: ```go package agentrun import ( "context" "fmt" "io" "math/rand" "time" "github.com/abrander/zoo/internal/llm" "github.com/abrander/zoo/internal/tools" ) // Hooks ... (add OnTurnReset) type Hooks struct { OnReasoningDelta func(delta string) OnContentDelta func(delta string) OnTurnEnd func() // OnTurnReset fires when a turn's stream fails transiently after // some of it has already been streamed, right before the turn is // retried, so the live view can discard the partial output. OnTurnReset func() OnTool func(name, arguments, result string, toolErr bool) } // Retry budget for a single LLM round trip (one request + stream // round trip), not the whole run: a run can last hours, and each turn // is independent, so a flaky endpoint should cost at most a few // seconds of backoff per turn. const ( llmMaxAttempts = 5 llmBaseDelay = time.Second llmMaxDelay = 15 * time.Second ) // llmSleep waits d, or returns early if ctx is done. Overridden in // tests. var llmSleep = func(ctx context.Context, d time.Duration) error { select { case <-time.After(d): return nil case <-ctx.Done(): return ctx.Err() } } // runLoop ... func runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) { for { if err := ctx.Err(); err != nil { return messages, err } completion, err := requestTurn(ctx, client, messages, hooks) if err != nil { return messages, err } choice := completion.Choices[0] messages = append(messages, choice.Message) if choice.FinishReason != "tool_calls" || len(choice.Message.ToolCalls) == 0 { return messages, nil } for _, tc := range choice.Message.ToolCalls { ... unchanged ... } } } // requestTurn performs one LLM round trip (request + full stream read), // retrying transient endpoint failures with exponential backoff and // jitter. It returns the completed turn's accumulated completion. func requestTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) { for attempt := 1; ; attempt++ { completion, streamed, err := streamTurn(ctx, client, messages, hooks) if err == nil { return completion, nil } // A canceled or timed-out run is not a failing endpoint. if ctxErr := ctx.Err(); ctxErr != nil { return nil, ctxErr } // Permanent failures (bad request, auth, malformed response) // and exhausted budgets fail immediately. if !llm.IsTransient(err) || attempt == llmMaxAttempts { if attempt > 1 { return nil, fmt.Errorf("llm endpoint failed after %d attempts: %w", attempt, err) } return nil, err } // The partial output of the failed attempt was already // published to the live view; discard it before the retry so // the retried turn starts clean. if streamed && hooks.OnTurnReset != nil { hooks.OnTurnReset() } if err := llmSleep(ctx, llmBackoff(attempt)); err != nil { return nil, err } } } ``` Wait, `llmBackoff(attempt)`: attempt is 1-based here (the attempt that just failed). Delays: after attempt 1 β†’ base 1s; attempt 2 β†’ 2s; attempt 3 β†’ 4s; attempt 4 β†’ 8s. So: ```go // llmBackoff returns the delay before retrying after the given failed // attempt (1-based): exponential growth from llmBaseDelay, capped at // llmMaxDelay, with equal jitter (half fixed, half random) so // concurrent agents don't retry in lockstep. func llmBackoff(attempt int) time.Duration { d := llmBaseDelay << (attempt - 1) if d > llmMaxDelay || d <= 0 { d = llmMaxDelay } return d/2 + time.Duration(rand.Int63n(int64(d/2)+1)) } ``` Go 1.26: the global rand is fine. Hmm wait, one issue: `llmBaseDelay << (attempt - 1)` β€” time.Duration is int64; 1s<<0 = 1s, 1s<<1=2s, 1s<<2=4s, 1s<<3=8s. Max attempts is 5, so the max shift is 3 (after attempt 4). No overflow. The `d <= 0` guard is for overflow in general. streamTurn: ```go // streamTurn performs one attempt at an LLM round trip: issue the // request and read the stream to completion, firing hooks as content // arrives. It reports whether the endpoint streamed any of the turn // before failing, so the caller can reset the live view before // retrying. func streamTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, bool, error) { streamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{ Messages: messages, Stream: true, Tools: tools.All(), }) if err != nil { return nil, false, fmt.Errorf("chat completion: %w", err) } var completion *llm.ChatCompletion var prevContent, prevReasoning string for { c, err := streamer.Get() if err == io.EOF { break } if err != nil { return nil, completion != nil, fmt.Errorf("stream completion: %w", err) } completion = c if len(c.Choices) > 0 { msg := c.Choices[0].Message if hooks.OnReasoningDelta != nil && len(msg.ReasoningContent) > len(prevReasoning) { hooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):]) } prevReasoning = msg.ReasoningContent if hooks.OnContentDelta != nil && len(msg.Content) > len(prevContent) { hooks.OnContentDelta(msg.Content[len(prevContent):]) } prevContent = msg.Content } } if hooks.OnTurnEnd != nil { hooks.OnTurnEnd() } if completion == nil || len(completion.Choices) == 0 { return nil, false, fmt.Errorf("model returned an empty completion") } return completion, true, nil } ``` Hmm, `streamed` β€” I said I'd track it based on content arrival. `completion != nil` means at least one chunk was received. But the first chunk might be role-only with no content. Is that "streamed" in the live-view sense? No delta was published (no content/reasoning growth), so there's nothing to reset in the live view. But `completion != nil` would be true β†’ OnTurnReset fires β†’ checkpoint + turn_reset event β†’ harmless (nothing was published… wait, was anything published? If the first chunk is role-only, no delta hook fires, so nothing was published. turn_reset + checkpoint is a no-op for viewers. Harmless.) But more precisely: streamed should mean "some delta was published." Let me track it explicitly: ```go streamed := false ... if len(msg.ReasoningContent) > len(prevReasoning) { streamed = true if hooks.OnReasoningDelta != nil { hooks.OnReasoningDelta(...) } } ... if len(msg.Content) > len(prevContent) { streamed = true ... } ``` That's precise: only reset the live view if something was actually published. Now β€” one more thing to double-check: the "empty completion" error. The stream completed cleanly (EOF via [DONE]) but no choices. `streamed` is false (no content). Transient? Not StatusError/DecodeError/ctx β†’ transient β†’ retried. Good. Wait, actually there's a subtle issue with the empty-completion case: if the server consistently returns an empty completion (e.g. the model refuses), we'll retry 5 times with ~15s of backoff and then fail. Acceptable. ### 6. `internal/agentrun/run.go` β€” streamHooks Add OnTurnReset: ```go OnTurnReset: func() { // The failed attempt's partial output was never persisted (that // happens in OnTurnEnd); drop it from the live view and the hub's // replay buffer so the retried turn starts clean. reasoningOpen, contentOpen = false, false reasoningBuf.Reset() contentBuf.Reset() r.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset}) r.hub.Checkpoint(jobID) }, ``` ### 7. `internal/livelog/hub.go` Add `TurnReset Type = "turn_reset"`. ### 8. `internal/web/templates.go` JS: handle "turn_reset": ```js case "turn_reset": // The turn's stream broke and is being retried; drop the partial // block so the retried turn starts clean. if (reasoningBody) { reasoningBody.parentElement.remove(); reasoningBody = null; } if (contentBody) { contentBody.parentElement.remove(); contentBody = null; } break; ``` `reasoningBody` is the `.block-body` div; its parent is the `.block` div. `parentElement.remove()` works. ### 9. Tests **`internal/llm/errors_test.go`** (new): - TestIsTransient: table of errors: - nil β†’ false - context.Canceled β†’ false - context.DeadlineExceeded β†’ false - &StatusError{500} β†’ true; 503 β†’ true; 429 β†’ true; 408 β†’ true; 400 β†’ false; 401 β†’ false; 404 β†’ false - &DecodeError{...} β†’ false - &StreamError{} β†’ true; &StreamError{Err: io.ErrUnexpectedEOF} β†’ true - &url.Error{Err: errors.New("connection refused")} β†’ true - wrapped: fmt.Errorf("chat completion: %w", &StatusError{500}) β†’ true; fmt.Errorf("x: %w", context.Canceled) β†’ false - plain errors.New("boom") β†’ true (default) **`internal/llm/streamer_test.go`**: add tests: - TestStreamerInterruptedStream: body with chunks but no [DONE] and clean EOF β†’ Get returns *StreamError after the last chunk… wait, let me trace: chunk 1 is parsed fine, returns completion. Get() again: scanner.Scan() β†’ false (EOF), Err() is nil, buf is nil (no data line found in this call) β†’ StreamError. - TestStreamerReadError: body that errors mid-read. Use a custom ReadCloser that returns an error after the first chunk. - TestStreamerStatusErrorType: NewStreamer with 500 β†’ errors.As *StatusError. - TestStreamerDecodeError: data line with invalid JSON β†’ *DecodeError. **`internal/agentrun/loop_test.go`** (new): use httptest: - Helper: newTestLLMClient(t, handler) *llm.Client β€” point at the httptest server. llm.NewClient(config.LLM{OpenAI: server.URL, Token: "t", Model: "m"}). - Override llmSleep to record delays and not actually sleep (or sleep for 1ms). - SSE body helper (reuse the pattern from streamer_test). Tests: 1. TestRunLoopRetriesTransientFailure: handler: first 2 requests β†’ 500 "boom"; 3rd β†’ valid SSE stream (content "hello", finish stop). Expect: runLoop returns messages with the assistant "hello", server received 3 requests, llmSleep called twice with increasing delays (>= base/2 etc.). 2. TestRunLoopNoRetryOnClientError: handler always 400. Expect: 1 request, error contains "status 400". 3. TestRunLoopGivesUpAfterMaxAttempts: handler always 500. Expect: 5 requests, error mentions "after 5 attempts". 4. TestRunLoopRetriesMidStreamBreak: handler: first request β†’ write one SSE chunk, then close the connection abruptly (hijack + close). Second β†’ valid stream. Expect success, 2 requests, OnTurnReset called once (since the first attempt streamed content). 5. TestRunLoopNoTurnResetWhenNothingStreamed: handler: first request β†’ 500 (no stream), second β†’ valid. OnTurnReset not called. 6. TestRunLoopRespectsContextDuringBackoff: handler always 500; cancel ctx after the first attempt (or pre-cancel? no β€” pre-cancel would fail at the ctx check at the top of the loop… actually, runLoop checks ctx.Err() at the top of the turn loop; a pre-canceled ctx would return immediately without any request. To test cancellation during sleep: use a real (short) llmSleep? Simpler: override llmSleep to return ctx.Err() immediately if ctx is done, and cancel ctx from a goroutine… hmm, that's racy. Alternative: have the handler cancel the ctx on the 2nd request? Let me think. Simpler approach: make llmSleep a variable that the test sets to a function that records the duration and returns ctx.Err() if ctx is done. Test: cancel ctx after the first failed request (from the handler's goroutine β€” the handler runs in the server's goroutine; cancel there). Then the 2nd request never happens; runLoop returns context.Canceled. Sequence: attempt 1 β†’ 500 β†’ transient β†’ sleep: ctx is done (handler canceled it) β†’ llmSleep returns ctx.Err() β†’ requestTurn returns it. Expect: 1 request, error is context.Canceled. Wait, but the handler runs synchronously per request; canceling ctx inside the handler before returning the 500 works: by the time runLoop reaches llmSleep, ctx is done. Deterministic. Hmm, but note: the request itself also uses ctx (http.NewRequestWithContext). The first request completes before cancellation. Fine. 7. TestRunLoopToolCallStillWorks (regression): a stream with tool_calls finish β†’ tool runs β†’ 2nd stream returns final content. Verify the tool was called. Actually, the existing code path for tool calls is unchanged, but a regression test is cheap and valuable. Hmm, tools.Call needs a tools.Context β€” let me check what tools.Context is and what Call does with, e.g., a "bash" tool β€” it runs in a sandbox via docker! That's heavy. Let me check tools.Context and see if there's a no-op context for tests. Let me look at tools/context.go and registry.go. Also, I should check how `tools.All()` and `tools.Call` work, to see if I can build a lightweight tools.Context for the test (maybe there's a null context). Let me read it. Additionally, for the SSE helper in agentrun tests, I need to produce a valid stream: `data: {...}\n\n` chunks + `data: [DONE]\n\n`. One more check β€” `llm.NewClient` uses `http.DefaultClient` β€” fine for httptest. Also, the `req2.Seed = 5`, `Temperature`, `TopP` overrides in StreamChatCompletion β€” irrelevant. Now, about `docker_integration_test.go` β€” there's an existing integration test in agentrun. Let me peek at its build tags to see if loop_test.go needs any special handling (it shouldn't β€” no docker needed). Let me read tools/context.go, registry.go, and the header of docker_integration_test.go.
Let me check the tools package (for testability of `runLoop`) and the existing integration test setup.
Tool πŸ”§ read_file {"end_line": 80, "path": "internal/tools/context.go", "start_line": 1}
{"end_line": 80, "path": "internal/tools/context.go", "start_line": 1}
{"path":"internal/tools/context.go","content":"// Package tools is the registry of functions an agent's tool-calling\n// loop can invoke, ported from ../a/tools but retargeted at a Docker\n// container (via Exec) and the Forgejo issue/PR that triggered the run\n// (via Forgejo()) instead of a local sandboxed filesystem.\npackage tools\n\nimport \"context\"\n\n// Context is bound to a single agent run (job), unlike ../a's single\n// global Context, since multiple agents run concurrently in zoo.\ntype Context interface {\n\t// Exec runs command inside the run's container via `sh -c` and\n\t// returns combined stdout+stderr.\n\tExec(ctx context.Context, command string) (string, error)\n\n\t// Forgejo returns the actions bound to the issue/PR that triggered\n\t// this run, so tools don't need to be told which repo/issue to act\n\t// on.\n\tForgejo() ForgejoActions\n}\n\n// ForgejoActions are the actions available to an agent against the\n// triggering issue/PR. Implemented by internal/agentrun, backed by\n// internal/forgejo.Client plus the run's owner/repo/issue.\ntype ForgejoActions interface {\n\tComment(body string) error\n\tOpenPullRequest(head, base, title, body string) error\n\tRequestReview(reviewers []string) error\n\tAddLabel(name string) error\n\tRemoveLabel(name string) error\n\tCloseIssue() error\n\tReopenIssue() error\n\n\t// AssignIssue hands the triggering issue/PR off to another agent by\n\t// their name (which doubles as their Forgejo username); the\n\t// resulting issue:assigned event triggers that agent's run.\n\tAssignIssue(agentName string) error\n\n\t// SubmitReview submits a review on the triggering pull request with\n\t// the given verdict (forgejo.ReviewStateApproved,\n\t// ReviewStateChangesRequest, or ReviewStateComment) and body.\n\tSubmitReview(state, body string) error\n}","start_line":1,"end_line":43,"total_lines":43,"truncated":false}
Tool πŸ”§ read_file {"end_line": 60, "path": "internal/tools/registry.go", "start_line": 1}
{"end_line": 60, "path": "internal/tools/registry.go", "start_line": 1}
{"path":"internal/tools/registry.go","content":"package tools\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\n\t\"github.com/abrander/zoo/internal/llm\"\n)\n\ntype toolFunc[P any] func(Context, P) (string, error)\n\ntype tool struct {\n\ttool llm.Tool\n\tcall func(ctx Context, args string) (string, error)\n}\n\nvar registry = map[string]tool{}\n\n// All lists all known tools.\nfunc All() []llm.Tool {\n\tts := make([]llm.Tool, 0, len(registry))\n\n\tfor _, t := range registry {\n\t\tts = append(ts, t.tool)\n\t}\n\n\treturn ts\n}\n\n// Register registers a tool function with the given definition. The\n// registry is a stateless name-\u003efunc map shared across concurrent runs;\n// all per-run state lives in the Context passed to Call.\nfunc Register[P any](def llm.Tool, fun toolFunc[P]) {\n\tif def.Type != \"function\" {\n\t\tpanic(\"tool must be a function\")\n\t}\n\n\tif def.Function.Name == \"\" {\n\t\tpanic(\"tool function must have a name\")\n\t}\n\n\tif _, found := registry[def.Function.Name]; found {\n\t\tpanic(fmt.Sprintf(\"tool with name %q already registered\", def.Function.Name))\n\t}\n\n\tregistry[def.Function.Name] = tool{\n\t\ttool: def,\n\t\tcall: func(ctx Context, args string) (string, error) {\n\t\t\tvar params P\n\n\t\t\tif err := json.Unmarshal([]byte(args), \u0026params); err != nil {\n\t\t\t\treturn \"\", err\n\t\t\t}\n\n\t\t\treturn fun(ctx, params)\n\t\t},\n\t}\n}\n\n// Call invokes a registered tool by name with the given (raw JSON)","start_line":1,"end_line":60,"total_lines":69,"truncated":true}
Tool πŸ”§ read_file {"end_line": 40, "path": "internal/agentrun/docker_integration_test.go", "start_line": 1}
{"end_line": 40, "path": "internal/agentrun/docker_integration_test.go", "start_line": 1}
{"path":"internal/agentrun/docker_integration_test.go","content":"//go:build docker\n\n// Run with: go test -tags docker ./internal/agentrun/...\n// Requires a reachable Docker daemon (DOCKER_HOST or the default socket).\npackage agentrun\n\nimport (\n\t\"context\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n)\n\nfunc TestDockerRuntimeExecRoundTrip(t *testing.T) {\n\trt, err := newDockerRuntime()\n\tif err != nil {\n\t\tt.Fatalf(\"docker client: %v\", err)\n\t}\n\n\tctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)\n\tdefer cancel()\n\n\tcontainerID, err := rt.createContainer(ctx, \"debian:unstable\", nil, \"zoo-test-run\")\n\tif err != nil {\n\t\tt.Fatalf(\"create container: %v\", err)\n\t}\n\tdefer rt.remove(context.Background(), containerID)\n\n\toutput, exitCode, err := rt.exec(ctx, containerID, \"echo hello-from-zoo\")\n\tif err != nil {\n\t\tt.Fatalf(\"exec: %v\", err)\n\t}\n\tif exitCode != 0 {\n\t\tt.Fatalf(\"expected exit code 0, got %d\", exitCode)\n\t}\n\tif !strings.Contains(output, \"hello-from-zoo\") {\n\t\tt.Fatalf(\"unexpected output: %q\", output)","start_line":1,"end_line":40,"total_lines":224,"truncated":true}
Thinking
`tools.Context` is an interface β€” I can create a fake in the loop tests. `tools.Call(toolsCtx, tc.FunctionCall)` β€” for the tool-call regression test, I need a registered tool whose execution doesn't require docker. All registered tools use ctx.Exec (docker). My fake Context can implement Exec as a stub that returns a fixed string. Let me check the rest of `Call` and one tool to confirm the parameter format. Actually, I'll just pick a tool like "list_files" β€” let me quickly check bash.go or list_files.go to confirm the parameter struct. Actually, for the regression test, the simplest thing is: use the `comment` tool (ForgejoActions) β€” my fake can implement ForgejoActions with a stub. Let me check the signature of the comment tool in tools/forgejo.go.
Tool πŸ”§ read_file {"end_line": 69, "path": "internal/tools/registry.go", "start_line": 56}
{"end_line": 69, "path": "internal/tools/registry.go", "start_line": 56}
{"path":"internal/tools/registry.go","content":"\t\t},\n\t}\n}\n\n// Call invokes a registered tool by name with the given (raw JSON)\n// arguments, against the given run's Context.\nfunc Call(ctx Context, fcall llm.FunctionCall) (string, error) {\n\tt, found := registry[fcall.Name]\n\tif !found {\n\t\treturn \"\", fmt.Errorf(\"unknown function: %s\", fcall.Name)\n\t}\n\n\treturn t.call(ctx, fcall.Arguments)\n}","start_line":56,"end_line":69,"total_lines":69,"truncated":false}
Tool πŸ”§ read_file {"end_line": 60, "path": "internal/tools/forgejo.go", "start_line": 1}
{"end_line": 60, "path": "internal/tools/forgejo.go", "start_line": 1}
{"path":"internal/tools/forgejo.go","content":"package tools\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/abrander/zoo/internal/llm\"\n)\n\ntype commentParams struct {\n\tBody string `json:\"body\"`\n}\n\nfunc init() {\n\ttool := llm.NewTool(\n\t\t\"comment\",\n\t\t\"Post a comment on the issue or pull request that triggered this run.\")\n\n\ttool.AddStringProperty(\"body\", \"The comment text, in Markdown\", true)\n\n\tRegister(tool, comment)\n}\n\nfunc comment(ctx Context, params commentParams) (string, error) {\n\tif err := ctx.Forgejo().Comment(params.Body); err != nil {\n\t\treturn \"\", err\n\t}\n\n\treturn \"Comment posted\", nil\n}\n\ntype openPullRequestParams struct {\n\tHead  string `json:\"head\"`\n\tBase  string `json:\"base\"`\n\tTitle string `json:\"title\"`\n\tBody  string `json:\"body\"`\n}\n\nfunc init() {\n\ttool := llm.NewTool(\n\t\t\"open_pull_request\",\n\t\t\"Open a pull request from a branch you've pushed into the repository's default branch (or another base branch).\")\n\n\ttool.AddStringProperty(\"head\", \"The branch containing your changes (usually the run's own branch)\", true)\n\ttool.AddStringProperty(\"base\", \"The branch to merge into, e.g. the repository's default branch\", true)\n\ttool.AddStringProperty(\"title\", \"The pull request title\", true)\n\ttool.AddStringProperty(\"body\", \"The pull request description, in Markdown\", true)\n\n\tRegister(tool, openPullRequest)\n}\n\nfunc openPullRequest(ctx Context, params openPullRequestParams) (string, error) {\n\tif err := ctx.Forgejo().OpenPullRequest(params.Head, params.Base, params.Title, params.Body); err != nil {\n\t\treturn \"\", err\n\t}\n\n\treturn \"Pull request opened\", nil\n}\n\ntype requestReviewParams struct {","start_line":1,"end_line":60,"total_lines":204,"truncated":true}
Thinking
All the information I need is in place. Now let's write the implementation. Plan: 1. `internal/llm/errors.go` (new): `StatusError`, `StreamError`, `DecodeError`, `IsTransient`. 2. `internal/llm/client.go`: `NewStreamer` returns `*StatusError` (body capped). 3. `internal/llm/streamer.go`: proper scanner error handling; `StreamError` on interruption; `DecodeError` on bad JSON. 4. `internal/llm/errors_test.go` (new): classification tests + streamer error type tests. 5. `internal/agentrun/loop.go`: retry loop (`requestTurn`, `streamTurn`, backoff, `llmSleep` variable), `Hooks.OnTurnReset`. 6. `internal/agentrun/run.go`: `OnTurnReset` in streamHooks. 7. `internal/livelog/hub.go`: `TurnReset` event type. 8. `internal/web/templates.go`: JS handling for `turn_reset`. 9. `internal/agentrun/loop_test.go` (new): httptest-based retry tests. Let's write the code. First, `internal/llm/errors.go`: ```go package llm import ( "context" "errors" "fmt" "net/http" ) // The client classifies failures into a small set of typed errors so // callers can decide what is worth retrying (see IsTransient): // // - StatusError: the endpoint answered with a non-200 status. // - StreamError: the response body broke or ended before the // stream's [DONE] marker, i.e. the connection dropped mid-turn. // - DecodeError: the endpoint sent a data line that is not valid // JSON, i.e. a protocol violation rather than a transient fault. // // Unrecognized errors (transport failures like connection refused or // reset, DNS errors, timeouts) are treated as transient by default: // when in doubt, the endpoint is assumed flaky and the caller's // bounded retry budget decides. // StatusError is returned when the endpoint answers with a non-200 // status. type StatusError struct { StatusCode int Body string } func (e *StatusError) Error() string { return fmt.Sprintf("llm endpoint returned status %d: %s", e.StatusCode, e.Body) } // StreamError is returned when the stream breaks or ends before its // [DONE] marker. Err is the underlying read error, if the body broke // with one; a nil Err means the body ended cleanly without [DONE]. type StreamError struct { Err error } func (e *StreamError) Error() string { if e.Err != nil { return fmt.Sprintf("llm stream interrupted: %v", e.Err) } return "llm stream ended before [DONE]" } func (e *StreamError) Unwrap() error { return e.Err } // DecodeError is returned when a data line is not valid JSON. type DecodeError struct { Err error Chunk string } func (e *DecodeError) Error() string { return fmt.Sprintf("decode completion chunk: %v: %s", e.Err, e.Chunk) } func (e *DecodeError) Unwrap() error { return e.Err } // IsTransient reports whether a failed LLM round trip is worth // retrying. Transport failures, 5xx/429/408 responses, and mid-stream // interruptions are transient: the endpoint may be down or overloaded, // and the same request may succeed later. Client errors (other 4xx) // and malformed responses are not: retrying the same request cannot // fix them. Context cancellation is never transient β€” the caller gave // up. func IsTransient(err error) bool { if err == nil { return false } if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return false } var se *StatusError if errors.As(err, &se) { return se.StatusCode == http.StatusRequestTimeout || se.StatusCode == http.StatusTooManyRequests || se.StatusCode >= 500 } var de *DecodeError if errors.As(err, &de) { return false } return true } ``` Next, `client.go` β€” NewStreamer: ```go // maxStatusBody caps how much of a non-200 response body is kept for // the error message. const maxStatusBody = 1024 func NewStreamer(resp *http.Response) (*Streamer, error) { if resp.StatusCode != http.StatusOK { body, _ := io.ReadAll(io.LimitReader(resp.Body, maxStatusBody)) resp.Body.Close() return nil, &StatusError{StatusCode: resp.StatusCode, Body: string(body)} } return &Streamer{ resp: resp, scanner: bufio.NewScanner(resp.Body), }, nil } ``` `streamer.go` Get(): ```go // Get returns the accumulated completion after merging in the next SSE // chunk. Callers should keep calling Get until it returns io.EOF. func (s *Streamer) Get() (*ChatCompletion, error) { if s.eof { return nil, io.EOF } prefix := []byte("data: ") var buf []byte for s.scanner.Scan() { buf = bytes.TrimSpace(s.scanner.Bytes()) if len(buf) == 0 { continue } if bytes.HasPrefix(buf, prefix) { break } } if err := s.scanner.Err(); err != nil { return nil, &StreamError{Err: err} } buf = bytes.TrimPrefix(buf, prefix) if len(buf) == 0 { // The body ended before any data line arrived: the stream was // cut off before [DONE]. s.resp.Body.Close() s.eof = true return nil, &StreamError{} } if bytes.Equal(buf, []byte("[DONE]")) { s.resp.Body.Close() s.eof = true return nil, io.EOF } var completion ChatCompletion // Unknown fields are tolerated on purpose: servers (e.g. newer // llama.cpp builds) may add fields we don't know about yet, and // failing on them would break compatibility. if err := json.Unmarshal(buf, &completion); err != nil { return nil, &DecodeError{Err: err, Chunk: string(buf)} } s.completion.Merge(&completion) return &s.completion, nil } ``` Wait β€” one edge case: if the scanner finds a data line and breaks, buf is that line. If the loop exits via EOF, buf is the last non-empty non-data line (or nil). But there's a subtle case: if the last line of the body is a data line but the body ends without a trailing newline β€” Scan() still returns it (bufio.Scanner returns the final token even without a trailing newline). OK. Another subtle case: a data line is found, and then... no, we break on the first data line. Each Get() call processes exactly one data line. OK. Hmm, wait β€” one more: if a non-data line (e.g. `event: x`) is the last line, and then EOF: buf = "event: x", no scanner error, buf is non-empty, not [DONE] β†’ json.Unmarshal("event: x") β†’ DecodeError! But this is a stream that ended without [DONE] β€” it should be a StreamError, not a DecodeError. Hmm. My `len(buf) == 0` check only catches the case where there was no trailing non-data line. Let me restructure: track whether we found a data line: ```go var buf []byte var found bool for s.scanner.Scan() { line := bytes.TrimSpace(s.scanner.Bytes()) if len(line) == 0 { continue } if bytes.HasPrefix(line, prefix) { buf = bytes.TrimPrefix(line, prefix) found = true break } } if err := s.scanner.Err(); err != nil { return nil, &StreamError{Err: err} } if !found { // The body ended before a data line arrived: the stream was cut // off before [DONE]. s.resp.Body.Close() s.eof = true return nil, &StreamError{} } if bytes.Equal(buf, []byte("[DONE]")) { s.resp.Body.Close() s.eof = true return nil, io.EOF } var completion ChatCompletion if err := json.Unmarshal(buf, &completion); err != nil { return nil, &DecodeError{Err: err, Chunk: string(buf)} } ``` This is cleaner and correct: - Found a data line β†’ either [DONE] or JSON (DecodeError if malformed). - No data line + read error β†’ StreamError{Err}. - No data line + clean EOF β†’ StreamError{} (interrupted). Now the agentrun loop. Full new loop.go: ```go package agentrun import ( "context" "fmt" "io" "math/rand" "time" "github.com/abrander/zoo/internal/llm" "github.com/abrander/zoo/internal/tools" ) // Hooks lets a caller observe a runLoop invocation live, as the model // streams and tool calls execute, in addition to the final []llm.Message // it returns. Any of these may be nil. type Hooks struct { // OnReasoningDelta and OnContentDelta fire with just the newly // streamed text for the current turn, not the accumulated total. OnReasoningDelta func(delta string) OnContentDelta func(delta string) // OnTurnEnd fires once per completed streamer round-trip, after the // model's message for that turn is fully received and before any of // its tool calls run. OnTurnEnd func() // OnTurnReset fires when a turn's stream fails transiently after // some of it has already been streamed, right before the turn is // retried, so the live view can discard the partial output (which // was never persisted). OnTurnReset func() // OnTool fires once per tool call, after it has run. OnTool func(name, arguments, result string, toolErr bool) } // The retry budget for one LLM round trip (request + full stream read), // not for the whole run: a run can last hours and each turn is // independent, so a failing endpoint should cost at most a few seconds // of backoff per turn before the run surfaces the error. const ( llmMaxAttempts = 5 llmBaseDelay = time.Second llmMaxDelay = 15 * time.Second ) // llmSleep waits d, returning early with ctx's error if the run is // canceled or times out while waiting. Overridden in tests. var llmSleep = func(ctx context.Context, d time.Duration) error { select { case <-time.After(d): return nil case <-ctx.Done(): return ctx.Err() } } // runLoop is a headless port of ../a's App.generate(): send messages + // tool defs, get a completion, run any tool_calls and append their // results, repeat until a plain finish or ctx is done. func runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) { for { if err := ctx.Err(); err != nil { return messages, err } completion, err := requestTurn(ctx, client, messages, hooks) if err != nil { return messages, err } choice := completion.Choices[0] messages = append(messages, choice.Message) if choice.FinishReason != "tool_calls" || len(choice.Message.ToolCalls) == 0 { return messages, nil } for _, tc := range choice.Message.ToolCalls { if err := ctx.Err(); err != nil { return messages, err } result, err := tools.Call(toolsCtx, tc.FunctionCall) toolErr := err != nil if toolErr { result = fmt.Sprintf("ERROR CALLING TOOL: %s", err.Error()) } if hooks.OnTool != nil { hooks.OnTool(tc.FunctionCall.Name, tc.FunctionCall.Arguments, result, toolErr) } messages = append(messages, llm.Message{ Role: "tool", Content: result, ToolCallID: tc.Id, Name: tc.FunctionCall.Name, }) } } } // requestTurn performs one LLM round trip (request + full stream // read), retrying transient endpoint failures (connection errors, // 5xx/429 responses, mid-stream breaks) with exponential backoff and // jitter. It returns the completed turn's accumulated completion. func requestTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) { for attempt := 1; ; attempt++ { completion, streamed, err := streamTurn(ctx, client, messages, hooks) if err == nil { return completion, nil } // A canceled or timed-out run is not a failing endpoint; stop // and report the run's own error. if ctxErr := ctx.Err(); ctxErr != nil { return nil, ctxErr } // Permanent failures (client errors, malformed responses) and // exhausted budgets fail immediately. if !llm.IsTransient(err) || attempt == llmMaxAttempts { if attempt > 1 { return nil, fmt.Errorf("llm endpoint failed after %d attempts: %w", attempt, err) } return nil, err } // The failed attempt's partial output was already published to // the live view; discard it so the retried turn starts clean. if streamed && hooks.OnTurnReset != nil { hooks.OnTurnReset() } if err := llmSleep(ctx, llmBackoff(attempt)); err != nil { return nil, err } } } // llmBackoff returns the delay before retrying after the given failed // attempt (1-based): exponential growth from llmBaseDelay, capped at // llmMaxDelay, with equal jitter (half fixed, half random) so // concurrent agents don't retry in lockstep. func llmBackoff(attempt int) time.Duration { d := llmBaseDelay << (attempt - 1) if d <= 0 || d > llmMaxDelay { d = llmMaxDelay } return d/2 + time.Duration(rand.Int63n(int64(d/2)+1)) } // streamTurn performs one attempt at an LLM round trip: issue the // request and read the stream to completion, firing hooks as content // arrives. It reports whether the endpoint streamed any of the turn // before failing, so the caller can reset the live view before // retrying. func streamTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, bool, error) { streamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{ Messages: messages, Stream: true, Tools: tools.All(), }) if err != nil { return nil, false, fmt.Errorf("chat completion: %w", err) } var completion *llm.ChatCompletion streamed := false var prevContent, prevReasoning string for { c, err := streamer.Get() if err == io.EOF { break } if err != nil { return nil, streamed, fmt.Errorf("stream completion: %w", err) } completion = c if len(c.Choices) > 0 { msg := c.Choices[0].Message if len(msg.ReasoningContent) > len(prevReasoning) { streamed = true if hooks.OnReasoningDelta != nil { hooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):]) } } prevReasoning = msg.ReasoningContent if len(msg.Content) > len(prevContent) { streamed = true if hooks.OnContentDelta != nil { hooks.OnContentDelta(msg.Content[len(prevContent):]) } } prevContent = msg.Content } } if hooks.OnTurnEnd != nil { hooks.OnTurnEnd() } if completion == nil || len(completion.Choices) == 0 { return nil, false, fmt.Errorf("model returned an empty completion") } return completion, true, nil } ``` Wait, there's a problem with the error wrapping in streamTurn: previously the request error was wrapped as "chat completion: %w" and the stream error as "stream completion: %w". I'm keeping that. But note: the StatusError from NewStreamer comes out of StreamChatCompletion β†’ wrapped with "chat completion: %w" β†’ errors.As still finds the *StatusError. Good. StreamError/DecodeError from Get() β†’ wrapped with "stream completion: %w" β†’ errors.As finds them. Good. One more check β€” `llmBackoff` uses the global `rand.Int63n`. In Go 1.26, the global rand is auto-seeded and goroutine-safe. Good. Hmm, one design question: `streamed` β€” I'm setting it when content/reasoning grows. But what about tool-call argument deltas? If the model streams a tool call (arguments) and then the stream breaks, no content/reasoning delta is published (tool calls aren't streamed live to the dashboard until they execute). So there's nothing to reset in the live view. `streamed=false` β†’ no OnTurnReset. Correct behavior! Next, streamHooks in run.go β€” add OnTurnReset. The current Hooks literal: ```go return Hooks{ OnReasoningDelta: func(delta string) {...}, OnContentDelta: func(delta string) {...}, OnTurnEnd: func() {...}, OnTool: func(...) {...}, } ``` Add: ```go OnTurnReset: func() { // The failed attempt's partial output was never persisted (that // happens in OnTurnEnd), so drop it from the live view and the // hub's replay buffer: the retried turn regenerates it. reasoningOpen, contentOpen = false, false reasoningBuf.Reset() contentBuf.Reset() r.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset}) r.hub.Checkpoint(jobID) }, ``` livelog: add `TurnReset Type = "turn_reset"`. JS in templates.go: ```js case "turn_reset": // The turn's stream broke and is being retried; drop the partial // block so the retried turn starts clean. if (reasoningBody) { reasoningBody.parentElement.remove(); reasoningBody = null; } if (contentBody) { contentBody.parentElement.remove(); contentBody = null; } break; ``` Now the tests. **internal/llm/errors_test.go**: ```go package llm import ( "context" "errors" "fmt" "io" "net/http" "net/url" "testing" ) func TestIsTransient(t *testing.T) { cases := []struct { name string err error want bool }{ {"nil", nil, false}, {"context canceled", context.Canceled, false}, {"deadline exceeded", context.DeadlineExceeded, false}, {"wrapped canceled", fmt.Errorf("stream completion: %w", context.Canceled), false}, {"status 500", &StatusError{StatusCode: 500}, true}, {"status 503", &StatusError{StatusCode: 503}, true}, {"status 429", &StatusError{StatusCode: 429}, true}, {"status 408", &StatusError{StatusCode: 408}, true}, {"status 400", &StatusError{StatusCode: 400}, false}, {"status 401", &StatusError{StatusCode: 401}, false}, {"status 404", &StatusError{StatusCode: 404}, false}, {"wrapped status 500", fmt.Errorf("chat completion: %w", &StatusError{StatusCode: 500}), true}, {"wrapped status 400", fmt.Errorf("chat completion: %w", &StatusError{StatusCode: 400}), false}, {"decode error", &DecodeError{Err: errors.New("bad json"), Chunk: "{"}, false}, {"wrapped decode error", fmt.Errorf("stream completion: %w", &DecodeError{Err: errors.New("bad json")}), false}, {"stream error", &StreamError{}, true}, {"stream error with read error", &StreamError{Err: io.ErrUnexpectedEOF}, true}, {"transport error", &url.Error{Op: "Post", URL: "http://x", Err: errors.New("connection refused")}, true}, {"unknown error", errors.New("boom"), true}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { if got := IsTransient(tc.err); got != tc.want { t.Errorf("IsTransient(%v) = %v, want %v", tc.err, got, tc.want) } }) } } ``` Additions to streamer_test.go: ```go func TestStreamerReturnsStatusError(t *testing.T) { resp := newTestResponse("boom") resp.StatusCode = http.StatusServiceUnavailable _, err := NewStreamer(resp) var se *StatusError if !errors.As(err, &se) { t.Fatalf("expected *StatusError, got %T: %v", err, err) } if se.StatusCode != http.StatusServiceUnavailable { t.Errorf("expected status 503, got %d", se.StatusCode) } } func TestStreamerCleanEndWithoutDoneIsStreamError(t *testing.T) { sse := sseChunk(t, Message{Content: "Hel", Role: "assistant"}, "") s, err := NewStreamer(newTestResponse(sse)) if err != nil { t.Fatal(err) } // First chunk parses fine... if _, err := s.Get(); err != nil { t.Fatalf("first chunk: %v", err) } // ...but the stream ends without [DONE]. _, err = s.Get() var se *StreamError if !errors.As(err, &se) { t.Fatalf("expected *StreamError, got %T: %v", err, err) } if !IsTransient(err) { t.Error("expected interrupted stream to be transient") } } func TestStreamerReadErrorIsStreamError(t *testing.T) { sse := sseChunk(t, Message{Content: "Hel", Role: "assistant"}, "") s, err := NewStreamer(&http.Response{ StatusCode: http.StatusOK, Body: &errAfterFirstRead{r: strings.NewReader(sse)}, }) if err != nil { t.Fatal(err) } if _, err := s.Get(); err != nil { t.Fatalf("first chunk: %v", err) } _, err = s.Get() var se *StreamError if !errors.As(err, &se) { t.Fatalf("expected *StreamError, got %T: %v", err, err) } } // errAfterFirstRead serves r's content, then fails every subsequent // read, simulating a connection that drops mid-stream. type errAfterFirstRead struct { r *strings.Reader once bool } func (e *errAfterFirstRead) Read(p []byte) (int, error) { if e.once { return 0, errors.New("connection reset by peer") } e.once = true return e.r.Read(p) } func (e *errAfterFirstRead) Close() error { return nil } ``` Hmm, wait β€” errAfterFirstRead: the first Read serves the entire sse content (strings.Reader.Read may return all of it). The scanner buffers... bufio.Scanner reads with its own buffer (default 4096). The first scanner.Scan() call triggers a Read that grabs everything. Then the second Get() β†’ scanner.Scan() β†’ the buffer is exhausted β†’ Read β†’ error. Good. But wait: if the first Read returns all the content including the trailing "\n\n" of the chunk... the scanner will tokenize: line 1 = "data: {...}", line 2 = "" (empty). First Get(): Scan β†’ "data: {...}" β†’ break. Second Get(): Scan β†’ "" β†’ continue β†’ Scan β†’ Read β†’ error β†’ Scan returns false, Err() = "connection reset by peer" β†’ StreamError{Err}. Also a decode error test: ```go func TestStreamerBadChunkIsDecodeError(t *testing.T) { s, err := NewStreamer(newTestResponse("data: {not json}\n\ndata: [DONE]\n\n")) if err != nil { t.Fatal(err) } _, err = s.Get() var de *DecodeError if !errors.As(err, &de) { t.Fatalf("expected *DecodeError, got %T: %v", err, err) } if IsTransient(err) { t.Error("expected malformed chunk to not be transient") } } ``` **internal/agentrun/loop_test.go**: ```go package agentrun import ( "context" "fmt" "net/http" "net/http/httptest" "strings" "sync" "testing" "time" "github.com/abrander/zoo/internal/config" "github.com/abrander/zoo/internal/llm" ) // fakeToolsCtx is a tools.Context that records calls instead of // touching Docker or Forgejo. type fakeToolsCtx struct { mu sync.Mutex executed []string } func (f *fakeToolsCtx) Exec(ctx context.Context, command string) (string, error) { f.mu.Lock() defer f.mu.Unlock() f.executed = append(f.executed, command) return "ok", nil } func (f *fakeToolsCtx) Forgejo() tools.ForgejoActions { return fakeForgejo{} } ``` Hmm, ForgejoActions is a big interface. Implementing all the methods is verbose. Alternative: use the `comment` tool in the tool-call test and implement a fake ForgejoActions with all methods as no-ops. Let me count the methods: Comment, OpenPullRequest, RequestReview, AddLabel, RemoveLabel, CloseIssue, ReopenIssue, AssignIssue, SubmitReview. 9 methods. Verbose but fine. Actually β€” do I even need a tool-call regression test? The tool-call path in runLoop is unchanged. But the restructuring (extracting streamTurn) touches it. A regression test that verifies a tool-call turn β†’ tool execution β†’ second turn β†’ final answer is worthwhile. Let me include it with a no-op ForgejoActions. SSE helper for the agentrun tests: ```go func sseBody(chunks ...string) string { var b strings.Builder for _, c := range chunks { b.WriteString("data: " + c + "\n\n") } b.WriteString("data: [DONE]\n\n") return b.String() } ``` And chunk JSON: ```go func contentChunk(content, finish string) string { return fmt.Sprintf(`{"choices":[{"index":0,"delta":{"role":"assistant","content":%q},"finish_reason":%q}]}`, content, finish) } ``` Watch out for finish_reason "" β€” in JSON, `"finish_reason":""` is fine (the streamer tolerates it). For a tool-call chunk: ```go func toolCallChunk(id, name, args, finish string) string { return fmt.Sprintf(`{"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":%q,"type":"function","function":{"name":%q,"arguments":%q}}]},"finish_reason":%q}]}`, id, name, args, finish) } ``` Test client: ```go func testLLMClient(t *testing.T, handler http.HandlerFunc) *llm.Client { t.Helper() srv := httptest.NewServer(handler) t.Cleanup(srv.Close) return llm.NewClient(config.LLM{OpenAI: srv.URL, Token: "test", Model: "test-model"}) } ``` And a sleep override helper: ```go func withFakeSleep(t *testing.T, sleep func(ctx context.Context, d time.Duration) error) { t.Helper() old := llmSleep llmSleep = sleep t.Cleanup(func() { llmSleep = old }) } ``` Tests: 1. TestRunLoopRetriesTransientStatus: ```go func TestRunLoopRetriesTransientStatus(t *testing.T) { var mu sync.Mutex requests := 0 var delays []time.Duration client := testLLMClient(t, func(w http.ResponseWriter, r *http.Request) { mu.Lock() requests++ n := requests mu.Unlock() if n <= 2 { w.WriteHeader(http.StatusInternalServerError) fmt.Fprint(w, "endpoint down") return } w.Header().Set("Content-Type", "text/event-stream") fmt.Fprint(w, sseBody(contentChunk("hello", "stop"))) }) withFakeSleep(t, func(ctx context.Context, d time.Duration) error { mu.Lock() delays = append(delays, d) mu.Unlock() return nil }) msgs, err := runLoop(context.Background(), client, &fakeToolsCtx{}, []llm.Message{{Role: "user", Content: "hi"}}, Hooks{}) if err != nil { t.Fatalf("runLoop: %v", err) } ... if requests != 3 { t.Errorf(...) } if len(delays) != 2 { ... } // Backoff must grow and stay within [d/2, d] of the exponential schedule. if delays[0] < time.Second/2 || delays[0] > time.Second { ... } if delays[1] < time.Second || delays[1] > 2*time.Second { ... } // Final message is the assistant's "hello". last := msgs[len(msgs)-1] if last.Role != "assistant" || last.Content != "hello" { ... } } ``` 2. TestRunLoopNoRetryOnClientError: ```go client := testLLMClient(t, func(w, r) { requests++; w.WriteHeader(400); fmt.Fprint(w, "bad request") }) _, err := runLoop(...) if requests != 1 { ... } var se *llm.StatusError if !errors.As(err, &se) || se.StatusCode != 400 { ... } ``` 3. TestRunLoopGivesUpAfterMaxAttempts: ```go client always 500. _, err := runLoop(...) if requests != llmMaxAttempts { ... } if !strings.Contains(err.Error(), fmt.Sprintf("after %d attempts", llmMaxAttempts)) { ... } ``` 4. TestRunLoopRetriesMidStreamBreak: ```go client := testLLMClient(t, func(w http.ResponseWriter, r *http.Request) { mu.Lock() requests++ n := requests mu.Unlock() if n == 1 { // Stream one chunk, then drop the connection. w.Header().Set("Content-Type", "text/event-stream") fmt.Fprint(w, sseChunk content "Hel") w.(http.Flusher).Flush() hj, ok := w.(http.Hijacker) if !ok { t.Error("no hijacker") } conn, _, err := hj.Hijack() if err == nil { conn.Close() } return } w.Header().Set("Content-Type", "text/event-stream") fmt.Fprint(w, sseBody(contentChunk("hello", "stop"))) }) resetCount := 0 hooks := Hooks{OnTurnReset: func() { resetCount++ }} msgs, err := runLoop(...) // success, requests == 2, resetCount == 1 ``` Hmm, wait β€” Hijack after WriteHeader + write + flush: the client reads "data: {...}\n\n" and then the connection is closed. The client's first Get() parses the chunk. Second Get(): scanner.Scan() β†’ Read β†’ EOF (clean close, no error!). Hmm β€” a clean close via conn.Close() sends a FIN β†’ the client sees a clean EOF, not a read error β†’ StreamError{} (no [DONE]). That's still a StreamError β†’ transient. But wait β€” does the client see a clean EOF, or a "connection reset"? A FIN after data β†’ clean EOF. Either way, StreamError. Good. Actually, hmm, there's a subtlety with Hijack: after Hijack, the server no longer manages the connection. Writing before the hijack is fine. Also, the httptest server's handler returns after the hijack β€” that's fine. But actually, is Hijack even necessary? A simpler alternative for a mid-stream break: write a chunk and then... if the handler just returns, the body is closed cleanly β†’ the client sees a clean EOF without [DONE] β†’ StreamError (my new code). That's exactly the "endpoint died mid-stream" case (well, a clean death, but). So I don't need Hijack at all! Just: ```go if n == 1 { w.Header().Set("Content-Type", "text/event-stream") fmt.Fprint(w, "data: {...}\n\n") w.(http.Flusher).Flush() return // body ends without [DONE] } ``` Wait, but does the client see the flushed chunk before the handler returns? The client's first Get() reads the buffered data β€” the chunk is in the TCP buffer, so yes, it reads it. Then the handler returns β†’ connection close β†’ the client's next Read β†’ EOF. Hmm, but there's a race: the client might read everything (chunk + EOF) in one go. First Get(): scanner.Scan() grabs the chunk line β†’ break β†’ returns the completion. Second Get(): Scan β†’ false (EOF), Err is nil β†’ StreamError{}. Deterministic enough. Actually, wait, one more subtlety: does the client's first Get() happen before the server's handler returns? The client blocks on the first Get() until data arrives. The server writes the chunk and flushes, then returns (closes). The client's first Get() gets the chunk (the data is available). Whether the FIN has arrived by the time of the second Get() doesn't matter β€” the scanner will eventually see EOF. Good. But hmm β€” one more: if the server closes immediately, could the client's first Read return the chunk and the FIN together, and the scanner's first Scan() call... no, Scan() returns one token at a time. The first Scan returns the "data: ..." line. No problem. To be safe against a "connection reset" (RST) instead of a clean FIN, both paths yield a StreamError. 5. TestRunLoopNoTurnResetWhenNothingStreamed: ```go // First request: 500 (nothing streamed). Second: success. resetCount should be 0. ``` 6. TestRunLoopContextCanceledDuringBackoff: ```go ctx, cancel := context.WithCancel(context.Background()) client := testLLMClient(t, func(w, r) { mu.Lock(); requests++; n := requests; mu.Unlock() if n == 1 { cancel() w.WriteHeader(500) fmt.Fprint(w, "down") return } ... }) _, err := runLoop(ctx, client, &fakeToolsCtx{}, msgs, Hooks{}) if requests != 1 { ... } if !errors.Is(err, context.Canceled) { ... } ``` Wait β€” trace: attempt 1 β†’ 500 (cancel() is called in the handler). streamTurn returns an error. requestTurn: ctx.Err() β†’ Canceled β†’ returns ctx.Err(). requests == 1. err is context.Canceled. But wait β€” the HTTP request itself uses ctx (NewRequestWithContext). The first request is issued before the cancel (the handler runs, then cancels). No problem. 7. TestRunLoopToolCallRegression: ```go // Turn 1: tool_calls finish with a "comment" call. Turn 2: final content. var mu sync.Mutex n := 0 client := testLLMClient(t, func(w, r) { mu.Lock(); n++; i := n; mu.Unlock() w.Header().Set("Content-Type", "text/event-stream") if i == 1 { fmt.Fprint(w, sseBody(toolCallChunk("call1", "comment", `{"body":"hi"}`, "tool_calls"))) } else { fmt.Fprint(w, sseBody(contentChunk("done", "stop"))) } }) toolsCtx := &fakeToolsCtx{} hooks := Hooks{OnTool: func(name, args, result string, toolErr bool) { toolName = name; toolResult = result }} msgs, err := runLoop(...) // err is nil, n == 2, toolName == "comment", toolResult == "Comment posted" // messages: user, assistant(tool_calls), tool, assistant("done") ``` fakeToolsCtx.Forgejo() must return a ForgejoActions whose Comment works: ```go type fakeForgejo struct{ mu *sync.Mutex; comments []string } func (f fakeForgejo) Comment(body string) error { ... return nil } ... all other methods: return nil ``` Let me write it as a struct with a pointer to shared state: ```go type fakeForgejoActions struct { mu *sync.Mutex comments []string } func (f fakeForgejoActions) Comment(body string) error { f.mu.Lock() defer f.mu.Unlock() f.comments = append(f.comments, body) return nil } func (f fakeForgejoActions) OpenPullRequest(head, base, title, body string) error { return nil } func (f fakeForgejoActions) RequestReview(reviewers []string) error { return nil } func (f fakeForgejoActions) AddLabel(name string) error { return nil } func (f fakeForgejoActions) RemoveLabel(name string) error { return nil } func (f fakeForgejoActions) CloseIssue() error { return nil } func (f fakeForgejoActions) ReopenIssue() error { return nil } func (f fakeForgejoActions) AssignIssue(agentName string) error { return nil } func (f fakeForgejoActions) SubmitReview(state, body string) error { return nil } ``` And fakeToolsCtx: ```go type fakeToolsCtx struct { mu sync.Mutex comments []string } func (f *fakeToolsCtx) Exec(ctx context.Context, command string) (string, error) { return "ok", nil } func (f *fakeToolsCtx) Forgejo() tools.ForgejoActions { return fakeForgejoActions{mu: &f.mu, comments: f.comments} } ``` Hmm, passing a slice by value shares the backing array, but appends inside fakeForgejoActions mutate its own copy... the append happens on the copy's slice header β€” if it grows, the original doesn't see it. Ugh. Simpler: make fakeForgejoActions a pointer type: ```go type fakeToolsCtx struct { mu sync.Mutex comments []string } func (f *fakeToolsCtx) Exec(ctx context.Context, command string) (string, error) { return "ok", nil } func (f *fakeToolsCtx) Forgejo() tools.ForgejoActions { return f } ``` And put the ForgejoActions methods directly on *fakeToolsCtx: ```go func (f *fakeToolsCtx) Comment(body string) error { f.mu.Lock() defer f.mu.Unlock() f.comments = append(f.comments, body) return nil } // ... the rest are no-ops ``` Clean. Now β€” the test file needs to import `tools` for the interface reference. No problem. One more thing to double-check: `runLoop`'s signature takes `toolsCtx tools.Context`. In the tests I pass `&fakeToolsCtx{}`. Now, the delay assertion in test 1: `llmBackoff(1)` = 1s/2 + rand(0..1s/2) ∈ [500ms, 1s]. `llmBackoff(2)` = 2s/2 + rand(0..2s/2) ∈ [1s, 2s]. Good. Also β€” I should double-check that `rand.Int63n` doesn't panic: d/2 is at least 500ms β†’ int64(d/2)+1 > 0. No problem. Now, is there anything else that depends on the old error strings ("status: %d: %s", "decode completion chunk: %w: %s")? Let me grep for those strings in the tests/web. Also, the `NewStreamer` change: `io.ReadAll(io.LimitReader(resp.Body, maxStatusBody))` β€” need to keep the `io` import (already imported in client.go). Wait, client.go currently imports: bytes, context, encoding/json, net/http, config. NewStreamer is in client.go β€” uses io (io.ReadAll). Let me recheck the current client.go... yes: `body, _ := io.ReadAll(resp.Body)` β€” but the imports shown are bytes, context, encoding/json, net/http, config. No "io"?! Let me re-read the top of client.go... From my earlier read: ```go import ( "bytes" "context" "encoding/json" "net/http" "github.com/abrander/zoo/internal/config" ) ``` But NewStreamer uses io.ReadAll... that should be a compile error. Unless... wait, let me re-read. Lines 38-41: ```go func NewStreamer(resp *http.Response) (*Streamer, error) { if resp.StatusCode != http.StatusOK { body, _ := io.ReadAll(resp.Body) ``` Hmm, is NewStreamer in client.go? In my earlier read, client.go was 65 lines total, and NewStreamer was at line 38... and the imports don't include "io". That shouldn't compile. Let me re-read client.go to confirm β€” maybe I misread and NewStreamer is actually in streamer.go. Looking at the read_file output for client.go again: lines 30-65 show StreamChatCompletion, and... wait, the output showed: ``` 29 30 func (c *Client) StreamChatCompletion(ctx context.Context, req *ChatCompletionRequest) (*Streamer, error) { ... 65 } ``` And the read of streamer.go showed NewStreamer at lines 21-32. Yes β€” NewStreamer is in streamer.go (which imports io). I confused myself. client.go ends at line 65 with StreamChatCompletion. Good, no problem. So the StatusError change goes into streamer.go (NewStreamer), and the imports there: bufio, bytes, encoding/json, fmt, io, net/http. fmt is still used (in the DecodeError construction? no β€” DecodeError.Error() is in errors.go). After the change, streamer.go's Get() uses no fmt... let me check: the new Get() β€” no fmt usage. NewStreamer β€” no fmt. So drop fmt from streamer.go's imports. Actually, wait, let me write the new streamer.go carefully and check the imports: bufio, bytes, encoding/json, io, net/http. No fmt. errors.go imports: context, errors, fmt, net/http. Now β€” let me grep for other uses of the error strings and make sure nothing parses them. Then, implement everything, run `go build ./...`, `go vet ./...`, `go test ./...` (the docker-tagged tests are excluded by default). One more consideration: the `zoo:failed` label on this issue β€” the scheduler adds it when a run fails. My run is a new run; I shouldn't touch the label (it's a record of the previous failure). I'll leave it as is. Also, should I update zoo.hcl.example? No β€” no config changes. Let me also think about whether `OnTurnEnd` should fire on a failed attempt... no β€” it only fires on a completed stream (clean EOF). On a failed attempt, the stream didn't complete. Good β€” and this is important for streamHooks: OnTurnEnd persists to the store. A failed attempt must not persist partial content. Wait, actually β€” hmm, one more edge in streamTurn: what if the stream completes cleanly (EOF via [DONE]) but completion is nil (no chunks at all β€” an empty body with just [DONE])? "model returned an empty completion" β†’ transient β†’ retry. OnTurnEnd fires (the stream did complete)... but the buffer is empty, so OnTurnEnd persists nothing (reasoningOpen/contentOpen are false). Harmless. Now, about the `Hooks` doc comment and the `runLoop` doc comment β€” I'll update them to mention the retry. Let me also reconsider the max attempts / delays. 5 attempts, worst-case total backoff 1+2+4+8 = 15s. If the endpoint is down for longer than that, the run fails with a clear error and the scheduler posts a failure comment + zoo:failed label. The human can re-trigger. That's a reasonable balance β€” I don't want a single turn to burn 5 minutes on a dead endpoint. I'll document this in a comment. Also: should the retry count be visible in the live log? The job log (logAppend) β€” I could append a line like "LLM endpoint failed (attempt 1/5): ...; retrying in 1.2s". That would be genuinely useful for debugging via the dashboard. Hmm β€” but requestTurn doesn't have access to logAppend. I could add a hook... `OnRetry func(attempt int, delay time.Duration, err error)`? That grows the Hooks surface. Let me think about scope. The issue: "some kind of retry logic in the agent loop". The core is: retry with backoff. Live visibility is nice-to-have. But as an architect, observability matters β€” a turn that silently retries for 15 seconds looks like a hang in the dashboard. Hmm. Actually β€” wait. There's already a natural place: the hub's Status event? No β€” the dashboard's "status" case closes the SSE and reloads. Not usable. Let me add a lightweight hook: `OnRetry func(attempt int, delay time.Duration, err error)` β€” fired before each backoff sleep. In streamHooks, logAppend("status", fmt.Sprintf("LLM endpoint failure (%d/%d): %v β€” retrying in %s", ...)). Hmm, what streams does the store's AppendLog support? Let me check the store's schema for the log streams... logAppend is used with "reasoning", "content", "tool". Let me check how the web renders log entries and whether an unknown stream is OK. Let me check the store's schema and the web's job-detail rendering.
Now let me check how job logs are stored/rendered (to decide whether retry attempts should be visible in the dashboard), and grep for any dependencies on the current error strings.
Tool πŸ”§ read_file {"end_line": 80, "path": "internal/store/schema.sql", "start_line": 1}
{"end_line": 80, "path": "internal/store/schema.sql", "start_line": 1}
{"path":"internal/store/schema.sql","content":"CREATE TABLE IF NOT EXISTS watermark (\n    key   TEXT PRIMARY KEY,\n    value TEXT NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS seen_events (\n    id      TEXT PRIMARY KEY,\n    seen_at DATETIME NOT NULL\n);\n\nCREATE TABLE IF NOT EXISTS jobs (\n    id          TEXT PRIMARY KEY,\n    event_kind  TEXT NOT NULL,\n    agent       TEXT NOT NULL,\n    owner       TEXT NOT NULL,\n    repo        TEXT NOT NULL,\n    issue_index INTEGER NOT NULL,\n    title       TEXT NOT NULL DEFAULT '',\n    status      TEXT NOT NULL,\n    created_at  DATETIME NOT NULL,\n    started_at  DATETIME,\n    finished_at DATETIME,\n    error       TEXT NOT NULL DEFAULT ''\n);\n\nCREATE INDEX IF NOT EXISTS idx_jobs_created_at ON jobs (created_at DESC);\n\nCREATE TABLE IF NOT EXISTS job_logs (\n    job_id TEXT NOT NULL,\n    seq    INTEGER NOT NULL,\n    stream TEXT NOT NULL,\n    line   TEXT NOT NULL,\n    ts     DATETIME NOT NULL,\n    PRIMARY KEY (job_id, seq)\n);\n\n-- Last-known assignee set per issue, used by the poller to tell a real\n-- (re-)assignment apart from any other update to an already-assigned\n-- issue. Webhooks say \"assigned\" outright; polling only ever sees state.\nCREATE TABLE IF NOT EXISTS issue_assignees (\n    issue_id INTEGER NOT NULL,\n    assignee TEXT NOT NULL,\n    PRIMARY KEY (issue_id, assignee)\n);","start_line":1,"end_line":44,"total_lines":44,"truncated":false}
Tool πŸ”§ grep_search {"context_after": 8, "query": "status: %d|decode completion chunk"}
{"context_after": 8, "query": "status: %d|decode completion chunk"}
/project/internal/llm/streamer.go:23:		return nil, fmt.Errorf("status: %d: %s", resp.StatusCode, body)
/project/internal/llm/streamer.go-24-	}
/project/internal/llm/streamer.go-25-
/project/internal/llm/streamer.go-26-	return &Streamer{
/project/internal/llm/streamer.go-27-		resp:    resp,
/project/internal/llm/streamer.go-28-		scanner: bufio.NewScanner(resp.Body),
/project/internal/llm/streamer.go-29-	}, nil
/project/internal/llm/streamer.go-30-}
/project/internal/llm/streamer.go-31-
--
/project/internal/llm/streamer.go:73:		return nil, fmt.Errorf("decode completion chunk: %w: %s", err, buf)
/project/internal/llm/streamer.go-74-	}
/project/internal/llm/streamer.go-75-
/project/internal/llm/streamer.go-76-	s.completion.Merge(&completion)
/project/internal/llm/streamer.go-77-
/project/internal/llm/streamer.go-78-	return &s.completion, nil
/project/internal/llm/streamer.go-79-}
Tool πŸ”§ grep_search {"context_after": 10, "context_before": 3, "query": "case \"tool\"|stream ==|Stream =="}
{"context_after": 10, "context_before": 3, "query": "case \"tool\"|stream ==|Stream =="}
/project/internal/web/templates.go-983-            if (contentBody) contentBody.textContent = contentBody.textContent.trim();
/project/internal/web/templates.go-984-            contentBody = null;
/project/internal/web/templates.go-985-            break;
/project/internal/web/templates.go:986:          case "tool":
/project/internal/web/templates.go-987-            newToolBlock(ev);
/project/internal/web/templates.go-988-            break;
/project/internal/web/templates.go-989-          case "status":
/project/internal/web/templates.go-990-            es.close();
/project/internal/web/templates.go-991-            location.reload();
/project/internal/web/templates.go-992-            return;
/project/internal/web/templates.go-993-        }
/project/internal/web/templates.go-994-
/project/internal/web/templates.go-995-        if (shouldScroll) window.scrollTo(0, document.body.scrollHeight);
/project/internal/web/templates.go-996-      };
--
/project/internal/web/web.go-222-			}
/project/internal/web/web.go-223-			blocks = append(blocks, block{Kind: l.Stream, Text: text})
/project/internal/web/web.go-224-
/project/internal/web/web.go:225:		case "tool":
/project/internal/web/web.go-226-			var entry store.ToolLogEntry
/project/internal/web/web.go-227-			if err := json.Unmarshal([]byte(l.Line), &entry); err != nil {
/project/internal/web/web.go-228-				blocks = append(blocks, block{Kind: "system", Text: l.Line})
/project/internal/web/web.go-229-
/project/internal/web/web.go-230-				continue
/project/internal/web/web.go-231-			}
/project/internal/web/web.go-232-
/project/internal/web/web.go-233-			blocks = append(blocks, block{Kind: "tool", Tool: &toolBlock{
/project/internal/web/web.go-234-				Name:      entry.Name,
/project/internal/web/web.go-235-				Arguments: entry.Arguments,
--
/project/internal/store/store.go-314-}
/project/internal/store/store.go-315-
/project/internal/store/store.go-316-// ToolLogEntry is the JSON shape stored in a LogLine's Line field for
/project/internal/store/store.go:317:// Stream == "tool": one complete tool call and its result.
/project/internal/store/store.go-318-type ToolLogEntry struct {
/project/internal/store/store.go-319-	Name      string `json:"name"`
/project/internal/store/store.go-320-	Arguments string `json:"arguments"`
/project/internal/store/store.go-321-	Result    string `json:"result"`
/project/internal/store/store.go-322-	Error     bool   `json:"error"`
/project/internal/store/store.go-323-}
/project/internal/store/store.go-324-
/project/internal/store/store.go-325-// AppendLog appends one log line for a job, assigning it the next
/project/internal/store/store.go-326-// sequence number for that job.
/project/internal/store/store.go-327-func (s *Store) AppendLog(ctx context.Context, jobID, stream, line string) error {
Thinking
Let's look at around lines 200-240 of web.go and check how log streams other than "tool" are rendered (the default case).
Tool πŸ”§ read_file {"end_line": 245, "path": "internal/web/web.go", "start_line": 190}
{"end_line": 245, "path": "internal/web/web.go", "start_line": 190}
{"path":"internal/web/web.go","content":"type toolBlock struct {\n\tName      string\n\tArguments string\n\tResult    string\n\tError     bool\n}\n\n// block is one self-contained, already-complete unit of job output: a\n// finished reasoning or assistant-message block, a finished tool call,\n// or a misc system note. Unlike the old flat log view, one store.LogLine\n// maps to exactly one block β€” grouping/streaming happens upstream, when\n// agentrun persists the row.\ntype block struct {\n\tKind string // \"reasoning\" | \"content\" | \"tool\" | \"system\"\n\tText string\n\tTool *toolBlock\n}\n\nfunc buildBlocks(logs []store.LogLine) []block {\n\tblocks := make([]block, 0, len(logs))\n\n\tfor _, l := range logs {\n\t\tswitch l.Stream {\n\t\tcase \"reasoning\", \"content\":\n\t\t\t// Model output routinely starts/ends with newlines. The block\n\t\t\t// body renders with white-space: pre-wrap, so those would show\n\t\t\t// up as visible blank lines inflating the block's height. Trim\n\t\t\t// them for display (internal newlines are kept) and drop\n\t\t\t// blocks that are nothing but whitespace.\n\t\t\ttext := strings.TrimSpace(l.Line)\n\t\t\tif text == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tblocks = append(blocks, block{Kind: l.Stream, Text: text})\n\n\t\tcase \"tool\":\n\t\t\tvar entry store.ToolLogEntry\n\t\t\tif err := json.Unmarshal([]byte(l.Line), \u0026entry); err != nil {\n\t\t\t\tblocks = append(blocks, block{Kind: \"system\", Text: l.Line})\n\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tblocks = append(blocks, block{Kind: \"tool\", Tool: \u0026toolBlock{\n\t\t\t\tName:      entry.Name,\n\t\t\t\tArguments: entry.Arguments,\n\t\t\t\tResult:    entry.Result,\n\t\t\t\tError:     entry.Error,\n\t\t\t}})\n\n\t\tdefault:\n\t\t\tblocks = append(blocks, block{Kind: \"system\", Text: l.Line})\n\t\t}\n\t}\n\n\treturn blocks","start_line":190,"end_line":245,"total_lines":374,"truncated":true}
Thinking
The default case renders any other stream as a "system" block. In other words, if we append a retry notification to the log with a stream like "system", it will be rendered as a system block in the static view. But wait β€” in the live view, system log lines are not published to the hub (only reasoning/content/tool events are). A retry notification appended to the store won't be visible live, but will be visible in the static (reloaded) view. Hmm. Options for live visibility: 1. Publish a `turn_reset` event to the hub (already planned) β€” the live viewer sees the partial block disappear, and then the new content streams in. This is live feedback that a retry is happening. 2. Additionally, persist a "system" log line recording the failure + retry, so it remains in the history. I think both are good. Let me add an `OnRetry` hook… no wait. Let me reconsider: do I want to add both OnTurnReset and OnRetry? - `OnTurnReset` fires only when partial content was streamed (a live-view concern). - Retry logging is useful for every retry, whether or not anything was streamed. I could unify: a single hook `OnRetry(attempt int, delay time.Duration, err error)` fired before every backoff sleep. streamHooks implementation: - logAppend("system", fmt.Sprintf("LLM endpoint failure (attempt %d/%d): %v β€” retrying in %s", attempt, llmMaxAttempts, err, delay)) - if partial content was open: reset the buffer, publish TurnReset, checkpoint. Hmm, but then the "reset" logic gets mixed into OnRetry, and the "was anything streamed" state lives in streamHooks (reasoningOpen/contentOpen) β€” which is exactly where it belongs! In fact, this is cleaner than OnTurnReset: streamHooks knows whether a block is open, so it can decide whether to publish turn_reset. Wait, but then runLoop's requestTurn doesn't need the `streamed` flag at all β€” it just fires OnRetry on every retry, and streamHooks decides whether to reset the live view based on its own open-block state. Hmm, but is that right? Let me think: attempt 1 streams partial reasoning (reasoningOpen=true). Attempt 1 fails. OnRetry fires β†’ streamHooks: reasoningOpen is true β†’ reset buffer, publish turn_reset, checkpoint. Good. Attempt 1 fails at the request level (nothing streamed, reasoningOpen=false). OnRetry fires β†’ streamHooks: no open block β†’ just log the line. Good. But a subtle point: what if reasoningOpen is true but the buffer is empty (a reasoning_start was published but no deltas yet)? Can that happen? reasoningOpen becomes true only when the first reasoning delta arrives (OnReasoningDelta sets reasoningOpen=true and then writes the delta). So open β‡’ non-empty buffer. No problem. Also: a turn where content is open but reasoning isn't, etc. β€” handled independently. So a single hook `OnRetry func(attempt int, delay time.Duration, err error)` is enough, and the `streamed` flag in streamTurn becomes unnecessary. Simpler! Wait, but hold on β€” is the "system" log line appropriate? Let me check how the "system" block is styled in the template… there's a `block-system` class. Let me check the CSS. Also, let me check whether there are existing uses of logAppend with other streams. Actually, let me grep for logAppend usages and the block-system CSS.
Tool πŸ”§ grep_search {"context_after": 6, "context_before": 2, "query": "logAppend\\(|block-system"}
{"context_after": 6, "context_before": 2, "query": "logAppend\\(|block-system"}
/project/internal/web/templates.go-477-  }
/project/internal/web/templates.go-478-
/project/internal/web/templates.go:479:  .block-system {
/project/internal/web/templates.go-480-    padding: 0.35rem 0.75rem;
/project/internal/web/templates.go-481-    color: var(--text-muted);
/project/internal/web/templates.go-482-    font-family: var(--mono);
/project/internal/web/templates.go-483-    font-size: 0.8rem;
/project/internal/web/templates.go-484-  }
/project/internal/web/templates.go-485-
--
/project/internal/web/templates.go-872-      </details>
/project/internal/web/templates.go-873-      {{else}}
/project/internal/web/templates.go:874:      <div class="block block-system">{{.Text}}</div>
/project/internal/web/templates.go-875-      {{end}}
/project/internal/web/templates.go-876-    {{end}}
/project/internal/web/templates.go-877-  </div>
/project/internal/web/templates.go-878-
/project/internal/web/templates.go-879-  {{if .Live}}
/project/internal/web/templates.go-880-  <script>
--
/project/internal/agentrun/run.go-315-			if reasoningOpen {
/project/internal/agentrun/run.go-316-				r.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})
/project/internal/agentrun/run.go:317:				logAppend("reasoning", reasoningBuf.String())
/project/internal/agentrun/run.go-318-				r.hub.Checkpoint(jobID)
/project/internal/agentrun/run.go-319-				reasoningOpen = false
/project/internal/agentrun/run.go-320-			}
/project/internal/agentrun/run.go-321-
/project/internal/agentrun/run.go-322-			if contentOpen {
/project/internal/agentrun/run.go-323-				r.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})
/project/internal/agentrun/run.go:324:				logAppend("content", contentBuf.String())
/project/internal/agentrun/run.go-325-				r.hub.Checkpoint(jobID)
/project/internal/agentrun/run.go-326-				contentOpen = false
/project/internal/agentrun/run.go-327-			}
/project/internal/agentrun/run.go-328-		},
/project/internal/agentrun/run.go-329-		OnTool: func(name, arguments, result string, toolErr bool) {
/project/internal/agentrun/run.go-330-			r.hub.Publish(jobID, livelog.Event{
--
/project/internal/agentrun/run.go-340-				r.logger.Warn("failed to marshal tool log entry", "job", jobID, "error", err)
/project/internal/agentrun/run.go-341-			} else {
/project/internal/agentrun/run.go:342:				logAppend("tool", string(line))
/project/internal/agentrun/run.go-343-			}
/project/internal/agentrun/run.go-344-
/project/internal/agentrun/run.go-345-			r.hub.Checkpoint(jobID)
/project/internal/agentrun/run.go-346-		},
/project/internal/agentrun/run.go-347-	}
/project/internal/agentrun/run.go-348-}
Thinking
The "system" block renders as a muted monospace note β€” perfect for retry notifications. Final design: **Hooks** (agentrun/loop.go): - `OnRetry func(attempt int, delay time.Duration, err error)` β€” fires before each backoff sleep, i.e. every time a transient failure is about to be retried. `attempt` is the number of the failed attempt (1-based), `delay` is the backoff about to be waited, `err` is the failure. **streamHooks** (agentrun/run.go): ```go OnRetry: func(attempt int, delay time.Duration, err error) { // Record the failure for the job's history... logAppend("system", fmt.Sprintf("LLM endpoint failure (attempt %d/%d): %v β€” retrying in %s", attempt, llmMaxAttempts, err, delay.Round(time.Second))) // ...and if part of the turn has already been streamed, discard // it from the live view and the replay buffer (it was never // persisted), so the retried turn starts clean. if reasoningOpen || contentOpen { reasoningOpen, contentOpen = false, false reasoningBuf.Reset() contentBuf.Reset() r.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset}) r.hub.Checkpoint(jobID) } }, ``` Hmm, wait β€” order: publish turn_reset to live subscribers, then checkpoint. A subscriber that connects between publish and checkpoint will receive the partial deltas + turn_reset replayed β†’ processed in order β†’ cleared. A subscriber that connects after the checkpoint sees nothing. Correct. But, hmm, one issue: `delay.Round(time.Second)` β€” delays like 734ms β†’ "1s". Fine for a human-facing message. Or use `%v` on the duration: "734ms". Actually, `%v` on a time.Duration prints "734ms" or "1.234s" β€” that's fine too. Let's use %v for accuracy... no actually, for a human-facing log line, rounding to the nearest second is more readable: "retrying in 1s". But the minimum delay is 500ms β†’ rounds to "1s" (0.5s rounds to 1s? time.Duration(500ms).Round(time.Second) = 1s, yes). OK, let's use Round(time.Second). Hmm, wait β€” actually, let me reconsider whether to include the full error text. Errors can be long (StatusError with body up to 1KB). The log line would be huge. Truncate? The StatusError body is capped at 1KB by NewStreamer. A 1KB log line is ugly but informative. Hmm. Let's keep it simple: include err as-is. The body cap keeps it bounded. No, actually, for the log line, let's truncate to about 200 chars for readability: Hmm, that adds a helper. Let me think... the most useful part of the error is the status code / reason, which is at the front. Let's write a small helper `truncate(s string, n int)`. OK. No β€” wait. Let me step back and reconsider the scope. What the issue asks for is retry logic. What I'm adding: 1. Typed errors + IsTransient in llm (core) 2. Retry with backoff+jitter in runLoop (core) 3. OnRetry hook β†’ system log line + turn reset in the live view (observability/UX) 4. turn_reset event + JS (UX) 5. Tests This is a complete, well-scoped change. Let's go. One more decision: the `llmMaxAttempts` constant is referenced from run.go (streamHooks) for the log message. Both are in the agentrun package β€” no problem. Now, let me also double-check the `Choice` merge behavior for the tool-call regression test: a tool-call chunk with `delta.tool_calls[0]` having id/type/name/arguments and finish_reason "tool_calls". Message.Merge handles ToolCalls. Choice.FinishReason is set. runLoop sees FinishReason == "tool_calls" and len(ToolCalls) > 0 β†’ executes the tool. Then the second turn: messages contains the assistant message with ToolCalls and the tool result. The server (my test handler) ignores the body and returns the final content. Good. Now β€” one more thing to verify: `tools.Call` for "comment" with args `{"body":"hi"}` β†’ commentParams{Body:"hi"} β†’ ctx.Forgejo().Comment("hi") β†’ my fake appends. Returns "Comment posted". Now let me write all the files. Order: 1. internal/llm/errors.go (new) 2. internal/llm/streamer.go (edit NewStreamer + Get) 3. internal/llm/errors_test.go (new) 4. internal/llm/streamer_test.go (add tests) 5. internal/agentrun/loop.go (rewrite) 6. internal/agentrun/run.go (edit streamHooks) 7. internal/livelog/hub.go (add TurnReset) 8. internal/web/templates.go (JS case) 9. internal/agentrun/loop_test.go (new) Let's start writing. First, errors.go:
I now have the full picture. Let me lay out the design before implementing: **Design** 1. **`internal/llm`** β€” typed error taxonomy so failures can be classified: `StatusError` (non-200), `StreamError` (connection broke / ended before `[DONE]`), `DecodeError` (malformed chunk), plus `IsTransient(err)` (transport errors, 5xx/429/408, mid-stream breaks β†’ retryable; other 4xx, malformed responses, ctx cancellation β†’ not). 2. **`internal/agentrun/loop.go`** β€” the agent loop retries a failed *turn* (request + full stream read) with exponential backoff + equal jitter, bounded per turn (5 attempts, ~15s worst-case backoff), never fighting a canceled/timed-out run. 3. **Observability** β€” an `OnRetry` hook: the run persists a "system" log line per retry attempt, and if partial output was already streamed, a `turn_reset` livelog event clears it from the dashboard so the retried turn starts clean. Let me implement. First, the llm error taxonomy:
Tool πŸ”§ write_file {"content": "package llm\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\t\"net/http\"\n)\n\n// The client classifies its failures into a small set of typed errors\n// so callers can decide what is worth retrying (see IsTransient):\n//\n// - StatusError: the endpoint answered with a non-200 status.\n// - StreamError: the response body broke or ended before the\n// stream's [DONE] marker, i.e. the connection dropped mid-turn.\n// - DecodeError: the endpoint sent a data line that is not valid\n// JSON, i.e. a protocol violation rather than a transient fault.\n//\n// Unrecognized errors (transport failures such as connection refused\n// or reset, DNS errors, timeouts) are treated as transient by default:\n// when in doubt, the endpoint is assumed flaky and the caller's\n// bounded retry budget decides.\n\n// StatusError is returned when the endpoint answers with a non-200\n// status.\ntype StatusError struct {\n\tStatusCode int\n\tBody string\n}\n\nfunc (e *StatusError) Error() string {\n\treturn fmt.Sprintf(\"llm endpoint returned status %d: %s\", e.StatusCode, e.Body)\n}\n\n// StreamError is returned when the stream breaks or ends before its\n// [DONE] marker. Err is the underlying read error if the body broke\n// with one; a nil Err means the body ended cleanly without [DONE].\ntype StreamError struct {\n\tErr error\n}\n\nfunc (e *StreamError) Error() string {\n\tif e.Err != nil {\n\t\treturn fmt.Sprintf(\"llm stream interrupted: %v\", e.Err)\n\t}\n\treturn \"llm stream ended before [DONE]\"\n}\n\nfunc (e *StreamError) Unwrap() error { return e.Err }\n\n// DecodeError is returned when a data line is not valid JSON.\ntype DecodeError struct {\n\tErr error\n\tChunk string\n}\n\nfunc (e *DecodeError) Error() string {\n\treturn fmt.Sprintf(\"decode completion chunk: %v: %s\", e.Err, e.Chunk)\n}\n\nfunc (e *DecodeError) Unwrap() error { return e.Err }\n\n// IsTransient reports whether a failed LLM round trip is worth\n// retrying. Transport failures, 5xx/429/408 responses, and mid-stream\n// interruptions are transient: the endpoint may be down or overloaded,\n// and the same request may succeed later. Client errors (other 4xx)\n// and malformed responses are not: retrying the same request cannot\n// fix them. Context cancellation is never transient β€” the caller gave\n// up.\nfunc IsTransient(err error) bool {\n\tif err == nil {\n\t\treturn false\n\t}\n\n\tif errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {\n\t\treturn false\n\t}\n\n\tvar se *StatusError\n\tif errors.As(err, &se) {\n\t\treturn se.StatusCode == http.StatusRequestTimeout ||\n\t\t\tse.StatusCode == http.StatusTooManyRequests ||\n\t\t\tse.StatusCode >= 500\n\t}\n\n\tvar de *DecodeError\n\tif errors.As(err, &de) {\n\t\treturn false\n\t}\n\n\treturn true\n}\n", "path": "internal/llm/errors.go"}
{"content": "package llm\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\t\"net/http\"\n)\n\n// The client classifies its failures into a small set of typed errors\n// so callers can decide what is worth retrying (see IsTransient):\n//\n//   - StatusError: the endpoint answered with a non-200 status.\n//   - StreamError: the response body broke or ended before the\n//     stream's [DONE] marker, i.e. the connection dropped mid-turn.\n//   - DecodeError: the endpoint sent a data line that is not valid\n//     JSON, i.e. a protocol violation rather than a transient fault.\n//\n// Unrecognized errors (transport failures such as connection refused\n// or reset, DNS errors, timeouts) are treated as transient by default:\n// when in doubt, the endpoint is assumed flaky and the caller's\n// bounded retry budget decides.\n\n// StatusError is returned when the endpoint answers with a non-200\n// status.\ntype StatusError struct {\n\tStatusCode int\n\tBody       string\n}\n\nfunc (e *StatusError) Error() string {\n\treturn fmt.Sprintf(\"llm endpoint returned status %d: %s\", e.StatusCode, e.Body)\n}\n\n// StreamError is returned when the stream breaks or ends before its\n// [DONE] marker. Err is the underlying read error if the body broke\n// with one; a nil Err means the body ended cleanly without [DONE].\ntype StreamError struct {\n\tErr error\n}\n\nfunc (e *StreamError) Error() string {\n\tif e.Err != nil {\n\t\treturn fmt.Sprintf(\"llm stream interrupted: %v\", e.Err)\n\t}\n\treturn \"llm stream ended before [DONE]\"\n}\n\nfunc (e *StreamError) Unwrap() error { return e.Err }\n\n// DecodeError is returned when a data line is not valid JSON.\ntype DecodeError struct {\n\tErr   error\n\tChunk string\n}\n\nfunc (e *DecodeError) Error() string {\n\treturn fmt.Sprintf(\"decode completion chunk: %v: %s\", e.Err, e.Chunk)\n}\n\nfunc (e *DecodeError) Unwrap() error { return e.Err }\n\n// IsTransient reports whether a failed LLM round trip is worth\n// retrying. Transport failures, 5xx/429/408 responses, and mid-stream\n// interruptions are transient: the endpoint may be down or overloaded,\n// and the same request may succeed later. Client errors (other 4xx)\n// and malformed responses are not: retrying the same request cannot\n// fix them. Context cancellation is never transient β€” the caller gave\n// up.\nfunc IsTransient(err error) bool {\n\tif err == nil {\n\t\treturn false\n\t}\n\n\tif errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {\n\t\treturn false\n\t}\n\n\tvar se *StatusError\n\tif errors.As(err, &se) {\n\t\treturn se.StatusCode == http.StatusRequestTimeout ||\n\t\t\tse.StatusCode == http.StatusTooManyRequests ||\n\t\t\tse.StatusCode >= 500\n\t}\n\n\tvar de *DecodeError\n\tif errors.As(err, &de) {\n\t\treturn false\n\t}\n\n\treturn true\n}\n", "path": "internal/llm/errors.go"}
OK
Thinking
Next, I'll update streamer.go: NewStreamer returns a StatusError, and Get() has proper scanner error handling, using StreamError/DecodeError.
Now update the streamer to use the typed errors (and fix the dead scanner-error check β€” it was inside the loop where `Err()` is always nil):
Tool πŸ”§ write_file {"content": "package llm\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"io\"\n\t\"net/http\"\n)\n\n// maxStatusBody caps how much of a non-200 response body is kept for\n// the error message.\nconst maxStatusBody = 1024\n\ntype Streamer struct {\n\tcompletion ChatCompletion\n\tresp *http.Response\n\tscanner *bufio.Scanner\n\teof bool\n}\n\nfunc NewStreamer(resp *http.Response) (*Streamer, error) {\n\tif resp.StatusCode != http.StatusOK {\n\t\tbody, _ := io.ReadAll(io.LimitReader(resp.Body, maxStatusBody))\n\t\tresp.Body.Close()\n\t\treturn nil, &StatusError{StatusCode: resp.StatusCode, Body: string(body)}\n\t}\n\n\treturn &Streamer{\n\t\tresp: resp,\n\t\tscanner: bufio.NewScanner(resp.Body),\n\t}, nil\n}\n\n// Get returns the accumulated completion after merging in the next SSE\n// chunk. Callers should keep calling Get until it returns io.EOF.\nfunc (s *Streamer) Get() (*ChatCompletion, error) {\n\tif s.eof {\n\t\treturn nil, io.EOF\n\t}\n\n\tprefix := []byte(\"data: \")\n\n\tvar buf []byte\n\tvar found bool\n\n\tfor s.scanner.Scan() {\n\t\tline := bytes.TrimSpace(s.scanner.Bytes())\n\n\t\tif len(line) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif bytes.HasPrefix(line, prefix) {\n\t\t\tbuf = bytes.TrimPrefix(line, prefix)\n\t\t\tfound = true\n\t\t\tbreak\n\t\t}\n\t}\n\n\tif err := s.scanner.Err(); err != nil {\n\t\treturn nil, &StreamError{Err: err}\n\t}\n\n\tif !found {\n\t\t// The body ended before a data line arrived: the stream was\n\t\t// cut off before [DONE].\n\t\ts.resp.Body.Close()\n\t\ts.eof = true\n\t\treturn nil, &StreamError{}\n\t}\n\n\tif bytes.Equal(buf, []byte(\"[DONE]\")) {\n\t\ts.resp.Body.Close()\n\t\ts.eof = true\n\t\treturn nil, io.EOF\n\t}\n\n\tvar completion ChatCompletion\n\n\t// Unknown fields are tolerated on purpose: servers (e.g. newer\n\t// llama.cpp builds) may add fields we don't know about yet, and\n\t// failing on them would break compatibility.\n\tif err := json.Unmarshal(buf, &completion); err != nil {\n\t\treturn nil, &DecodeError{Err: err, Chunk: string(buf)}\n\t}\n\n\ts.completion.Merge(&completion)\n\n\treturn &s.completion, nil\n}\n", "path": "internal/llm/streamer.go"}
{"content": "package llm\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"io\"\n\t\"net/http\"\n)\n\n// maxStatusBody caps how much of a non-200 response body is kept for\n// the error message.\nconst maxStatusBody = 1024\n\ntype Streamer struct {\n\tcompletion ChatCompletion\n\tresp       *http.Response\n\tscanner    *bufio.Scanner\n\teof        bool\n}\n\nfunc NewStreamer(resp *http.Response) (*Streamer, error) {\n\tif resp.StatusCode != http.StatusOK {\n\t\tbody, _ := io.ReadAll(io.LimitReader(resp.Body, maxStatusBody))\n\t\tresp.Body.Close()\n\t\treturn nil, &StatusError{StatusCode: resp.StatusCode, Body: string(body)}\n\t}\n\n\treturn &Streamer{\n\t\tresp:    resp,\n\t\tscanner: bufio.NewScanner(resp.Body),\n\t}, nil\n}\n\n// Get returns the accumulated completion after merging in the next SSE\n// chunk. Callers should keep calling Get until it returns io.EOF.\nfunc (s *Streamer) Get() (*ChatCompletion, error) {\n\tif s.eof {\n\t\treturn nil, io.EOF\n\t}\n\n\tprefix := []byte(\"data: \")\n\n\tvar buf []byte\n\tvar found bool\n\n\tfor s.scanner.Scan() {\n\t\tline := bytes.TrimSpace(s.scanner.Bytes())\n\n\t\tif len(line) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tif bytes.HasPrefix(line, prefix) {\n\t\t\tbuf = bytes.TrimPrefix(line, prefix)\n\t\t\tfound = true\n\t\t\tbreak\n\t\t}\n\t}\n\n\tif err := s.scanner.Err(); err != nil {\n\t\treturn nil, &StreamError{Err: err}\n\t}\n\n\tif !found {\n\t\t// The body ended before a data line arrived: the stream was\n\t\t// cut off before [DONE].\n\t\ts.resp.Body.Close()\n\t\ts.eof = true\n\t\treturn nil, &StreamError{}\n\t}\n\n\tif bytes.Equal(buf, []byte(\"[DONE]\")) {\n\t\ts.resp.Body.Close()\n\t\ts.eof = true\n\t\treturn nil, io.EOF\n\t}\n\n\tvar completion ChatCompletion\n\n\t// Unknown fields are tolerated on purpose: servers (e.g. newer\n\t// llama.cpp builds) may add fields we don't know about yet, and\n\t// failing on them would break compatibility.\n\tif err := json.Unmarshal(buf, &completion); err != nil {\n\t\treturn nil, &DecodeError{Err: err, Chunk: string(buf)}\n\t}\n\n\ts.completion.Merge(&completion)\n\n\treturn &s.completion, nil\n}\n", "path": "internal/llm/streamer.go"}
OK
Thinking
Next up is the agent loop with retry logic. I'll rewrite loop.go:
Tool πŸ”§ write_file {"content": "package agentrun\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\t\"math/rand\"\n\t\"time\"\n\n\t\"github.com/abrander/zoo/internal/llm\"\n\t\"github.com/abrander/zoo/internal/tools\"\n)\n\n// Hooks lets a caller observe a runLoop invocation live, as the model\n// streams and tool calls execute, in addition to the final []llm.Message\n// it returns. Any of these may be nil.\ntype Hooks struct {\n\t// OnReasoningDelta and OnContentDelta fire with just the newly\n\t// streamed text for the current turn, not the accumulated total.\n\tOnReasoningDelta func(delta string)\n\tOnContentDelta func(delta string)\n\n\t// OnTurnEnd fires once per completed streamer round-trip, after the\n\t// model's message for that turn is fully received and before any of\n\t// its tool calls run.\n\tOnTurnEnd func()\n\n\t// OnRetry fires before each backoff wait, i.e. whenever a turn's\n\t// round trip failed transiently and is about to be retried. attempt\n\t// is the 1-based number of the failed attempt, delay is the wait\n\t// that is about to happen, and err is the failure.\n\tOnRetry func(attempt int, delay time.Duration, err error)\n\n\t// OnTool fires once per tool call, after it has run.\n\tOnTool func(name, arguments, result string, toolErr bool)\n}\n\n// The retry budget for one LLM round trip (request + full stream read),\n// not for the whole run: a run can last hours and each turn is\n// independent, so a failing endpoint should cost at most a few seconds\n// of backoff per turn before the run surfaces the error.\nconst (\n\tllmMaxAttempts = 5\n\tllmBaseDelay = time.Second\n\tllmMaxDelay = 15 * time.Second\n)\n\n// llmSleep waits d, returning early with ctx's error if the run is\n// canceled or times out while waiting. Overridden in tests.\nvar llmSleep = func(ctx context.Context, d time.Duration) error {\n\tselect {\n\tcase <-time.After(d):\n\t\treturn nil\n\tcase <-ctx.Done():\n\t\treturn ctx.Err()\n\t}\n}\n\n// runLoop is a headless port of ../a's App.generate(): send messages +\n// tool defs, get a completion, run any tool_calls and append their\n// results, repeat until a plain finish or ctx is done. Transient LLM\n// endpoint failures (connection errors, 5xx/429 responses, mid-stream\n// breaks) are retried per turn with exponential backoff β€” see\n// requestTurn.\nfunc runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) {\n\tfor {\n\t\tif err := ctx.Err(); err != nil {\n\t\t\treturn messages, err\n\t\t}\n\n\t\tcompletion, err := requestTurn(ctx, client, messages, hooks)\n\t\tif err != nil {\n\t\t\treturn messages, err\n\t\t}\n\n\t\tchoice := completion.Choices[0]\n\t\tmessages = append(messages, choice.Message)\n\n\t\tif choice.FinishReason != \"tool_calls\" || len(choice.Message.ToolCalls) == 0 {\n\t\t\treturn messages, nil\n\t\t}\n\n\t\tfor _, tc := range choice.Message.ToolCalls {\n\t\t\tif err := ctx.Err(); err != nil {\n\t\t\t\treturn messages, err\n\t\t\t}\n\n\t\t\tresult, err := tools.Call(toolsCtx, tc.FunctionCall)\n\n\t\t\ttoolErr := err != nil\n\t\t\tif toolErr {\n\t\t\t\tresult = fmt.Sprintf(\"ERROR CALLING TOOL: %s\", err.Error())\n\t\t\t}\n\n\t\t\tif hooks.OnTool != nil {\n\t\t\t\thooks.OnTool(tc.FunctionCall.Name, tc.FunctionCall.Arguments, result, toolErr)\n\t\t\t}\n\n\t\t\tmessages = append(messages, llm.Message{\n\t\t\t\tRole: \"tool\",\n\t\t\t\tContent: result,\n\t\t\t\tToolCallID: tc.Id,\n\t\t\t\tName: tc.FunctionCall.Name,\n\t\t\t})\n\t\t}\n\t}\n}\n\n// requestTurn performs one LLM round trip (request + full stream\n// read), retrying transient endpoint failures with exponential\n// backoff and jitter. It returns the completed turn's accumulated\n// completion.\nfunc requestTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) {\n\tfor attempt := 1; ; attempt++ {\n\t\tcompletion, err := streamTurn(ctx, client, messages, hooks)\n\t\tif err == nil {\n\t\t\treturn completion, nil\n\t\t}\n\n\t\t// A canceled or timed-out run is not a failing endpoint; stop\n\t\t// and report the run's own error.\n\t\tif ctxErr := ctx.Err(); ctxErr != nil {\n\t\t\treturn nil, ctxErr\n\t\t}\n\n\t\t// Permanent failures (client errors, malformed responses) and\n\t\t// exhausted budgets fail immediately.\n\t\tif !llm.IsTransient(err) || attempt == llmMaxAttempts {\n\t\t\tif attempt > 1 {\n\t\t\t\treturn nil, fmt.Errorf(\"llm endpoint failed after %d attempts: %w\", attempt, err)\n\t\t\t}\n\t\t\treturn nil, err\n\t\t}\n\n\t\tdelay := llmBackoff(attempt)\n\n\t\tif hooks.OnRetry != nil {\n\t\t\thooks.OnRetry(attempt, delay, err)\n\t\t}\n\n\t\tif err := llmSleep(ctx, delay); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n}\n\n// llmBackoff returns the delay before retrying after the given failed\n// attempt (1-based): exponential growth from llmBaseDelay, capped at\n// llmMaxDelay, with equal jitter (half fixed, half random) so\n// concurrent agents don't retry in lockstep.\nfunc llmBackoff(attempt int) time.Duration {\n\td := llmBaseDelay << (attempt - 1)\n\tif d <= 0 || d > llmMaxDelay {\n\t\td = llmMaxDelay\n\t}\n\treturn d/2 + time.Duration(rand.Int63n(int64(d/2)+1))\n}\n\n// streamTurn performs one attempt at an LLM round trip: issue the\n// request and read the stream to completion, firing hooks as content\n// arrives.\nfunc streamTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) {\n\tstreamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{\n\t\tMessages: messages,\n\t\tStream: true,\n\t\tTools: tools.All(),\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"chat completion: %w\", err)\n\t}\n\n\tvar completion *llm.ChatCompletion\n\n\tvar prevContent, prevReasoning string\n\n\tfor {\n\t\tc, err := streamer.Get()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"stream completion: %w\", err)\n\t\t}\n\n\t\tcompletion = c\n\n\t\tif len(c.Choices) > 0 {\n\t\t\tmsg := c.Choices[0].Message\n\n\t\t\tif hooks.OnReasoningDelta != nil && len(msg.ReasoningContent) > len(prevReasoning) {\n\t\t\t\thooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):])\n\t\t\t}\n\t\t\tprevReasoning = msg.ReasoningContent\n\n\t\t\tif hooks.OnContentDelta != nil && len(msg.Content) > len(prevContent) {\n\t\t\t\thooks.OnContentDelta(msg.Content[len(prevContent):])\n\t\t\t}\n\t\t\tprevContent = msg.Content\n\t\t}\n\t}\n\n\tif hooks.OnTurnEnd != nil {\n\t\thooks.OnTurnEnd()\n\t}\n\n\tif completion == nil || len(completion.Choices) == 0 {\n\t\treturn nil, fmt.Errorf(\"model returned an empty completion\")\n\t}\n\n\treturn completion, nil\n}\n", "path": "internal/agentrun/loop.go"}
{"content": "package agentrun\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\t\"math/rand\"\n\t\"time\"\n\n\t\"github.com/abrander/zoo/internal/llm\"\n\t\"github.com/abrander/zoo/internal/tools\"\n)\n\n// Hooks lets a caller observe a runLoop invocation live, as the model\n// streams and tool calls execute, in addition to the final []llm.Message\n// it returns. Any of these may be nil.\ntype Hooks struct {\n\t// OnReasoningDelta and OnContentDelta fire with just the newly\n\t// streamed text for the current turn, not the accumulated total.\n\tOnReasoningDelta func(delta string)\n\tOnContentDelta   func(delta string)\n\n\t// OnTurnEnd fires once per completed streamer round-trip, after the\n\t// model's message for that turn is fully received and before any of\n\t// its tool calls run.\n\tOnTurnEnd func()\n\n\t// OnRetry fires before each backoff wait, i.e. whenever a turn's\n\t// round trip failed transiently and is about to be retried. attempt\n\t// is the 1-based number of the failed attempt, delay is the wait\n\t// that is about to happen, and err is the failure.\n\tOnRetry func(attempt int, delay time.Duration, err error)\n\n\t// OnTool fires once per tool call, after it has run.\n\tOnTool func(name, arguments, result string, toolErr bool)\n}\n\n// The retry budget for one LLM round trip (request + full stream read),\n// not for the whole run: a run can last hours and each turn is\n// independent, so a failing endpoint should cost at most a few seconds\n// of backoff per turn before the run surfaces the error.\nconst (\n\tllmMaxAttempts = 5\n\tllmBaseDelay   = time.Second\n\tllmMaxDelay    = 15 * time.Second\n)\n\n// llmSleep waits d, returning early with ctx's error if the run is\n// canceled or times out while waiting. Overridden in tests.\nvar llmSleep = func(ctx context.Context, d time.Duration) error {\n\tselect {\n\tcase <-time.After(d):\n\t\treturn nil\n\tcase <-ctx.Done():\n\t\treturn ctx.Err()\n\t}\n}\n\n// runLoop is a headless port of ../a's App.generate(): send messages +\n// tool defs, get a completion, run any tool_calls and append their\n// results, repeat until a plain finish or ctx is done. Transient LLM\n// endpoint failures (connection errors, 5xx/429 responses, mid-stream\n// breaks) are retried per turn with exponential backoff β€” see\n// requestTurn.\nfunc runLoop(ctx context.Context, client *llm.Client, toolsCtx tools.Context, messages []llm.Message, hooks Hooks) ([]llm.Message, error) {\n\tfor {\n\t\tif err := ctx.Err(); err != nil {\n\t\t\treturn messages, err\n\t\t}\n\n\t\tcompletion, err := requestTurn(ctx, client, messages, hooks)\n\t\tif err != nil {\n\t\t\treturn messages, err\n\t\t}\n\n\t\tchoice := completion.Choices[0]\n\t\tmessages = append(messages, choice.Message)\n\n\t\tif choice.FinishReason != \"tool_calls\" || len(choice.Message.ToolCalls) == 0 {\n\t\t\treturn messages, nil\n\t\t}\n\n\t\tfor _, tc := range choice.Message.ToolCalls {\n\t\t\tif err := ctx.Err(); err != nil {\n\t\t\t\treturn messages, err\n\t\t\t}\n\n\t\t\tresult, err := tools.Call(toolsCtx, tc.FunctionCall)\n\n\t\t\ttoolErr := err != nil\n\t\t\tif toolErr {\n\t\t\t\tresult = fmt.Sprintf(\"ERROR CALLING TOOL: %s\", err.Error())\n\t\t\t}\n\n\t\t\tif hooks.OnTool != nil {\n\t\t\t\thooks.OnTool(tc.FunctionCall.Name, tc.FunctionCall.Arguments, result, toolErr)\n\t\t\t}\n\n\t\t\tmessages = append(messages, llm.Message{\n\t\t\t\tRole:       \"tool\",\n\t\t\t\tContent:    result,\n\t\t\t\tToolCallID: tc.Id,\n\t\t\t\tName:       tc.FunctionCall.Name,\n\t\t\t})\n\t\t}\n\t}\n}\n\n// requestTurn performs one LLM round trip (request + full stream\n// read), retrying transient endpoint failures with exponential\n// backoff and jitter. It returns the completed turn's accumulated\n// completion.\nfunc requestTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) {\n\tfor attempt := 1; ; attempt++ {\n\t\tcompletion, err := streamTurn(ctx, client, messages, hooks)\n\t\tif err == nil {\n\t\t\treturn completion, nil\n\t\t}\n\n\t\t// A canceled or timed-out run is not a failing endpoint; stop\n\t\t// and report the run's own error.\n\t\tif ctxErr := ctx.Err(); ctxErr != nil {\n\t\t\treturn nil, ctxErr\n\t\t}\n\n\t\t// Permanent failures (client errors, malformed responses) and\n\t\t// exhausted budgets fail immediately.\n\t\tif !llm.IsTransient(err) || attempt == llmMaxAttempts {\n\t\t\tif attempt > 1 {\n\t\t\t\treturn nil, fmt.Errorf(\"llm endpoint failed after %d attempts: %w\", attempt, err)\n\t\t\t}\n\t\t\treturn nil, err\n\t\t}\n\n\t\tdelay := llmBackoff(attempt)\n\n\t\tif hooks.OnRetry != nil {\n\t\t\thooks.OnRetry(attempt, delay, err)\n\t\t}\n\n\t\tif err := llmSleep(ctx, delay); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n}\n\n// llmBackoff returns the delay before retrying after the given failed\n// attempt (1-based): exponential growth from llmBaseDelay, capped at\n// llmMaxDelay, with equal jitter (half fixed, half random) so\n// concurrent agents don't retry in lockstep.\nfunc llmBackoff(attempt int) time.Duration {\n\td := llmBaseDelay << (attempt - 1)\n\tif d <= 0 || d > llmMaxDelay {\n\t\td = llmMaxDelay\n\t}\n\treturn d/2 + time.Duration(rand.Int63n(int64(d/2)+1))\n}\n\n// streamTurn performs one attempt at an LLM round trip: issue the\n// request and read the stream to completion, firing hooks as content\n// arrives.\nfunc streamTurn(ctx context.Context, client *llm.Client, messages []llm.Message, hooks Hooks) (*llm.ChatCompletion, error) {\n\tstreamer, err := client.StreamChatCompletion(ctx, &llm.ChatCompletionRequest{\n\t\tMessages: messages,\n\t\tStream:   true,\n\t\tTools:    tools.All(),\n\t})\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"chat completion: %w\", err)\n\t}\n\n\tvar completion *llm.ChatCompletion\n\n\tvar prevContent, prevReasoning string\n\n\tfor {\n\t\tc, err := streamer.Get()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"stream completion: %w\", err)\n\t\t}\n\n\t\tcompletion = c\n\n\t\tif len(c.Choices) > 0 {\n\t\t\tmsg := c.Choices[0].Message\n\n\t\t\tif hooks.OnReasoningDelta != nil && len(msg.ReasoningContent) > len(prevReasoning) {\n\t\t\t\thooks.OnReasoningDelta(msg.ReasoningContent[len(prevReasoning):])\n\t\t\t}\n\t\t\tprevReasoning = msg.ReasoningContent\n\n\t\t\tif hooks.OnContentDelta != nil && len(msg.Content) > len(prevContent) {\n\t\t\t\thooks.OnContentDelta(msg.Content[len(prevContent):])\n\t\t\t}\n\t\t\tprevContent = msg.Content\n\t\t}\n\t}\n\n\tif hooks.OnTurnEnd != nil {\n\t\thooks.OnTurnEnd()\n\t}\n\n\tif completion == nil || len(completion.Choices) == 0 {\n\t\treturn nil, fmt.Errorf(\"model returned an empty completion\")\n\t}\n\n\treturn completion, nil\n}\n", "path": "internal/agentrun/loop.go"}
OK
Thinking
Next, I'll update streamHooks in run.go to implement OnRetry. I need to check the current exact text of the Hooks literal in order to edit it.
Now wire the `OnRetry` hook into `streamHooks` in run.go:
Tool πŸ”§ edit_file {"find": "\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},", "path": "internal/agentrun/run.go", "replace": "\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},\n\t\tOnRetry: func(attempt int, delay time.Duration, err error) {\n\t\t\t// Keep a trace of the failure in the job's history, so a\n\t\t\t// run that eventually succeeds (or fails after exhausting\n\t\t\t// the budget) still shows what the endpoint did.\n\t\t\tlogAppend(\"system\", fmt.Sprintf(\"LLM endpoint failure (attempt %d/%d): %s β€” retrying in %s\",\n\t\t\t\tattempt, llmMaxAttempts, truncateForLog(err.Error(), 200), delay.Round(time.Second)))\n\n\t\t\t// If part of the turn was already streamed, discard it from\n\t\t\t// the live view and the hub's replay buffer: it was never\n\t\t\t// persisted (that happens in OnTurnEnd), and the retried\n\t\t\t// turn regenerates it.\n\t\t\tif reasoningOpen || contentOpen {\n\t\t\t\treasoningOpen, contentOpen = false, false\n\t\t\t\treasoningBuf.Reset()\n\t\t\t\tcontentBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset})\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t}\n\t\t},"}
{"find": "\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},", "path": "internal/agentrun/run.go", "replace": "\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},\n\t\tOnRetry: func(attempt int, delay time.Duration, err error) {\n\t\t\t// Keep a trace of the failure in the job's history, so a\n\t\t\t// run that eventually succeeds (or fails after exhausting\n\t\t\t// the budget) still shows what the endpoint did.\n\t\t\tlogAppend(\"system\", fmt.Sprintf(\"LLM endpoint failure (attempt %d/%d): %s β€” retrying in %s\",\n\t\t\t\tattempt, llmMaxAttempts, truncateForLog(err.Error(), 200), delay.Round(time.Second)))\n\n\t\t\t// If part of the turn was already streamed, discard it from\n\t\t\t// the live view and the hub's replay buffer: it was never\n\t\t\t// persisted (that happens in OnTurnEnd), and the retried\n\t\t\t// turn regenerates it.\n\t\t\tif reasoningOpen || contentOpen {\n\t\t\t\treasoningOpen, contentOpen = false, false\n\t\t\t\treasoningBuf.Reset()\n\t\t\t\tcontentBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset})\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t}\n\t\t},"}
1 occurrences replaced
Thinking
Next, I'll add the `truncateForLog` helper to run.go. Let's add it near streamHooks. Also, I need to check the imports in run.go β€” `time` and `fmt` are already imported. Let's add the helper function after streamHooks.
Add the `truncateForLog` helper after `streamHooks`:
Tool πŸ”§ edit_file {"find": "\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}", "path": "internal/agentrun/run.go", "replace": "\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}\n\n// truncateForLog caps a log line at n runes, appending an ellipsis when\n// it is cut, so a verbose error cannot balloon a job log row.\nfunc truncateForLog(s string, n int) string {\n\tr := []rune(s)\n\tif len(r) <= n {\n\t\treturn s\n\t}\n\treturn string(r[:n]) + \"…\"\n}"}
{"find": "\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}", "path": "internal/agentrun/run.go", "replace": "\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}\n\n// truncateForLog caps a log line at n runes, appending an ellipsis when\n// it is cut, so a verbose error cannot balloon a job log row.\nfunc truncateForLog(s string, n int) string {\n\tr := []rune(s)\n\tif len(r) <= n {\n\t\treturn s\n\t}\n\treturn string(r[:n]) + \"…\"\n}"}
1 occurrences replaced
Thinking
Wait a sec β€” that edit matched the end of the OnTool hook (the last hook in the literal). Let me check whether it was applied at the correct location β€” the `r.hub.Checkpoint(jobID)\n\t\t},\n\t}` pattern... OnTurnEnd ends with `contentOpen = false\n\t\t\t}\n\t\t},` β€” no wait, that's not right. Let me re-read the end of the file to confirm.
Let me verify that edit landed in the right place:
Tool πŸ”§ read_file {"end_line": 385, "path": "internal/agentrun/run.go", "start_line": 281}
{"end_line": 385, "path": "internal/agentrun/run.go", "start_line": 281}
{"path":"internal/agentrun/run.go","content":"// streamHooks builds the Hooks a single Run passes to runLoop: every\n// delta is published live to the hub for connected dashboard viewers,\n// and once a reasoning/content block or tool call is complete, it's\n// persisted to the store as one row and the hub's replay buffer for\n// jobID is checkpointed β€” so a viewer connecting from this point on\n// sees it via the persisted history instead of a live replay, and is\n// never shown it twice.\nfunc (r *Runner) streamHooks(jobID string, logAppend func(stream, line string)) Hooks {\n\tvar reasoningBuf, contentBuf strings.Builder\n\n\treasoningOpen, contentOpen := false, false\n\n\treturn Hooks{\n\t\tOnReasoningDelta: func(delta string) {\n\t\t\tif !reasoningOpen {\n\t\t\t\treasoningOpen = true\n\t\t\t\treasoningBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningStart})\n\t\t\t}\n\n\t\t\treasoningBuf.WriteString(delta)\n\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningDelta, Text: delta})\n\t\t},\n\t\tOnContentDelta: func(delta string) {\n\t\t\tif !contentOpen {\n\t\t\t\tcontentOpen = true\n\t\t\t\tcontentBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentStart})\n\t\t\t}\n\n\t\t\tcontentBuf.WriteString(delta)\n\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentDelta, Text: delta})\n\t\t},\n\t\tOnTurnEnd: func() {\n\t\t\tif reasoningOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ReasoningEnd})\n\t\t\t\tlogAppend(\"reasoning\", reasoningBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\treasoningOpen = false\n\t\t\t}\n\n\t\t\tif contentOpen {\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.ContentEnd})\n\t\t\t\tlogAppend(\"content\", contentBuf.String())\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t\tcontentOpen = false\n\t\t\t}\n\t\t},\n\t\tOnRetry: func(attempt int, delay time.Duration, err error) {\n\t\t\t// Keep a trace of the failure in the job's history, so a\n\t\t\t// run that eventually succeeds (or fails after exhausting\n\t\t\t// the budget) still shows what the endpoint did.\n\t\t\tlogAppend(\"system\", fmt.Sprintf(\"LLM endpoint failure (attempt %d/%d): %s β€” retrying in %s\",\n\t\t\t\tattempt, llmMaxAttempts, truncateForLog(err.Error(), 200), delay.Round(time.Second)))\n\n\t\t\t// If part of the turn was already streamed, discard it from\n\t\t\t// the live view and the hub's replay buffer: it was never\n\t\t\t// persisted (that happens in OnTurnEnd), and the retried\n\t\t\t// turn regenerates it.\n\t\t\tif reasoningOpen || contentOpen {\n\t\t\t\treasoningOpen, contentOpen = false, false\n\t\t\t\treasoningBuf.Reset()\n\t\t\t\tcontentBuf.Reset()\n\t\t\t\tr.hub.Publish(jobID, livelog.Event{Type: livelog.TurnReset})\n\t\t\t\tr.hub.Checkpoint(jobID)\n\t\t\t}\n\t\t},\n\t\tOnTool: func(name, arguments, result string, toolErr bool) {\n\t\t\tr.hub.Publish(jobID, livelog.Event{\n\t\t\t\tType:      livelog.Tool,\n\t\t\t\tName:      name,\n\t\t\t\tArguments: arguments,\n\t\t\t\tResult:    result,\n\t\t\t\tError:     toolErr,\n\t\t\t})\n\n\t\t\tline, err := json.Marshal(store.ToolLogEntry{Name: name, Arguments: arguments, Result: result, Error: toolErr})\n\t\t\tif err != nil {\n\t\t\t\tr.logger.Warn(\"failed to marshal tool log entry\", \"job\", jobID, \"error\", err)\n\t\t\t} else {\n\t\t\t\tlogAppend(\"tool\", string(line))\n\t\t\t}\n\n\t\t\tr.hub.Checkpoint(jobID)\n\t\t},\n\t}\n}\n\n// truncateForLog caps a log line at n runes, appending an ellipsis when\n// it is cut, so a verbose error cannot balloon a job log row.\nfunc truncateForLog(s string, n int) string {\n\tr := []rune(s)\n\tif len(r) \u003c= n {\n\t\treturn s\n\t}\n\treturn string(r[:n]) + \"…\"\n}\n\nfunc seedMessage(ev forgejo.Event, branch, defaultBranch, instructions string, comments []forgejo.IssueComment, review *forgejo.ReviewDetail) string {\n\traw, _ := json.MarshalIndent(json.RawMessage(ev.Raw), \"\", \"  \")\n\n\tvar instructionsSection string\n\tif instructions != \"\" {\n\t\tinstructionsSection = fmt.Sprintf(\"Instructions for this event, from zoo.hcl:\\n%s\\n\\n\", instructions)\n\t}","start_line":281,"end_line":385,"total_lines":466,"truncated":true}
Thinking
run.go looks fine. Next: 1. Add `TurnReset` to the livelog hub. 2. Add a JS case for `turn_reset` to templates.go.
Now the `TurnReset` event type in livelog and the dashboard JS handling:
Tool πŸ”§ edit_file {"find": "\tContentEnd Type = \"content_end\"\n\tTool Type = \"tool\"\n\tStatus Type = \"status\"\n)", "path": "internal/livelog/hub.go", "replace": "\tContentEnd Type = \"content_end\"\n\tTool Type = \"tool\"\n\tStatus Type = \"status\"\n\n\t// TurnReset tells live viewers that a turn's stream broke and is\n\t// being retried: any partial reasoning/content block on screen\n\t// belongs to the failed attempt and should be dropped.\n\tTurnReset Type = \"turn_reset\"\n)"}
{"find": "\tContentEnd   Type = \"content_end\"\n\tTool         Type = \"tool\"\n\tStatus       Type = \"status\"\n)", "path": "internal/livelog/hub.go", "replace": "\tContentEnd   Type = \"content_end\"\n\tTool         Type = \"tool\"\n\tStatus       Type = \"status\"\n\n\t// TurnReset tells live viewers that a turn's stream broke and is\n\t// being retried: any partial reasoning/content block on screen\n\t// belongs to the failed attempt and should be dropped.\n\tTurnReset Type = \"turn_reset\"\n)"}
ERROR CALLING TOOL: "\tContentEnd   Type = \"content_end\"\n\tTool         Type = \"tool\"\n\tStatus       Type = \"status\"\n)" not found in "internal/livelog/hub.go"
Tool πŸ”§ edit_file {"find": " case \"tool\":\n newToolBlock(ev);\n break;\n case \"status\":", "path": "internal/web/templates.go", "replace": " case \"tool\":\n newToolBlock(ev);\n break;\n case \"turn_reset\":\n // The turn's stream broke and is being retried; drop the\n // partial block so the retried turn starts clean.\n if (reasoningBody) { reasoningBody.parentElement.remove(); reasoningBody = null; }\n if (contentBody) { contentBody.parentElement.remove(); contentBody = null; }\n break;\n case \"status\":"}
{"find": "          case \"tool\":\n            newToolBlock(ev);\n            break;\n          case \"status\":", "path": "internal/web/templates.go", "replace": "          case \"tool\":\n            newToolBlock(ev);\n            break;\n          case \"turn_reset\":\n            // The turn's stream broke and is being retried; drop the\n            // partial block so the retried turn starts clean.\n            if (reasoningBody) { reasoningBody.parentElement.remove(); reasoningBody = null; }\n            if (contentBody) { contentBody.parentElement.remove(); contentBody = null; }\n            break;\n          case \"status\":"}
1 occurrences replaced
Thinking
The edit to hub.go failed β€” probably due to a whitespace mismatch. Let me try reading the exact lines around the constants in hub.go.
Tool πŸ”§ read_file {"end_line": 22, "path": "internal/livelog/hub.go", "start_line": 11}
{"end_line": 22, "path": "internal/livelog/hub.go", "start_line": 11}
{"path":"internal/livelog/hub.go","content":"\nconst (\n\tReasoningStart Type = \"reasoning_start\"\n\tReasoningDelta Type = \"reasoning_delta\"\n\tReasoningEnd   Type = \"reasoning_end\"\n\tContentStart   Type = \"content_start\"\n\tContentDelta   Type = \"content_delta\"\n\tContentEnd     Type = \"content_end\"\n\tTool           Type = \"tool\"\n\tStatus         Type = \"status\"\n)\n","start_line":11,"end_line":22,"total_lines":164,"truncated":true}
Thinking
The alignment is different (extra spaces). Let me redo the edit with the exact text.
Tool πŸ”§ edit_file {"find": "\tTool Type = \"tool\"\n\tStatus Type = \"status\"\n)", "path": "internal/livelog/hub.go", "replace": "\tTool Type = \"tool\"\n\tStatus Type = \"status\"\n\n\t// TurnReset tells live viewers that a turn's stream broke and is\n\t// being retried: any partial reasoning/content block on screen\n\t// belongs to the failed attempt and should be dropped.\n\tTurnReset Type = \"turn_reset\"\n)"}
{"find": "\tTool           Type = \"tool\"\n\tStatus         Type = \"status\"\n)", "path": "internal/livelog/hub.go", "replace": "\tTool           Type = \"tool\"\n\tStatus         Type = \"status\"\n\n\t// TurnReset tells live viewers that a turn's stream broke and is\n\t// being retried: any partial reasoning/content block on screen\n\t// belongs to the failed attempt and should be dropped.\n\tTurnReset Type = \"turn_reset\"\n)"}
1 occurrences replaced
Thinking
Alright, let's build and verify so far, and then write the tests.
Now let's verify everything compiles before writing tests:
Tool πŸ”§ bash {"command": "cd /project && go build ./... && go vet ./..."}
{"command": "cd /project && go build ./... && go vet ./..."}
go: downloading github.com/docker/docker v28.5.2+incompatible
go: downloading github.com/hashicorp/hcl/v2 v2.24.0
go: downloading codeberg.org/mvdkleijn/forgejo-sdk/forgejo/v2 v2.2.0
go: downloading github.com/google/uuid v1.6.0
go: downloading modernc.org/sqlite v1.56.0
go: downloading github.com/docker/go-connections v0.8.1
go: downloading github.com/docker/go-units v0.5.0
go: downloading github.com/moby/docker-image-spec v1.3.1
go: downloading github.com/opencontainers/image-spec v1.1.1
go: downloading github.com/containerd/errdefs v1.0.0
go: downloading github.com/containerd/errdefs/pkg v0.3.0
go: downloading github.com/distribution/reference v0.6.0
go: downloading github.com/opencontainers/go-digest v1.0.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0
go: downloading go.opentelemetry.io/otel/trace v1.45.0
go: downloading go.opentelemetry.io/otel v1.45.0
go: downloading github.com/42wim/httpsig v1.2.3
go: downloading github.com/go-fed/httpsig v1.1.0
go: downloading github.com/hashicorp/go-version v1.7.0
go: downloading golang.org/x/crypto v0.39.0
go: downloading github.com/felixge/httpsnoop v1.1.0
go: downloading go.opentelemetry.io/otel/metric v1.45.0
go: downloading github.com/agext/levenshtein v1.2.1
go: downloading github.com/apparentlymart/go-textseg/v15 v15.0.0
go: downloading github.com/mitchellh/go-wordwrap v1.0.1
go: downloading github.com/zclconf/go-cty v1.16.3
go: downloading github.com/go-logr/logr v1.4.4
go: downloading github.com/google/go-cmp v0.7.0
go: downloading golang.org/x/sys v0.47.0
go: downloading github.com/go-logr/stdr v1.2.2
go: downloading go.opentelemetry.io/auto/sdk v1.2.1
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading golang.org/x/text v0.40.0
go: downloading modernc.org/libc v1.74.4
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading modernc.org/mathutil v1.7.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
Thinking
The build succeeded (only download messages). Next, I'll write the tests. 1. `internal/llm/errors_test.go` β€” classification tests. 2. Additions to `internal/llm/streamer_test.go` β€” error type tests. 3. `internal/agentrun/loop_test.go` β€” retry behavior tests.
Build passes. Now the tests β€” first the llm error classification and streamer error types: