add a function that every so often it sumerizes what has been done and what is still needed to be done so you reduce local and online models going off track. and also reduce the prompt size from getting too big.
Thanks, this is already in doing and some implementation already done. Coming soon in main branch.
See e.g. for more information:
stackblitz-labs:main ← thecodacus:context-selection
opened 06:22PM - 14 Jan 25 UTC
# LLM Context Optimization Implementation
## Overview
This PR implements int… elligent context management for LLM interactions by introducing two key optimizations:
1. Code Context Optimization: Dynamically selects relevant files for the context buffer based on the current conversation
2. Chat Context Optimization: Summarizes chat history to maintain context while reducing token usage
PR attempts to solve : https://github.com/stackblitz-labs/bolt.diy/issues/1041
## Key Changes
### 1. Context Buffer Management
- Added new system for selecting relevant files using LLM analysis
- Implemented file filtering with ignore patterns similar to .gitignore
- Limited context buffer to 5 files maximum for efficient token usage
- Added support for relative and absolute file paths
### 2. Chat Summarization
- Implemented chat history summarization to compress context while preserving key information
- Created progress tracking for multi-step context optimization process
- Added annotation system for tracking context and summary metadata
## Technical Details
### Context Selection System
Key implementation details from `select-context.ts`:
```typescript
export async function selectContext(props: {
messages: Message[];
env: Env;
apiKeys?: Record<string, string>;
files: FileMap;
providerSettings?: Record<string, IProviderSetting>;
promptId?: string;
contextOptimization?: boolean;
summary: string;
}) {
// ... initialization ...
const resp = await generateText({
system: `
You are a software engineer. You are working on a project. You need to select files that are relevant to the task from the list of files above.
`,
prompt: `
${summaryText}
Users Question: ${processedMessages.filter((x) => x.role == 'user').pop()?.content}
`
});
// Parse response for file selection
const updateContextBuffer = response.match(/<updateContextBuffer>([\s\S]*?)<\/updateContextBuffer>/);
}
```
### Chat Summary Generation
From `create-summary.ts`:
```typescript
export async function createSummary(props: {
messages: Message[];
env: Env;
apiKeys?: Record<string, string>;
providerSettings?: Record<string, IProviderSetting>;
}) {
// ... initialization ...
const extractTextContent = (message: Message) =>
Array.isArray(message.content)
? (message.content.find((item) => item.type === 'text')?.text as string) || ''
: message.content;
const resp = await generateText({
system: `
You are a software engineer. You need to summarize the work till now and provide a summary of the chat.
`,
prompt: `
please provide a summary of the chat till now.
below is the latest chat:
${slicedMessages.map((x) => `[${x.role}] ${extractTextContent(x)}`).join('\n')}
`
});
}
```
### Progress Tracking
Implementation of progress annotations in `api.chat.ts`:
```typescript
type ProgressAnnotation = {
type: 'progress';
value: number;
message: string;
};
// Usage in stream
dataStream.writeMessageAnnotation({
type: 'progress',
value: progressCounter++,
message: 'Generating Chat Summary'
} as ProgressAnnotation);
```
## Migration Impact
- Breaking changes to the chat API response format to include progress and context annotations
- New type definitions for context and progress annotations
- Updated stream handling to support multi-stage processing
## Future Improvements
- Add configurable context buffer size limits
- Add support for custom ignore patterns
- Add support for partial file content selection
1 Like