Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8540cdb
feat(web): add language model inputModalities capability plumbing
whoisthey Jun 26, 2026
a473b49
docs: add CHANGELOG entry for language model inputModalities
whoisthey Jun 26, 2026
4b57d27
refactor(schemas): split document types out of inputModalities
whoisthey Jun 26, 2026
0baabcb
docs(schemas): clarify what counts as a document type
whoisthey Jun 26, 2026
5e4045b
fix(web): widen getLanguageModelKey param to keyable subset
whoisthey Jun 26, 2026
507d758
chore(schemas,web): keep schema dist fresh and resolve types from source
whoisthey Jun 26, 2026
ed307ed
Merge branch 'main' into whoisthey/language-model-input-modalities
whoisthey Jun 26, 2026
a1aeb37
First pass file attachments, picker and drag and drop, with preview c…
whoisthey Jun 26, 2026
18ff55e
Merge branch 'main' into whoisthey/language-model-input-modalities
whoisthey Jun 26, 2026
ebd573f
Merge branch 'whoisthey/language-model-input-modalities' into whoisth…
whoisthey Jun 26, 2026
5d69749
escape key handle for modal and add missing description component
whoisthey Jun 26, 2026
c52e1cf
Merge branch 'main' into whoisthey/text-file-attachments
whoisthey Jun 27, 2026
de291cc
refactor(web): resolve model capabilities from models.dev, not config…
whoisthey Jun 27, 2026
d00f02e
Merge branch 'whoisthey/language-model-input-modalities' into whoisth…
whoisthey Jun 27, 2026
fb8843b
paste-to-attachment handling, raw paste keychord enabled, toast with …
whoisthey Jun 27, 2026
bf79260
stronger typing for contract
whoisthey Jun 27, 2026
dbcfc8a
remove blocking models.dev catalog request and add cache warm on startup
whoisthey Jun 27, 2026
7ba297b
cleanup warming
whoisthey Jun 27, 2026
bc028ae
Merge branch 'whoisthey/language-model-input-modalities' into whoisth…
whoisthey Jun 27, 2026
070f832
remove button from toast
whoisthey Jun 27, 2026
c050eb9
Merge branch 'main' into whoisthey/language-model-input-modalities
whoisthey Jun 27, 2026
3d22e2a
Merge branch 'whoisthey/language-model-input-modalities' into whoisth…
whoisthey Jun 27, 2026
a62a25b
add separate state to track pending attachments to avoid visual flick…
whoisthey Jun 27, 2026
38472b7
Merge branch 'main' into whoisthey/text-file-attachments
whoisthey Jun 27, 2026
af8a9db
explicitly import already hoisted uuid and change bad crypto call for…
whoisthey Jun 27, 2026
2222dba
add changelog entry
whoisthey Jun 27, 2026
14a237d
remove granular file attachment controls in favor of single per-messa…
whoisthey Jun 27, 2026
cb8181d
pass attachments through the login/upgrade redirect
whoisthey Jun 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- [EE] Added mermaid diagram rendering to Ask Sourcebot answers, with pan/zoom, copy/export, in-thread deep links, and an interleaved right-panel view. [#1369](https://github.com/sourcebot-dev/sourcebot/pull/1369)
- [EE] Added a context-window usage gauge to the Ask Sourcebot chat details, showing how much of the selected model's context window each turn occupies. Window sizes are resolved from the models.dev catalog. [#1370](https://github.com/sourcebot-dev/sourcebot/pull/1370)
- Added language model input-modality and document capability resolution, automatically resolved from the models.dev catalog (falls back to text-only for uncatalogued/self-hosted models). [#1372](https://github.com/sourcebot-dev/sourcebot/pull/1372)
- [EE] Added text file attachments to Ask Sourcebot, letting users attach text/code/config files to a chat message via the paperclip button, drag-and-drop, or paste, with large pastes auto-converted to attachments. [#1374](https://github.com/sourcebot-dev/sourcebot/pull/1374)

### Fixed
- Send anonymous server-side PostHog events as personless so unauthenticated requests don't inflate person counts. [#1367](https://github.com/sourcebot-dev/sourcebot/pull/1367)
Expand Down
2 changes: 2 additions & 0 deletions packages/web/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@
"react-day-picker": "^9.14.0",
"react-device-detect": "^2.2.3",
"react-dom": "19.2.4",
"react-dropzone": "^15.0.0",
"react-hook-form": "^7.53.0",
"react-hotkeys-hook": "^4.5.1",
"react-icons": "^5.6.0",
Expand All @@ -200,6 +201,7 @@
"tailwindcss-animate": "^1.0.7",
"use-stick-to-bottom": "^1.1.3",
"usehooks-ts": "^3.1.0",
"uuid": "^14.0.0",
"vscode-icons-js": "^11.6.1",
"zod": "^3.25.76",
"zod-to-json-schema": "^3.24.5"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,15 @@
import Image from 'next/image';
import { SearchModeSelector } from "@/app/(app)/components/searchModeSelector";
import { Separator } from "@/components/ui/separator";
import { ChatBox } from "@/features/chat/components/chatBox";
import { ChatBox, ChatBoxHandle } from "@/features/chat/components/chatBox";
import { ChatBoxToolbar } from "@/features/chat/components/chatBox/chatBoxToolbar";
import { ChatPaneDropzone } from "@/features/chat/components/chatBox/chatPaneDropzone";
import { NotConfiguredErrorBanner } from "@/features/chat/components/notConfiguredErrorBanner";
import { LanguageModelInfo, RepoSearchScope } from "@/features/chat/types";
import { useCreateNewChatThread } from "@/features/chat/useCreateNewChatThread";
import { DISABLED_MCP_SERVER_IDS_LOCAL_STORAGE_KEY } from "@/features/chat/constants";
import { getRepoImageSrc } from '@/lib/utils';
import { useMemo, useState } from "react";
import { useMemo, useRef, useState } from "react";
import { useLocalStorage } from "usehooks-ts";

interface LandingPageProps {
Expand All @@ -33,6 +34,7 @@ export const LandingPage = ({
const { createNewChatThread, isLoading } = useCreateNewChatThread();
const [isContextSelectorOpen, setIsContextSelectorOpen] = useState(false);
const [disabledMcpServerIds, setDisabledMcpServerIds] = useLocalStorage<string[]>(DISABLED_MCP_SERVER_IDS_LOCAL_STORAGE_KEY, [], { initializeWithValue: false });
const chatBoxRef = useRef<ChatBoxHandle>(null);
const isChatBoxDisabled = languageModels.length === 0;

const selectedSearchScopes = useMemo(() => [
Expand Down Expand Up @@ -67,11 +69,16 @@ export const LandingPage = ({
</div>

{/* ChatBox */}
<div className="w-full">
<ChatPaneDropzone
className="w-full"
onFilesDropped={(files) => chatBoxRef.current?.addFiles(files)}
disabled={isChatBoxDisabled}
>
<div className="border rounded-md w-full shadow-sm">
<ChatBox
onSubmit={(children) => {
createNewChatThread(children, selectedSearchScopes, disabledMcpServerIds);
ref={chatBoxRef}
onSubmit={(children, _editor, attachments) => {
createNewChatThread(children, selectedSearchScopes, disabledMcpServerIds, attachments);
Comment thread
whoisthey marked this conversation as resolved.
}}
className="min-h-[50px]"
isRedirecting={isLoading}
Expand Down Expand Up @@ -107,7 +114,7 @@ export const LandingPage = ({
{isChatBoxDisabled && (
<NotConfiguredErrorBanner className="mt-4" />
)}
</div>
</ChatPaneDropzone>
</div>
</div>
)
Expand Down
5 changes: 3 additions & 2 deletions packages/web/src/app/(app)/chat/chatLandingPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { CustomSlateEditor } from "@/features/chat/customSlateEditor";
import { ServiceErrorException } from "@/lib/serviceError";
import { isServiceError, measure } from "@/lib/utils";
import { LandingPageChatBox } from "./components/landingPageChatBox";
import { ChatLandingDropzone } from "./components/chatLandingDropzone";
import { RepositoryCarousel } from "../components/repositoryCarousel";
import { Separator } from "@/components/ui/separator";
import { DemoCards } from "./components/demoCards";
Expand Down Expand Up @@ -56,7 +57,7 @@ export async function ChatLandingPage() {
})() : undefined;

return (
<div className="flex flex-col items-center h-full overflow-hidden">
<ChatLandingDropzone disabled={languageModels.length === 0}>
<div className="flex flex-col items-center h-full overflow-y-auto pt-8 pb-8 md:pt-16 w-full px-5">
<div className="max-h-44 w-auto">
<SourcebotLogo
Expand Down Expand Up @@ -92,6 +93,6 @@ export async function ChatLandingPage() {
</>
)}
</div>
</div>
</ChatLandingDropzone>
)
}
44 changes: 44 additions & 0 deletions packages/web/src/app/(app)/chat/components/chatLandingDropzone.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
'use client';

import { ChatBoxHandle } from "@/features/chat/components/chatBox";
import { ChatPaneDropzone } from "@/features/chat/components/chatBox/chatPaneDropzone";
import { createContext, ReactNode, useCallback, useContext, useRef } from "react";

type RegisterChatBoxHandle = (handle: ChatBoxHandle | null) => void;

const LandingChatBoxContext = createContext<RegisterChatBoxHandle | null>(null);

// Lets the (nested) landing chat box register its imperative handle so the
// pane-level drop zone can forward dropped files into it. Returns a no-op when
// rendered outside the provider.
export const useRegisterLandingChatBox = (): RegisterChatBoxHandle => {
return useContext(LandingChatBoxContext) ?? (() => { });
}

interface ChatLandingDropzoneProps {
disabled?: boolean;
children: ReactNode;
}

// Wraps the entire unstarted-chat landing pane in a drag-and-drop target.
// The chat box lives deeper in the tree (and behind a server/client boundary),
// so it registers its handle via context rather than a direct ref.
export const ChatLandingDropzone = ({ disabled, children }: ChatLandingDropzoneProps) => {
const handleRef = useRef<ChatBoxHandle | null>(null);

const register = useCallback<RegisterChatBoxHandle>((handle) => {
handleRef.current = handle;
}, []);

return (
<LandingChatBoxContext.Provider value={register}>
<ChatPaneDropzone
className="flex flex-col items-center h-full overflow-hidden"
onFilesDropped={(files) => handleRef.current?.addFiles(files)}
disabled={disabled}
>
{children}
</ChatPaneDropzone>
</LandingChatBoxContext.Provider>
)
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { LanguageModelInfo, SearchScope } from "@/features/chat/types";
import { useCreateNewChatThread } from "@/features/chat/useCreateNewChatThread";
import { RepositoryQuery, SearchContextQuery } from "@/lib/types";
import { useState } from "react";
import { useRegisterLandingChatBox } from "./chatLandingDropzone";
import { useLocalStorage } from "usehooks-ts";
import { DISABLED_MCP_SERVER_IDS_LOCAL_STORAGE_KEY, SELECTED_SEARCH_SCOPES_LOCAL_STORAGE_KEY } from "@/features/chat/constants";
import { SearchModeSelector } from "../../components/searchModeSelector";
Expand All @@ -31,14 +32,16 @@ export const LandingPageChatBox = ({
const [selectedSearchScopes, setSelectedSearchScopes] = useLocalStorage<SearchScope[]>(SELECTED_SEARCH_SCOPES_LOCAL_STORAGE_KEY, [], { initializeWithValue: false });
const [disabledMcpServerIds, setDisabledMcpServerIds] = useLocalStorage<string[]>(DISABLED_MCP_SERVER_IDS_LOCAL_STORAGE_KEY, [], { initializeWithValue: false });
const [isContextSelectorOpen, setIsContextSelectorOpen] = useState(false);
const registerChatBox = useRegisterLandingChatBox();
const isChatBoxDisabled = languageModels.length === 0;

return (
<div className="w-full max-w-[800px] mt-4">
<div className="border rounded-md w-full shadow-sm">
<ChatBox
onSubmit={(children) => {
createNewChatThread(children, selectedSearchScopes, disabledMcpServerIds);
ref={registerChatBox}
onSubmit={(children, _editor, attachments) => {
createNewChatThread(children, selectedSearchScopes, disabledMcpServerIds, attachments);
}}
className="min-h-[50px]"
isRedirecting={isLoading}
Expand Down Expand Up @@ -74,6 +77,6 @@ export const LandingPageChatBox = ({
{isChatBoxDisabled && (
<NotConfiguredErrorBanner className="mt-4" />
)}
</div >
</div>
)
}
11 changes: 9 additions & 2 deletions packages/web/src/ee/features/chat/agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ import { randomUUID } from "crypto";
import _dedent from "dedent";
import { ANSWER_TAG, FILE_REFERENCE_PREFIX } from "@/features/chat/constants";
import { Source } from "@/features/chat/types";
import { addLineNumbers, fileReferenceToString, getAnswerPartFromAssistantMessage, getTurnProgressState, getUserMessageText } from "@/features/chat/utils";
import { addLineNumbers, fileReferenceToString, formatAttachmentsForPrompt, getAnswerPartFromAssistantMessage, getTurnProgressState, getUserMessageAttachments, getUserMessageText } from "@/features/chat/utils";
import { createTools } from "./tools";
import { getConnectedMcpClients } from "@/ee/features/chat/mcp/mcpClientFactory";
import { getMcpTools, McpToolsResult } from "@/ee/features/chat/mcp/mcpToolSets";
Expand Down Expand Up @@ -105,9 +105,16 @@ export const createMessageStream = async ({
let messageHistory: ModelMessage[] =
messages.map((message, index): ModelMessage | undefined => {
if (message.role === 'user') {
// Fold inline-text attachments into this turn's content (not the
// system prompt) so they stay bound to their turn, re-emitted from
// the persisted parts.
const text = getUserMessageText(message);
Comment thread
brendan-kellam marked this conversation as resolved.
const attachmentsBlock = formatAttachmentsForPrompt(
getUserMessageAttachments(message),
);
return {
role: 'user',
content: getUserMessageText(message),
content: attachmentsBlock ? `${text}\n\n${attachmentsBlock}` : text,
};
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { useToast } from '@/components/hooks/use-toast';
import { Button } from '@/components/ui/button';
import { Separator } from '@/components/ui/separator';
import { CustomSlateEditor } from '@/features/chat/customSlateEditor';
import { AdditionalChatRequestParams, CustomEditor, LanguageModelInfo, SBChatMessage, SearchScope, Source } from '@/features/chat/types';
import { AdditionalChatRequestParams, AttachmentData, CustomEditor, LanguageModelInfo, SBChatMessage, SearchScope, Source } from '@/features/chat/types';
import { createUIMessage, getAllMentionElements, getTurnProgressState, getUserMessageText, resetEditor, slateContentToString } from '@/features/chat/utils';
import { useChat } from '@ai-sdk/react';
import { CreateUIMessage, DefaultChatTransport, lastAssistantMessageIsCompleteWithApprovalResponses } from 'ai';
Expand All @@ -15,8 +15,9 @@ import { useStickToBottom } from 'use-stick-to-bottom';
import { Descendant } from 'slate';
import { useMessagePairs } from '../../useMessagePairs';
import { useSelectedLanguageModel } from '@/features/chat/useSelectedLanguageModel';
import { ChatBox } from '@/features/chat/components/chatBox';
import { ChatBox, ChatBoxHandle } from '@/features/chat/components/chatBox';
import { ChatBoxToolbar } from '@/features/chat/components/chatBox/chatBoxToolbar';
import { ChatPaneDropzone } from '@/features/chat/components/chatBox/chatPaneDropzone';
import { ChatThreadListItem } from './chatThreadListItem';
import { ErrorBanner } from './errorBanner';
import { McpFailedServersBanner } from './mcpFailedServersBanner';
Expand Down Expand Up @@ -72,6 +73,7 @@ export const ChatThread = ({
}: ChatThreadProps) => {
const [isErrorBannerVisible, setIsErrorBannerVisible] = useState(false);
const hasSubmittedInputMessage = useRef(false);
const chatBoxRef = useRef<ChatBoxHandle>(null);
const { scrollRef, contentRef, scrollToBottom, isAtBottom } = useStickToBottom({ initial: false });
const { toast } = useToast();
const router = useRouter();
Expand Down Expand Up @@ -347,11 +349,11 @@ export const ChatThread = ({
}
}, [error]);

const onSubmit = useCallback(async (children: Descendant[], editor: CustomEditor) => {
const onSubmit = useCallback(async (children: Descendant[], editor: CustomEditor, attachments: AttachmentData[]) => {
const text = slateContentToString(children);
const mentions = getAllMentionElements(children);

const message = createUIMessage(text, mentions.map(({ data }) => data), selectedSearchScopes, disabledMcpServerIds);
const message = createUIMessage(text, mentions.map(({ data }) => data), selectedSearchScopes, disabledMcpServerIds, attachments);
sendMessage(message);

scrollToBottom();
Expand Down Expand Up @@ -381,6 +383,11 @@ export const ChatThread = ({
return (
<ToolApprovalProvider value={addToolApprovalResponse}>
<McpServerIconContext.Provider value={mcpServerIconMap}>
<ChatPaneDropzone
className="flex flex-col flex-1 min-h-0 w-full"
onFilesDropped={(files) => chatBoxRef.current?.addFiles(files)}
disabled={!isOwner || languageModels.length === 0}
>
{error && (
<ErrorBanner
error={error}
Expand Down Expand Up @@ -470,6 +477,7 @@ export const ChatThread = ({
<div className="border rounded-md w-full shadow-sm">
<CustomSlateEditor>
<ChatBox
ref={chatBoxRef}
onSubmit={onSubmit}
className="min-h-[80px]"
preferredSuggestionsBoxPlacement="top-start"
Expand Down Expand Up @@ -520,6 +528,7 @@ export const ChatThread = ({
</div>
)}
</div>
</ChatPaneDropzone>
</McpServerIconContext.Provider>
</ToolApprovalProvider>
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@
import scrollIntoView from 'scroll-into-view-if-needed';
import { Reference, referenceSchema, SBChatMessage, Source } from "@/features/chat/types";
import { useExtractReferences } from '../../useExtractReferences';
import { getAnswerPartFromAssistantMessage, getLastStepParts, getUserMessageText, groupMessageIntoSteps, isSBChatToolPart, repairReferences } from '@/features/chat/utils';
import { getAnswerPartFromAssistantMessage, getLastStepParts, getUserMessageAttachments, getUserMessageText, groupMessageIntoSteps, isSBChatToolPart, repairReferences } from '@/features/chat/utils';
import { AnswerCard } from './answerCard';
import { MessageAttachments } from './messageAttachments';
import { DetailsCard } from './detailsCard';
import { ApprovalRequestedToolPart, ToolApprovalBanner } from './toolApprovalBanner';
import { MarkdownRenderer, REFERENCE_PAYLOAD_ATTRIBUTE } from './markdownRenderer';
Expand Down Expand Up @@ -68,6 +69,10 @@
return getUserMessageText(userMessage);
}, [userMessage]);

const userAttachments = useMemo(() => {
return getUserMessageAttachments(userMessage);
}, [userMessage]);

// Take the assistant message and repair any references that are not properly formatted.
// This applies to parts that are text (i.e., text & reasoning).
const assistantMessage = useMemo(() => {
Expand Down Expand Up @@ -272,7 +277,7 @@
markdownRenderer.removeEventListener('mouseout', handleMouseOut);
markdownRenderer.removeEventListener('click', handleClick);
};
}, [answerPart, selectedReference?.id]); // Re-run when answerPart changes to ensure we catch new content

Check warning on line 280 in packages/web/src/ee/features/chat/components/chatThread/chatThreadListItem.tsx

View workflow job for this annotation

GitHub Actions / lint

React Hook useEffect has missing dependencies: 'setHoveredReference' and 'setSelectedReference'. Either include them or remove the dependency array

// When the selected reference changes, highlight all associated reference elements
// and scroll to the nearest one, if needed.
Expand Down Expand Up @@ -418,27 +423,30 @@
ref={leftPanelRef}
className="py-4 h-full"
>
<div className="flex flex-row gap-2 mb-4">
{isTurnInProgress ? (
<Loader2 className="w-4 h-4 animate-spin flex-shrink-0 mt-1.5" />
) : (
<CheckCircle className="w-4 h-4 text-green-700 flex-shrink-0 mt-1.5" />
<div className="mb-4">
{userAttachments.length > 0 && (
<MessageAttachments attachments={userAttachments} className="mb-1.5 ml-6" />
)}
<MarkdownRenderer
content={userQuestion.trim()}
className="prose-p:m-0"
escapeHtml={true}
/>

<div className="flex flex-row gap-2">
{isTurnInProgress ? (
<Loader2 className="w-4 h-4 animate-spin flex-shrink-0 mt-1.5" />
) : (
<CheckCircle className="w-4 h-4 text-green-700 flex-shrink-0 mt-1.5" />
)}
<MarkdownRenderer
content={userQuestion.trim()}
className="prose-p:m-0"
escapeHtml={true}
/>
</div>
</div>

{isThinking && (
<div className="space-y-4 mb-4">
<Skeleton className="h-4 max-w-32" />
<div className="space-y-2">
<Skeleton className="h-3 max-w-72" />
<Skeleton className="h-3 max-w-64" />
<Skeleton className="h-3 max-w-56" />
</div>
<div className="space-y-2 mb-4">
<Skeleton className="h-3 w-full max-w-80" />
<Skeleton className="h-3 w-full max-w-72" />
<Skeleton className="h-3 w-full max-w-56" />
</div>
)}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,7 @@ export const StepPartRenderer = ({ part, toolTokenUsageMap }: { part: SBChatMess
case 'data-source':
case 'data-mcp-server':
case 'data-mcp-failed-server':
case 'data-attachment':
case 'file':
case 'source-document':
case 'source-url':
Expand Down
Loading
Loading