1. The Case for Local-First AI in Software Engineering
While frontier cloud models (such as Claude 3.5 Sonnet and GPT-4o) offer state-of-the-art reasoning, relying exclusively on cloud APIs poses significant drawbacks for enterprise and privacy-conscious developers:
- Data Sovereignty & IP Protection: Proprietary intellectual property, proprietary encryption keys, and internal database schemas cannot be transmitted over public internet connections.
- Offline Operation: Developers who travel, or who work on restricted networks, need code assistance that does not depend on a cloud service. A local model answers without internet access for the chat itself.
- Unbounded Inference Costs: Iterative agentic loops generating hundreds of compiler-assisted test runs can rapidly accumulate API costs if routed to expensive cloud reasoning models.
With modern open-weight coding models like Qwen 2.5 Coder (7B, 14B, 32B) and DeepSeek R1 Distill, running high-capability coding models locally on modern GPUs or Apple Silicon has become practical.
2. Generic OpenAI-Compatible Endpoint Architecture
Rather than building custom proprietary drivers for every local runtime, the AI open-source community converged on the OpenAI-compatible HTTP REST specification (POST /v1/chat/completions). Local inference servers expose this endpoint locally on loopback addresses:
- Ollama: Default binding at
http://localhost:11434/v1 - LM Studio: Default binding at
http://localhost:1234/v1 - vLLM / LocalAI: Default binding at
http://localhost:8000/v1
openai-compatible.cjs)
http://localhost:11434/v1/chat/completions3. Proxima's OpenAI-Compatible Adapter Implementation
In the Proxima repository, generic local model routing is implemented in proxima/electron/api/byok/providers/openai-compatible.cjs. The adapter accepts a custom base URL, optional authorization headers, and serializes message structures conforming to the standard chat completion format:
const https = require('https');
const http = require('http');
async function sendOpenAICompatibleRequest({ endpoint, apiKey, model, messages, temperature = 0.2, stream = false }) {
const url = new URL(endpoint.endsWith('/chat/completions') ? endpoint : `${endpoint}/chat/completions`);
const isHttps = url.protocol === 'https:';
const transport = isHttps ? https : http;
const payload = JSON.stringify({
model: model || 'default',
messages: messages,
temperature: temperature,
stream: stream
});
const options = {
method: 'POST',
hostname: url.hostname,
port: url.port || (isHttps ? 443 : 80),
path: url.pathname + url.search,
headers: {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(payload),
...(apiKey ? { 'Authorization': `Bearer ${apiKey}` } : {})
}
};
return new Promise((resolve, reject) => {
const req = transport.request(options, (res) => {
let data = '';
res.on('data', chunk => { data += chunk; });
res.on('end', () => {
try {
const json = JSON.parse(data);
if (res.statusCode >= 400) {
reject(new Error(json.error?.message || `HTTP ${res.statusCode}`));
} else {
resolve(json.choices?.[0]?.message?.content || '');
}
} catch (e) {
reject(new Error(`Failed to parse local model response: ${e.message}`));
}
});
});
req.on('error', (err) => {
reject(new Error(`Failed connecting to local model at ${endpoint}: ${err.message}`));
});
req.write(payload);
req.end();
});
}
module.exports = { sendOpenAICompatibleRequest };
4. Configuring Ollama & LM Studio in Proxima
Option A: Ollama Configuration
1. Start Ollama and pull your chosen coding model:
ollama pull qwen2.5-coder:14b
ollama serve
2. In the Proxima app, open Settings and add a custom BYOK provider. Proxima stores the provider name, endpoint, key and model in its encrypted key store, so there is no config file to edit by hand. Use the full chat-completions URL: Proxima sends requests to the endpoint exactly as you enter it and does not append a path.
Provider name: ollama
Endpoint: http://127.0.0.1:11434/v1/chat/completions
API key: ollama (Ollama ignores it, but Proxima requires at least 5 characters)
Model: qwen2.5-coder:14b
The endpoint must be a valid http or https URL. Once saved, the model appears
in list_available_models under BYOK, and you call it with
ask_model using the provider name you chose.
Option B: LM Studio Configuration
1. In LM Studio, load your model (e.g. DeepSeek-R1-Distill-Qwen-14B-GGUF).
2. Navigate to the Local Server tab and click Start Server.
3. Add it in Proxima as a custom BYOK provider with the endpoint http://127.0.0.1:1234/v1/chat/completions and the model name that LM Studio shows for the loaded model.
5. Tool Calling on Local Models
Unlike frontier cloud models with fine-tuned JSON-schema parsers, local models with 7B–14B parameters can occasionally struggle with complex, multi-parameter tool calls.
To ensure high execution reliability with local models:
- Use Strict Zod Type Constraints: Keep tool parameters primitive (strings, numbers, booleans). Avoid deeply nested JSON objects.
- Reduce Tool Set Exposure: When a small model sits behind your editor, expose only the few Proxima tools it needs (for example
ask_model,generate_code,fix_error,ddg_search) rather than all 41 at once, so the model has fewer choices to confuse. Most MCP clients can filter tools per server; Antigravity, for instance, supportsenabledToolsanddisabledToolsinmcp_config.json. - Enforce System Prompt Formatting: Include strict system prompt instructions specifying that function calls must conform exactly to valid JSON without Markdown conversational wrappers.
6. Local-Only Privacy Boundaries
When running Proxima connected to a local Ollama or LM Studio instance:
- Prompts stay local: With a local model selected, your prompts, files and stack traces travel only over the loopback address
127.0.0.1to your own Ollama or LM Studio server. Proxima sends no usage telemetry. - No Cloud Account Requirement: You do not need to register an account or pass identity verification tokens.
- Works offline: Chatting with the local model does not need the internet. Features that fetch from the web (
ddg_search,web_scrape) and the installed app's update check do, and the update check simply fails quietly when you are offline.
7. Diagnostic Failure Analysis
| Symptom | Root Cause | Diagnosis | Fix |
|---|---|---|---|
ECONNREFUSED 127.0.0.1:11434 |
The Ollama background daemon is not running. | Run curl http://localhost:11434/api/version in terminal. |
Start Ollama using ollama serve or launch the Ollama desktop app. |
Context length exceeded / output truncated |
Model context window is smaller than file payload being read. | Check model context configuration in LM Studio or Ollama Modelfile (num_ctx). |
Set num_ctx 32768 in Ollama Modelfile or use smart-slicer.js to extract sub-scopes. |
CUDA out of memory (OOM) |
Model parameter size exceeds available GPU VRAM. | Run nvidia-smi to inspect GPU memory usage. |
Switch to a smaller quantization (e.g. Q4_K_M) or offload fewer layers to GPU. |
8. Architectural Takeaways
Connecting offline models to modern IDE workflows gives developers complete privacy and deterministic costs without sacrificing tool-calling automation.