wici-ai/inferssd

This page introduces an external open-source repository. It is not an HDATF product.

What it does

InferSSD is a local inference engine that streams large MoE models from SSD when they cannot fit entirely in RAM or GPU memory. It extends llama.cpp rather than supplying a model, taking supported GGUF weights and user requests to produce generated responses. It reads the experts selected for each token from SSD, caches them in RAM and uses CPU and GPU paths according to the environment. Browser chat and an OpenAI-compatible API allow external clients to connect. Weights must be downloaded separately and can require substantial SSD space plus runtime working memory. Official documentation describes Apple Silicon Metal and Linux CUDA environments; source builds also need Python and C/C++ tooling. The README header calls Windows work in progress while its body describes Windows 11 packages, so the actual release's supported backend needs checking. Neither universal model support nor hardware-independent speed is established; SSD, cache and request conditions matter. The network API lacks authentication and needs access controls, while model licenses remain separate from the engine. Published throughput does not guarantee answer quality or performance on our hardware.

License

Apache-2.0 Permissive, with a patent grant. Commercial use is allowed; keep the notices and state your changes.

More in this category

• canivel/dynolab
  Dyno Lab is an Apple Silicon research app for observing local model behavior and testing hypotheses. Researchers supply questions, prompts, models and experiment settings to obtain responses, activation measurements and saved artifacts. It provides request inspection, research notebooks, comparative experiments, linear probes and small sparse-autoencoder workflows. The app bundles Python and MLX, but model weights must be downloaded and started separately. It requires Apple Silicon and macOS 14 or later; pairing a Windows NVIDIA worker for larger GGUF models is an experimental GPU-pool feature. Capturing an already-loaded model and experiments that load another copy have different memory requirements. Python SDK, HTTP API and a local stdio MCP bridge are available, but starting MCP alone does not launch a model or experiment. Results are stored locally, yet an external assistant may send results it reads to its provider, so exposure needs review. Activation magnitude and model-emitted thinking are not safety certification or a faithful explanation of internal computation. Official material distinguishes exploratory experiments from validated safety monitors.
• huawei-csl/MACaQ
  MACaQ is a calibrated quantization toolkit for reducing the numerical precision of supported model weights and, on selected backends, activations to make local inference more practical. Its inputs include matching original model checkpoints and representative calibration text, prompts or conditioning images, while its outputs include compact GGUF or safetensors checkpoints and recorded settings. In the inspected dense text path, activation statistics guide weight balancing and a block Hadamard transform before low-bit quantization and codebook refinement. Separate adapters cover documented text, diffusion and speech model families, so model agnostic should not be read as automatic compatibility with every architecture. The llama.cpp path offers two-, three- and four-bit weight formats, whereas other adapters offer W4A4, meaning four-bit weights and activations, with some components intentionally left at higher precision. Full transformed GGUF workflows use the bundled runtime, while MACaQ-R refines selected native MoE weight codes without changing their formats and is documented to work with standard llama.cpp. Preparation requires backend-specific dependencies, matching source weights, calibration data and working storage; the guides specify Python 3.12, Linux for vLLM and a CPU alternative for llama.cpp. Quantization is not repeated when saved checkpoints are loaded, but runtime compatibility and original model licensing still apply, and the diffusion guide explicitly notes unclear licensing for one community audio checkpoint. The README reports memory, speed and model-quality results, but these were not reproduced here and cannot establish the same trade-off for another model, device or application.
• Generated-Computers/Locals
  This local language-model server handles requests from multiple agents on Apple Silicon Macs. It is an execution layer loading MLX-family models with batching and prompt caching, not a set of model weights itself. Conversations and tool definitions enter through APIs, returning text, tool calls and structured JSON. It translates supported requests from OpenAI-compatible, Anthropic Messages and Ollama formats. Starting requires Apple Silicon M1 or newer, macOS 14 or newer, enough memory for the selected model and an initial model download. Existing apps need separate server and model configuration or the supplied launch connectors. It runs one model at a time and does not directly serve GGUF files. Published acceleration figures are the author's results for specified hardware and concurrency; single requests, long inputs and other quantization formats can behave differently. The default local server has no authentication, so external exposure needs care, and it does not replace every upstream feature such as Ollama embeddings or image understanding.
• SheepSeb/tinyworldformer
  tinyworldformer is an experiment in generating and playing small games with two neural networks on a Raspberry Pi Pico 2 W. Its world model takes a token-grid frame and an action and predicts the next frame and reward. A separate actor reads features derived from the generated frame and chooses the next action, rather than controlling an ordinary game engine on the board. The world model reuses one transformer block across several passes and adds decoding constraints such as placing each ball or player only once. Python and PyTorch provide training, while quantization-aware training and export produce an int8 Rust inference engine for the microcontroller, host and WebAssembly. Training starts from transitions recorded in the supplied small games, and the board can optionally train a fresh streaming reinforcement-learning actor. Evaluation code compares generated transitions with the original game rules, because an actor can exploit mistakes in its learned environment. The project is not a general video world model or industrial simulator, the README leaves actual Pico frame rate unmeasured, and no repository license was identified in the inspected tree or package manifest.
• skyzh/tiny-llm
  tiny-llm is a hands-on course for systems engineers who want to understand local LLM inference rather than treat a serving engine as a black box. It takes Qwen3 model weights and tokenized conversations through attention, sampling and repeated decoding to produce text. Learners implement operators with MLX arrays, Python, C++ and Metal, while MLX provides a correctness reference and performance baseline. Later chapters connect KV caches, which retain past attention keys and values, to continuous batching and paged memory management. Week 4 adds a teaching coding agent that validates JSON actions, obtains approval for effects and records observable results. The learner package contains unfinished exercises and is separate from the reference implementation. The supported native environment is an Apple Silicon Mac with the specified Python dependencies, Xcode and native extension toolchain. This is teaching material rather than a production sandbox or durable transaction system, and the real-model agent CLI does not compose all the later checkpoint and branching mechanisms.

Only repositories in the ranked Trendshift lists are included, and the lists are used only to find candidates. We do not copy their ranks. Descriptions, licenses and star counts come from each GitHub repository. The notes are our own reading. We have not tested these projects, and a place on a trending list does not prove quality.