jundot/omlx

This page introduces an external open-source repository. It is not an HDATF product.

What it does

A local LLM inference server for Macs with Apple Silicon, managed from the macOS menu bar. It uses continuous batching and keeps KV cache in memory and on SSD so past context can be reused across requests.

How it helps ATF

Worth comparing if Harness hands work to locally hosted models. Its KV cache reuse across requests is a reference for long agent sessions with changing context.

License

Apache-2.0 Permissive, with a patent grant. Commercial use is allowed; keep the notices and state your changes.

More in this category

• TheTom/turboquant_plus
  TurboQuant+ is a research implementation of TurboQuant that compresses the transformer KV cache using PolarQuant and Walsh-Hadamard rotation. It holds a Python reference implementation, validation papers and benchmark data.
• karpathy/nanochat
  A minimal codebase for training and chatting with your own LLM on a single GPU node, covering tokenization, pretraining, finetuning, evaluation and inference. One depth setting determines the other hyperparameters.
• google-ai-edge/gallery
  An app that showcases on-device generative AI and lets people try open-source models locally and offline on mobile devices. It includes chat with a thinking mode and agent skills that add tools such as Wikipedia.
• maderix/ANE
  A research proof of concept that trains neural networks directly on Apple's Neural Engine through reverse-engineered private APIs, without CoreML training APIs, Metal or the GPU. The author states it is not a maintained framework.
• shiyu-coder/Kronos
  Kronos is a family of decoder-only foundation models pre-trained on financial candlestick (K-line) data. A tokenizer turns price and volume sequences into discrete tokens, and a Transformer trained on them is used for forecasting and other quantitative tasks.

Only repositories in the ranked Trendshift lists are included, and the lists are used only to find candidates. We do not copy their ranks. Descriptions, licenses and star counts come from each GitHub repository. The notes are our own reading. We have not tested these projects, and a place on a trending list does not prove quality.