Running a 26B parameter model with only 6GB RAM using mmap
This guide covers running Gemma 4 26B MOE (Mixture of Experts) locally on an Apple Silicon Mac using llama.cpp with memory-mapped files. The MOE architecture activates only 8 of 128 experts per token, making it incredibly efficient.
| Spec | Value |
|---|
.env or any environment variable files—only the user may change them.git reset --hard, rm, git checkout/git restore to an older commit) unless the user gives an explicit, written instruction in this conversation. Treat t
| import argparse | |
| import json | |
| import logging | |
| import os | |
| import re | |
| import shutil | |
| from concurrent.futures import ProcessPoolExecutor, as_completed | |
| from dataclasses import dataclass | |
| from datetime import datetime | |
| from typing import Any, Callable, Dict, List, Literal, Optional, Tuple |
| base_model: meta-llama/Meta-Llama-3-70B | |
| model_type: LlamaForCausalLM | |
| tokenizer_type: AutoTokenizer | |
| load_in_8bit: false | |
| load_in_4bit: true | |
| strict: false | |
| datasets: | |
| - path: /home/migel/ai_datasets/tess-v1.5b-chatml.jsonl |
| // 3D Dom viewer, copy-paste this into your console to visualise the DOM as a stack of solid blocks. | |
| // You can also minify and save it as a bookmarklet (https://www.freecodecamp.org/news/what-are-bookmarklets/) | |
| (() => { | |
| const SHOW_SIDES = false; // color sides of DOM nodes? | |
| const COLOR_SURFACE = true; // color tops of DOM nodes? | |
| const COLOR_RANDOM = false; // randomise color? | |
| const COLOR_HUE = 190; // hue in HSL (https://hslpicker.com) | |
| const MAX_ROTATION = 180; // set to 360 to rotate all the way round | |
| const THICKNESS = 20; // thickness of layers | |
| const DISTANCE = 10000; // ¯\\_(ツ)_/¯ |
| # 2023-11-27 MIT LICENSE | |
| Here's the open source version of my ChatGPT game MonkeyIslandAmsterdam.com. | |
| It's an unofficial image+text-based adventure game edition of Monkey Island in Amsterdam, my home town. | |
| Please use it however you want. It'd be nice to see more ChatGPT-based games appear from this. If you get inspired by it, please link back to my X https://x.com/levelsio or this Gist so more people can do the same! | |
| Send me your ChatGPT text adventure game on X, I'd love to try it! |
| import { Data, Override } from "framer" | |
| const state = Data({ | |
| doubleTapIndex: 0, | |
| doubleTapTimer: setTimeout(null, null), | |
| longPress: false, | |
| longPressTimer: setTimeout(null, null), | |
| }) | |
| export function doubleTap(): Override { |
| ''' Script for downloading all GLUE data. | |
| Note: for legal reasons, we are unable to host MRPC. | |
| You can either use the version hosted by the SentEval team, which is already tokenized, | |
| or you can download the original data from (https://download.microsoft.com/download/D/4/6/D46FF87A-F6B9-4252-AA8B-3604ED519838/MSRParaphraseCorpus.msi) and extract the data from it manually. | |
| For Windows users, you can run the .msi file. For Mac and Linux users, consider an external library such as 'cabextract' (see below for an example). | |
| You should then rename and place specific files in a folder (see below for an example). | |
| mkdir MRPC | |
| cabextract MSRParaphraseCorpus.msi -d MRPC |
Info: https://www.nginx.com/resources/admin-guide/nginx-https-upstreams/
Source: http://nategood.com/client-side-certificate-authentication-in-ngi
This is SSL, so you'll need an cert-key pair for you/the server, the api users/the client and a CA pair. You will be the CA in this case (usually a role played by VeriSign, thawte, GoDaddy, etc.), signing your client's certs. There are plenty of tutorials out there on creating and signing certificates, so I'll leave the details on this to someone else and just quickly show a sample here to give a complete tutorial. NOTE: This is just a quick sample of creating certs and not intended for production.
