brucethemoose
brucethemoose
/ gist:5a111c3e7d3aeaf3bdb52071c6944ccf
Created
February 22, 2024 22:53
Split text into chunks by token length, never exceeding the token length, considering sentences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from transformers import AutoTokenizer | |
| import re | |
| ident_tokenizer = AutoTokenizer.from_pretrained( | |
| "model_name" | |
| ) | |
| def get_tokenized_length(input_string): | |
| # Example implementation: count the number of characters | |
| # Replace this with your actual tokenization logic | |
| return len(ident_tokenizer.encode(input_string)) |
brucethemoose
/ gist:ea64f498b0aa51adcc88f543bf80ed97
Last active
March 15, 2023 22:28
InvokeAI torch.compile
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Insert Here: | |
| https://github.com/invoke-ai/InvokeAI/blob/d9dab1b6c770b49771ab30cb66fd1348a5639cd3/invokeai/backend/stable_diffusion/diffusers_pipeline.py#L346 | |
| odict = { | |
| 'aggressive_fusion': True, | |
| 'benchmark_harness': True, | |
| 'dce': True, | |
| 'developer_warnings': True, | |
| 'disable_progress': True, | |
| 'epilogue_fusion': True, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Log uploaded on Sunday, March 27, 2022, 12:00:48 PM | |
| Loaded mods: | |
| Harmony(brrainz.harmony)[mv:1.2.0.0]: 0Harmony(2.2.0), HarmonyMod(1.2.0) | |
| Core(Ludeon.RimWorld): (no assemblies) | |
| Royalty(Ludeon.RimWorld.Royalty): (no assemblies) | |
| Ideology(Ludeon.RimWorld.Ideology): (no assemblies) | |
| Dubs Performance Analyzer - Wiri's Butchery(Dubwise.DubsPerformanceAnalyzerButchered): PerformanceAnalyzer(1.0.0) | |
| HugsLib(UnlimitedHugs.HugsLib)[ov:9.0.1]: HugsLib(av:1.0.0,fv:9.0.1) | |
| Geological Landforms(m00nl1ght.GeologicalLandforms): GeologicalLandforms(1.0.0) | |
| WorldEdit 2.0(FunkyShit.Mods.WorldEdit): WorldEdit 2.0(1.0.0) |
brucethemoose
/ gist:937690aa53ca4a8719671af3ee557ac0
Created
February 26, 2018 18:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @feupkeep = 4 | |
| @feproduction = 12 | |
| @fecost = 500 | |
| @fetime = 730 | |
| building_ancient_factory = { | |
| #building_defines | |
| tags = { mining_network } | |
| is_listed = no |
brucethemoose
/ gist:67ed5c9dbcedf5aae56d7f9a9ae4fa24
Created
February 26, 2018 16:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ai_allow = { | |
| OR = { | |
| tile = { | |
| has_building = no | |
| } | |
| tile = { | |
| has_building = building_power_plant_1 | |
| } | |
| } | |
| NOR = { |