Nested dataset collection queries in Galaxy degrade from ~8ms to 843 seconds
when PostgreSQL's planner chooses hash/merge joins for dataset_collection_element
self-joins. The fix: replace DCE-to-DCE joins with nested ARRAY(subquery) expressions.
Marius van den Beek mvdbeek
mvdbeek
/ test_url_builder_benchmark.py
Created
March 20, 2026 13:39
Benchmark: UrlBuilder.url_path_for caching — 87x speedup for history contents serialization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """Benchmark for UrlBuilder.url_path_for caching. | |
| Demonstrates the performance difference when serializing N history items, | |
| each requiring up to 2 url_path_for calls. Without caching, each call | |
| does a linear scan through all registered routes. With caching, only the | |
| first call per route name scans; subsequent calls go directly to the | |
| matching route. | |
| """ | |
| import sys |
mvdbeek
/ nested_collection_query_planner.md
Last active
March 14, 2026 09:28
PostgreSQL query planner catastrophe with dataset_collection_element self-joins in Galaxy
mvdbeek
/ bwa_mem2_schema.json
Created
March 9, 2026 10:47
BWA-MEM2 Galaxy tool parameter request schema
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "$defs": { | |
| "BatchDataInstance": { | |
| "additionalProperties": false, | |
| "properties": { | |
| "src": { | |
| "enum": [ | |
| "hda", | |
| "ldda", | |
| "hdca" |
When metadata_strategy: directory_celery (or celery_extended) is configured, if the Celery process is interrupted (OOM killed, process restart, etc.) while executing a set_job_metadata task, jobs become permanently stuck in a non-terminal state (running) with no recovery mechanism.
The handler blocks forever on .get() when a worker dies.
mvdbeek
/ ISSUE_21642.md
Created
February 26, 2026 12:58
Triage artifacts for Galaxy issue #21642 - Remote data fetch not respecting quota
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/lib/galaxy/model/migrations/alembic/env.py b/lib/galaxy/model/migrations/alembic/env.py | |
| index 98091912b80..c9db41cc84d 100644 | |
| --- a/lib/galaxy/model/migrations/alembic/env.py | |
| +++ b/lib/galaxy/model/migrations/alembic/env.py | |
| @@ -1,7 +1,7 @@ | |
| import logging | |
| import re | |
| +from collections.abc import Callable | |
| from typing import ( | |
| - Callable, |
mvdbeek
/ merged_test_output.md
Created
February 24, 2026 10:15
IWC Workflow Test Results - Run 22339836776
Galaxy's workflow engine currently defines input forms server-side for data_input steps, requiring server round-trips and complex translation code (step_state_to_tool_state, _parse_state_into_dict). PR #19313 successfully migrated data_collection_input to client-side rendering. This plan extends that pattern to data_input, which is simpler since it manages only three parameters: optional, format, and tag.
mvdbeek
/ fetch_small_accessions.py
Created
February 13, 2026 19:40
ENA/Galaxy stress test: fetch random small accessions, upload via ascp, run cat1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python3 | |
| """ | |
| Fetch 100 random small (<500MB) SRR/ERR accessions from ENA and write their FTP links to a file. | |
| """ | |
| import csv | |
| import io | |
| import random | |
| import sys | |
| import urllib.request |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| rnaseq_de: | |
| workflow_id: 8cd7bdc8d3cf35cd | |
| workflow_target_type: stored_workflow | |
| request_state: | |
| Sample sheet of sequencing reads: | |
| class: Collection | |
| collection_type: sample_sheet:paired | |
| name: RNA-Seq DE sample sheet | |
| elements: | |
| - class: Collection |
NewerOlder