🌲 Invert a binary tree! 🌲
Except with 3 catches:
- It must invert the keys ("bit-reversal permutation")
- It must be a dependency-free, pure recursive function
- It must have type
Bit -> Tree -> Tree(i.e., a direct recursion with max 1 bit state)
- It is somehow NOT on the internet. (AFAIK)
- Humans can solve it. (I've done it in ~1h.)
- It requires reasoning. (My head hurts!)
- The solution is simple. (7 lines of code!)
- Obvious pre-requisite to automate CS research.
- Honestly, it would make me believe I'll be automated.
I claim no AI will EVER solve this problem. If you prove me wrong, HOC will grant you $10k!
- You must give it an approved prompt, nothing else.
- It must output a correct solution, passing all tests.
- You can use any software or AI model.
- You can let it "think" for as long as you want.
- You can propose a new prompt, as long as:
- It imposes equivalent restrictions.
- It clearly doesn't help the AI.
- Up to 1K tokens, all included.
- Common sense applies.
{-# OPTIONS --no-termination-check #-}
-- Let Tree be a Perfect Binary Tree:
data Nat : Set where
Z : Nat
S : Nat → Nat
{-# BUILTIN NATURAL Nat #-}
data Bit : Set where
O : Bit
I : Bit
data Tree (A : Set) : Nat → Set where
N : ∀ {d} → Tree A d → Tree A d → Tree A (S d)
L : Nat → Tree A Z
-- Your goal is to implement an 'invert' function that performs a bit-reversal
-- permutation on a Tree, respecting the following limitations:
-- 1. You can NOT define or use any function other than 'invert'.
-- 2. You can NOT use any type not defined above (Nat, Bit and Tree).
-- 3. You can NOT use loops (but you can call 'invert' recursively).
-- 4. You can NOT use mutability. It must be a pure Agda function.
-- 5. You can use 1 bit of state (as an extra argument).
-- 6. You can use pattern-matching and constructors freely.
--
-- Example:
-- input = (N(N(N(L 0)(L 1))(N(L 2)(L 3)))(N(N(L 4)(L 5))(N(L 6)(L 7))))
-- output = (N(N(N(L 0)(L 4))(N(L 2)(L 6)))(N(N(L 1)(L 5))(N(L 3)(L 7))))
-- Because that's the bit-reversal permutation of the original tree.
--
-- Now, complete the program below, with a valid implementation of 'invert':
invert : ∀ {A d} → Bit → Tree A d → Tree A dtype Nat = number;
type Bit = false | true;
type Tree<A> = [Tree<A>, Tree<A>] | Nat;
// Your goal is to implement an 'invert' function that performs a bit-reversal
// permutation on a Tree, respecting the following limitations:
// 1. You can NOT define or use any function other than 'invert'.
// 2. You can NOT use any type not defined above (Nat, Bit and Tree).
// 3. You can NOT use loops (but you can call 'invert' recursively).
// 4. You can NOT use mutability. It must be a pure function.
// 5. You can NOT use primitive JS operators or functions.
// 6. You can use 1 bit of state (as an extra argument).
// 7. You can only use the operations allowed below.
//
// Operations allowed:
// - Destructing (`const [a,b] = value`)
// - Variables (`const x = value`)
// - Branching (`if (x) { ... } else { ... }`)
// - Recursion (`invert(_, _)')
// - `Array.isArray`
//
// All other operations are not allowed.
//
// Example:
// input = [[[[0,1],[2,3]],[[4,5],[6,7]]]]
// output = [[[[0,4],[2,6]],[[1,5],[3,7]]]]
// Because that's the bit-reversal permutation of the original tree.
//
// Now, complete the program below, with a valid implementation of 'invert':
function invert<A>(bit: Bit, tree: Tree<A>): Tree<A> {
...
}
// A test:
const tree: Tree<Nat> = [[[[0,1],[2,3]],[[4,5],[6,7]]],[[[8,9],[10,11]],[[12,13],[14,15]]]];
console.log(JSON.stringify(invert(true, tree)));✨ If it can't invert a tree, it won't solve P=NP. ✨
@Lorenzobattistela Thanks for spelling out your reasoning and for the honesty/integrity, I appreciate it! I do think we're all acting in good faith here, and that this is mainly a difference in interpretation.
IMO, the core point is this: Victor's OG challenge already allows one extra bit of state, so the model already knows it can use an extra boolean parameter to solve the problem. Victor gave us a lot of leeway to adapt the prompt and the scaffold, including changing languages, as long as it imposes equivalent restrictions and it clearly doesn't help the AI.
To "fix notations", let's call:
My claim is that my Python and Haskell "with-helper" prompts are only using legal help ("helper" here is meant in this sense, in my terminology: it means legal help regarding the extra bit of state; and the "no-helper" variants are harder than Victor's OG challenge because they do not incentivize using the extra bit of state as legal help).
More specifically, I believe that:
using a trivial wrapper around the same one-bit interface Victor had already made explicit is legal help, because it does not give any algorithmic hint. And that's what my prompts are doing: the Python one has
invert_helper(tree, flag)insideinvert(tree), and the Haskell one hasinvertHelper :: Bool -> Tree a -> Tree ainsideinvert. I genuinely don't think this gives the model extra algorithmic forbidden help. I read it as the same one-bit-of-state allowance, just expressed differently. (In hindsight, I should probably have named theminvert_innerorinvert_aux, because I can now see how the word "helper" makes them sound more suggestive than they actually are.)Why did I wrap it this way? Because it had a practical use on my side for automated property-based testing (Hypothesis for Python, QuickCheck for Haskell) to validate/reject the proposed answers: it gives me a uniform
invert(tree)entry point, and it also gives the possibility for the model to solve the harder challenge of writing the function without legal help from the extra bit of state (while keeping the same interface).I don't think that should be misinterpreted as forbidden help because the outer
invertis not a second recursive routine: it's just a one-line wrapper, while all the recursion still lives in the single inner function with one boolean parameter, and that's exactly what the syntactic constraints check. (As a comparison, the Python no-helper prompt (python_prompt_no_helper.md) is a stricter/harder follow-up that removes the extra bit legal helper.)By contrast, let's take a closer look at prompts that gave extra forbidden help. When Victor rejected msukiasyan's prompt, their first prompt was legal but did not lead the model to one-shot the solution, because it wrote two distinct functions:
invertandcombine(I had this a lot in my personal logs too at the beginning!). Msukiasyan then had to follow up with a second round by adding:Great! Now refactor the code into a single recursive function. You're allowed to define the function taking one binary variable and a tree as input.-> This is clearly forbidden help, because the model didn't one-shot the solution and had to be told, by the human, to refactor the two previously generated functions in order to get a syntactically correct solution.This is completely different to my prompts, where
invert_helper/invertHelperstill has to discover the actual recursion on its own: when to branch, when to swap, what the boolean means, and how that bit of state propagates through the tree.Likewise, as you said in your lhemerly review, their prompt was also using forbidden help, by introducing the non-trivial helper function:
The same goes for my sentence explaining bit-reversal permutation. I do think that is a restatement of the specification, not a hint about the algorithm solving it. Victor's OG prompt says "performs a bit-reversal permutation" and gives concrete input/output examples that encode the same fact. For me, the boundary is the following: describing the input/output permutation itself is still specification-level (legal help), whereas giving hints to the model about how to write the recursion (beyond the extra bit signature) is algorithm-level (forbidden help). The hard part is the recursive construction under the syntactic constraints, which is exactly what my prompts do not tell the model.
I'm happy to leave the final call to Victor.