This is an attempt at a presentation of the concept of variadic generics for Rust. It's not a proposal so much as a summary of existing work, and a toolbox for creating an eventual proposal.
Variadic generics (aka variadic templates, or variadic tuples), are an often-requested feature that would enable traits, functions and data structures to be generic over a multitude of types, instead of a single type.
To give a quick example, a Rust variadic function might look like this:
fn make_tuple_sing<...T: Sing>(t: (...T)) {
for member in ...t {
member.sing();
}
}
let kpop_band = (KPopStar::new(), KPopStar::new());
let rock_band = (RockStar::new(), RockStar::new(), RockStar::new(), RockStar::new());
let mixed_band = (KPopStar::new(), RockStar::new());
make_tuple_sing(kpop_band);
make_tuple_sing(rock_band);
make_tuple_sing(mixed_band);Note that variadic generics are a broader feature than variadic functions. There are many languages implementing a feature that lets user call a function with an arbitrary number of parameters, which are sometimes called variadic functions. The extra parameters are dynamically typed (C, JS, Python) or a shorthand for passing a slice (ex: Java, Go, C#).
As far as I'm aware, there are only two widespread languages implementing variadic generics (C++ and D), which is what this document is about.
(Zig has powerful compile-time reflection which gives similar benefits, but it doesn't really have "templates" the way C++ or Rust understand it, so I won't include it here)
This has been the focus of much attention, as the use case is a natural extension of the one that motivated const-generics.
Currently, implementing a trait for a tuple is usually done by writing a macro implementing the trait over a given number of fields, then calling the macro multiple times.
This has the same problems that array implementations of traits without const generics had:
- It requires lots of awkward boilerplate.
- It's unpleasant to code and maintain; compiler errors are a lot less readable than with regular generics.
- It only implements the trait up to a fixed number of fields, usually 12. That means that a user with a 13-uple is out of luck.
Variadic generics would provide an idiomatic way to implement a trait for arbitrary tuples.
The ecosystem has a few utility methods that operate on pairs of objects, such as Iterator::zip or async-std's Future::join.
There are often use cases where one might want to call these methods to combine more than two objects; currently, the default way to do so is to call the method multiple times and chain the results, eg a.join(b).join(c) which essentially returns Joined(Joined(a, b), c) (kind of like "cons lists" in Lisp).
A common workaround is instead implement the utility method as a macro, which can take an arbitrary number of parameters, but this isn't always convenient.
I have never seen this use case suggested before, but it seems like an obvious feature to me; it's also the main use case for variadics in D.
Rust has a lot of crates centered around providing macros to enhance your types. These macros often follow a pattern of "list your type's fields, and then for each of the fields, do something similar". For instance:
Rust has a large ecosystem dedicating to enhancing data structures with #[attribute]
#[derive(serde::Serialize, serde::Deserialize, Debug)]
struct Point {
x: i32,
y: i32,
}The Serialize, Deserialize and Debug macros all follow the same principle of "do something with x, then do something with y", where the "something" in question can be easily defined with traits. For each trait, this is done by generating a string of tokens that compiles to a trait implementation for Point.
By contrast, a serialization function in D will look like
void serialize_struct(T)(Writer writer, string name, const T value)
{
writer.startObject(name);
static foreach (memberName; __traits(allMembers, T))
{{
auto memberValue = __traits(getMember, value, member);
alias PlainMemberT = typeof(cast() memberValue);
static if (isStruct!PlainMemberT)
{
serialize_struct(writer, memberName, memberValue);
}
else
{
// Serialize leaf types (eg integers, strings, etc)
// ...
}
}}
}There are a lot of subtle differences between D's semantics and Rust's, which means some concepts can't be trivially ported; I personally think these differences, like much of D's generics, are under-studied, so this document won't go into details.
The gist of it is that D's generics are closer to Rust macros than Rust generics. They feel like a scripting language, where entire chunks of code can be disabled or reinterpreted based on template parameters; so naturally post-monomorphization errors are much more frequent.
That said, the enthusiastic adoption of static foreach in D shows that there is a strong demand for an easy-to-use, idiomatic feature to write code that applies to each field of a data structure.
Some variadic generics proposals mention other possible use cases. I think they are less compelling, so I'll cover them very briefly:
-
Fn traits: Fn traits currently work with compiler magic, so that they be called with arbitrary numbers and types of arguments. Proposals mention that implementing variadic tuples would offer more flexibility when working with higher-order functions, though I'm personally not sure how relevant that use case would be.
-
Tuple manipulation: Proposals mention that tuples could be enhanced to be flattened, or indexed with constant integers, like C++ tuples allow:
// In C++11 you can do: do_something(std::get<1>(tuple));
I think these kinds of features would be a lot more likely to bump into post-monomorphisation errors, and have limited real-world uses.
There's an interesting conversation about what we want Rust generics to be like; some of that conversation has already started, with the question of whether to allow maybe-panicking code in const generics. Whether to bring tuples closer to reified types, with flattening and indexing and so on, should be part of that conversation, which is broader than this document.
This section is not so much a proposal for variadic templates, so much as a list of questions that need to be answered to implement the feature.
It's not as structured as I'd like. I'll start with what the syntax might look like, what operations we'll want to allow, and branch out from there.
First and foremost, we want:
- To express "this template function/type/trait takes a parameter that can represent an arbitrary number of types".
- To require that each of those types implements a given trait.
- To declare tuples of variadic types, and pass them around (eg
let my_tuple : /* VARIADIC_TYPES */;) - In some case, to "flatten" tuples, and interpret them as a comma-separated list of values, like the spread operator in JavaScript.
Common suggestions include: Ts..., ...Ts, Ts.., ..Ts, Ts @ ... Using ... is closer to existing C++/D syntax, but .. is closer to existing Rust syntax for "and then a bunch of things". The Ts @ .. syntax in particular mimics existing subslice patterns.
Finally, we want to concisely express "Execute this code for every member of this variadic tuple". There are two common solutions for this:
- Split the tuple into a "head" and a "tail" binding (eg
let (head, ...tail) = my_tuple), process the head, and recursively call the function on the tail binding. - A special
for _ in _loop which iterates over tuples. Syntax could befor member in ...tupleorfor member ..in tupleor something similar.
I don't have strong preferences as far as "which exact symbols should we use, in what order", so the examples in this document just use an arbitrary syntax.
Let's consider the future::join use-case again:
let a = some_future(1u8);
let b = some_future("hello");
let c = some_future(42.0);
assert_eq!(future::join(a, b, c).await, (1u8, "hello", 42.0));Thejoin function takes (SomeFuture<u8>, SomeFuture<&str>, SomeFuture<f32>) and returns JoinedFuture<(u8, str, f32)>. To enable this, we need a way for a function declaration to apply a mapping to the types of a variadic tuple. The declaration of join might look like:
fn join<...Fs: Future>(futures: ...Fs) -> JoinedFuture<...Fs::Output>;Conceptually, we are mapping our variadic parameter to a type constructor that looks like <type T where T: Future> => <T as Future>::Output (pseudocode).
- Note: In our example, that mapping is expressed by adding the
...token before a type-expression that includes a variadic argument, and instantiating the type-expression for each member of the variadic (similar to how macro_rules! repetitions work). Other syntaxes are possible.
The mapping should carry over all the usual rules of type inference. For instance:
fn unwrap_all<...Ts>(options: ...Option<Ts>) -> (...Ts);
let (a, b, c) = unwrap_all(Some(1), Some("hello"), Some(false));For the above code to work the way non-variadic code would, type inference should understand that unwrap_all expects Options, and match the types Ts to the option payloads (that is, Ts should be i32, &str, bool).
The above examples are fairly simple, with a single mapping. Some use-cases are likely to be more complex. For instance, let's imagine a function that splits a tuple of Eithers into tuples of Lefts and Rights.
fn split<...Lefts, ...Rights>(eithers: ...Either<Lefts, Rights>)
-> (...Option<Lefts>, ...Option<Rights>);The syntax used in this example is simplified, and makes some implicit assumptions: that Lefts and Rights variadics have the same tuple size, and that our ... syntax iterates through both of them "simultaneously" (like the Iterator::zip).
We can imagine use-cases where these assumptions don't hold; for instance, functions that take multiple independent variadic parameter but don't zip them, or a syntax where (...As, ...Bs) with As = (i8, i16), Bs = (str, bool)evaluates to((i8, str), (i8, bool), (i16, str), (i16, bool))`. It's unclear how often there would be a real-life need for the use-cases, though.
Either way, the language spec should either document these assumptions, or provide a syntax to include them in a declaration, that both the developer and the compiler can reason about, eg:
fn split<...Lefts, ...Rights>(eithers: ...Either<Lefts, Rights>)
-> (...Option<Lefts>, ...Option<Rights>)
where Lefts: SAME_TUPLE_SIZE_AS<Rights>;(though I'm personally not super hot on the above syntax; when doing array, we usually don't write fn foobar<A: Array<T>, B: Array<T2>>(...) where A: SameSizeAs<B>;, we use const-generics)
Some users might want to use variadics to concatenate tuples, eg:
fn concat<...As, ...Bs>(a: (...As), b: (...Bs)) -> (...As, ...Bs);Much of the previous section applies here. It's easy to come up with a syntax that leaves a lot of edge-cases unspecified.
In particular, if users are allowed to use multiple variadics in a single type list, this may make patterns harder to reason about, unless sensible limitations are specified:
fn cut_in_two<...As, ...Bs>(ab: (...As, ...Bs)) -> (...As), (...Bs);
cut_in_two(0, 1, 2, 3, 4); // Where does A stop and B start?In theory, variadics can enable a statically-typed language to treat types almost like first-class values. Beyond map, a language could port most of the classic list transformations (filter, fold, any, all, sort, dedup, etc) to sequences of types (and this is indeed what D does).
It's unclear how much real-world use these transformations would have. One could, for instance, write a type constructor IntegerTypes<Ts...> that would return a subset of the input tuple with only integer types; but it's not obvious what practical applications this constructor would have that cannot be achieved now.
Rust generally frowns upon adding complex type operations for the sake of making them possible, the way some functional programming languages do (no Higher-Kinded Types for you). Anybody wanting to push towards D-style non-linear variadics (that is, type constructors with variadics that aren't N-to-N), will have an uphill climb ahead of them. Among other things, they'll be expected to research how these additions would impact type inference, undecidability issues, and post-monomorphization errors.
Declaring variadic types in only one-half of the feature. The other is how to use them.
A for loop is the most idiomatic way to express "I want to do the same thing, over and over again, with each item of this collection".
Adapting it for variadics is straightforward:
fn hash_tuple<...Ts: Hash, S: Hasher>(tuple: (...Ts), state: &mut S) {
for member ...in tuple {
member.hash(state);
};
}We also need tuple-for to return the data at the end of its block, for its respective members:
fn unwrap_all<...Ts>(options: ...Option<Ts>) -> (...Ts) {
for option ...in options {
option.unwrap()
}
}(that last feature would probably be incompatible with break and continue statements; I won't go into details, but there are several ways to adress that)
And in some cases, we need to iterate other types as well as values:
fn to_vecs<...Ts>(options: ...Option<Ts>) -> (...Vec<Ts>) {
for option, type T ...in options {
if let Some(value) = option {
vec![value]
}
else {
Vec::new::<T>()
}
}
}(though I don't expect these cases to be common; Rust type inference is strong, and the example above is kind of dumb)
Given the following code:
let array = [1, 2, 3];
for x in &array {
do_stuff(x);
}we expect x to be of type &i32, not i32.
Similarly, with variadics:
// tuple == (1, 2, 3);
for x ...in &tuple {
do_stuff(x);
}x should also be of type &i32.
The exact rules will probably be a little more magical than with iterators, since tuple-for operates on variadics, and probably won't be built on a trait that can be implemented in user code.
Some use-cases might require the implementation
C++ provides a convenient way to fold variadics over basic operators:
template<typename ...Ts> int f(Ts... ts) {
return 0 + ... + ts;
}This would make it possible to concatenate them or pass them inside arguments list, eg:
my_variadic_function(arg1, ...my_tuple, arg2, ...my_other_tuple, more_args);let a = 1u8;
let b = "hello";
let c = 42.0;
assert_eq!(compute_hash_of(a, b, c), 1337_u64);Design Elements
Error story Avoid post-monomorphisation errors
Allow partial matches Be able to take multiple tuples Map tuples to other types
Interaction with macros (eg println!("somestufff {}", ...my_tuple))
Non-variadic tuples
Design questions that we leave aside for a MVP
TODO - Add a part about variadic functions at the beginning Ctrl-F "above" Add "rust" to ``` blocks
- cons-list
- Allowing template to do indexing over arbitrary tuples
- Concatenating tuples
- Variadic functions
"Recursive" tuples (like C++ variadics)
"@ .." to reuse existing syntax
https://github.com/fredpointzero/rfcs/blob/variadic_tuples/text/0000-variadic-tuples.md https://github.com/alexanderlinne/rfcs/blob/variadic_generics/text/0000-variadic-generics.md https://github.com/memoryleak47/variadic-generics/blob/master/0000-variadic-generics.md rust-lang/rfcs#376
We have to distinguish between variadic functions and variadic types Generally speaking, existing languages implement variadic functions in one of three fashions:
- Have a special variadic argument at the end of a function. That variadic only handles a single type, and is more of a shorthand for passing a slice (ex: Java, Go, C#).
- Have a special, dynamically-type variadic argument at the end of a function. Usually in dynamically-type languages, this variadic handles different types, but these types can only be differentiated at runtime, with no compile-time validation (ex: C, Python, Javascript).
Rust macros
template<class ... Types> struct Tuple {};
void something(Ts...)(Ts ts)
The thing I'm personally not too enthusiastic about is derives as a big motivation for this. If you wanted to able to inspect all parts of the type the derive was used on like proc-macro derives can, you would basically end up with
const Input: syn::DeriveInputinstead ofconst Name: Stringandconst FieldNames: [String]. The only improvement over proc-macros then would be the ability to declare some assumptions the derive macro has of the input (types of a struct's fields implement a certain trait, maybe some other things), and I would argue that most macros have more complex requirements of their input than that. I think without post-morphization errors, only <5% of what people want derives to do could be done with something like#[variadic_derive], which stands in stark contrast to the effort that would be required to enable it.The issue of proc-macro compile times is also being tackled from other sides (see watt and cargo-watt), which I find a lot more compelling.