|
| 1 | +# Converting ZygoteRules.@adjoint to `rrule`s |
| 2 | + |
| 3 | +[ZygoteRules.jl](https://github.com/FluxML/ZygoteRules.jl) is a legacy package similar to ChainRules but supporting [Zygote.jl](https://github.com/FluxML/Zygote.jl) only. |
| 4 | + |
| 5 | +If you have some rules written with ZygoteRules it is a good idea to upgrade them to use ChainRules instead. |
| 6 | +Zygote will still be able to use them, but so will other AD systems, |
| 7 | +and you will get access to some more advanced features. |
| 8 | +(Which Zygote may or may not then ignore). |
| 9 | + |
| 10 | +## Example |
| 11 | +Consider the function |
| 12 | +```julia |
| 13 | +struct Foo |
| 14 | + a::Float64, |
| 15 | + b::Float64 |
| 16 | +end |
| 17 | + |
| 18 | +f(x, y::Foo, z) = 2*x + y.a |
| 19 | +``` |
| 20 | + |
| 21 | +### ZygoteRules |
| 22 | +```julia |
| 23 | +@adjoint function f(x, y::Foo, z) |
| 24 | + f_pullback(Ω̄) = (2Ω̄, NamedTuple(;a=Ω̄, b=nothing), nothing) |
| 25 | + return f(x, y, z), f_pullback |
| 26 | +end |
| 27 | +``` |
| 28 | + |
| 29 | +### ChainRules |
| 30 | +```julia |
| 31 | +function rrule(::typeof(f), x, y::Foo, z) |
| 32 | + f_pullback(Ω̄) = (NoFields(), 2Ω̄, Tangent{Foo}(;a=Ω̄), ZeroTangent()) |
| 33 | + return f(x, y, z), f_pullback |
| 34 | +end |
| 35 | +``` |
| 36 | + |
| 37 | +## Write as a `rrule(::typeof(f), ...)` |
| 38 | +No magic macro here, `rrule` is the function that it is. |
| 39 | +The function it is the rule for is the first argument, or second argument if you need to take a `[RuleConfig`](@ref). |
| 40 | + |
| 41 | +## Include the derivative with respect to the function object itself |
| 42 | +The `ZygoteRules.@adjoint` macro automagically[^1] inserts a extra `nothing` in the return for the function it generates to represent the derivative of output with respect to the function object. |
| 43 | +ChainRules as a philosophy avoids magic as much as possible, and thus require you to return it explicitly. |
| 44 | +If it is a plain function (like `typeof(sin)`), then the differential will be [`NoTangent`](@ref). |
| 45 | + |
| 46 | + |
| 47 | +[^1]: unless you write it in functor form (i.e. `@adjoint (f::MyType)(args...)=...`), in that case like for `rrule` you need to include it explictly. |
| 48 | + |
| 49 | +## Tangent Type changes |
| 50 | +ChainRules uses tangent types that must represent vector spaces (i.e. tangent spaces). |
| 51 | +They need to have things like `+` defined on them. |
| 52 | +ZygoteRules takes a more adhoc approach to this. |
| 53 | + |
| 54 | +### `nothing` becomes an `AbstractZero` |
| 55 | +ZygoteRules uses nothing to represent some sense of zero, in a primal type agnostic way. |
| 56 | +There are many senses of zero. |
| 57 | +ChainRules represents two of them, as subtypes of [`AbstractZero`](@ref). |
| 58 | + |
| 59 | +[`ZeroTangent`](@ref) for the case that there is no relationship between the primal output and the primal input. |
| 60 | +[`NoTangent`](@ref) for the case that conceptually the tangent space don't exist. |
| 61 | +e.g. what is the Tangent to a String or an index: those can't be perturbed. |
| 62 | + |
| 63 | +See [FAQ on difference between `ZeroTangent` and `NoTangent`](@ref faq_abstract_zero). |
| 64 | +At the end of the day it doesn't matter too much if you get them wrong. |
| 65 | +`NoTangent` and `ZeroTangent` more or less act identically. |
| 66 | + |
| 67 | +### `Tuple`s and `NamedTuple`s become `Tangent{T}`s |
| 68 | +Zygote uses `Tuple`s and `NamedTuple`s to represent the structural tangents for `Tuple`s and `struct`s respectively. |
| 69 | +ChainRules core provides a generic `Tangent{T}`(@ref Tangent) to represent the structural tangent of a primal type `T`. |
| 70 | +It takes positional arguments if representing tangent for a `Tuple`. |
| 71 | +Or keyword argument to represent the tangent for a `struct`. |
| 72 | +When representing a `struct` you only need to list the nonzero fields -- any not given are implicit considered to be [`ZeroTangent`](@ref). |
| 73 | + |
| 74 | +When we say structural tangent we mean tangent types that are based only on the structure of the primal. |
| 75 | +This is in contrast to a natural tangent which captures some knowledge based on what the primal type represents. |
| 76 | +(E.g. for arrays a natural tangent is often the same kind of array). |
| 77 | +For more details see the the [design docs on the many tangent types](@ref manytypes) |
| 78 | + |
| 79 | + |
| 80 | +## Calling back into AD (`ZygoteRules.pullback`) |
| 81 | +Rules that need to call back into the AD system, e.g, for higher order functions like `map(f, xs)`, need to be changed. |
| 82 | +In `ZygoteRules` you can use `ZygoteRules.pullback` or `ZygoteRules._pullback`, which will always result in calling into Zygote. |
| 83 | +Since ChainRules is AD agnostic, you can't do that. |
| 84 | +Instead you use a [`RuleConfig`](@ref) to specify requirements of an AD system e.g `::RuleConfig{>:CanReverseMode}` work for Zygote, |
| 85 | +and then use [`rrule_via_ad`](@ref). |
| 86 | + |
| 87 | +See the [docs on calling back into AD](@ref config) for more details. |
| 88 | + |
| 89 | +## Consider adding some thunks |
| 90 | + |
| 91 | +A feature ChainRulesCore offers that ZygoteRules doesn't is support for thunks. |
| 92 | +Where work is delayed until it is needed, and avoided if it never is. |
| 93 | +See docs on [`@thunk`](@ref), [`Thunk`](@ref), [`InplaceableThunk`](@ref). |
| 94 | + |
| 95 | +You don't have to though. |
| 96 | +It is easy to go overboard with using thunks. |
| 97 | + |
| 98 | +## Testing Changes |
| 99 | + |
| 100 | +One of the advantages of using ChainRules is that you can easily and robustly test it with [ChainRulesTestUtils.jl](https://juliadiff.org/ChainRulesTestUtils.jl/stable/). |
| 101 | +This both uses finite differencing to test accuracy of derivative, as well as checking the correctness of the API. |
| 102 | +It should catch anything you might have gotten wrong referred to in this page. |
| 103 | + |
| 104 | +The test for the above example is `test_rrule(f, 2.5, Foo(9.9, 7.2), 31.0)`. |
| 105 | +You can see it looks a lot like an example call to `rrule`, just with the prefix `test_` added to the start. |
| 106 | + |
| 107 | +## `@nograd` becomes `@non_differentiable` |
| 108 | +Probably more or less with no changes. |
| 109 | +[`@non_differentiable`](@ref) also lets you specify a signature. |
| 110 | + |
| 111 | +## No such thing a `literal_getproperty` |
| 112 | +That is just `getproperty`, it takes `Symbol`. |
| 113 | +It should constant-fold. |
| 114 | +It likely doesn't though as Zygote doesn't play nice with the optimizer. |
| 115 | + |
| 116 | +## Take embedded spaces and types seriously |
| 117 | +Traditionally Zygote has taken a very laissez faire attitude towards types and mathematical spaces. |
| 118 | +Sometimes treating `Real`s as embedded in the `Complex` plane; some time not. |
| 119 | +Sometimes treating sparse and structuredly-sparse matrix as embedded in the space of dense matrixes. |
| 120 | +Writing rules that apply to any `Array{T}` which perhaps are only applicable for `Array{<:Real}` and not so much for `Array{Quaternion}`. |
| 121 | +Traditionally ChainRules takes a much more considered approach. |
| 122 | + |
| 123 | +See for example our [docs on how to handle complex numbers](@ref complexfunctions) correctly. |
| 124 | +(The outcome of several long long long discussions with a number of expert in our community) |
| 125 | + |
| 126 | +Now, I am not here to tell you what to do in your package, but this is a good time to reconsider how seriously you take these things in the rules you are converting. |
| 127 | + |
| 128 | +## What if I miss something |
| 129 | + |
| 130 | +It is not great, but it probably OK. |
| 131 | +Zygote's ChainRules interface is fairly forgiving. |
| 132 | +Other AD systems may not be. |
| 133 | +If you test with [ChainRulesTestUtils.jl](https://juliadiff.org/ChainRulesTestUtils.jl/stable/) then you can be confident that you didn't miss anything. |
0 commit comments