|
| 1 | +# How to create new AudioNode |
| 2 | + |
| 3 | +In this docs we present recommended patterns for creating new AudioNodes. |
| 4 | + |
| 5 | +## Layers |
| 6 | + |
| 7 | +Ususally, each AudioNode has three layers: |
| 8 | + |
| 9 | +- Core (C++) |
| 10 | + |
| 11 | + Class implementing core audio processing logic. Should be implemented in highly performant manner using internal data structures (if possible). |
| 12 | + |
| 13 | +```cpp |
| 14 | +class GainNode : public AudioNode { |
| 15 | + public: |
| 16 | + explicit GainNode(const std::shared_ptr<BaseAudioContext> &context, const GainOptions &options); |
| 17 | + |
| 18 | + [[nodiscard]] std::shared_ptr<AudioParam> getGainParam() const; |
| 19 | + |
| 20 | + protected: |
| 21 | + std::shared_ptr<AudioBuffer> processNode( |
| 22 | + const std::shared_ptr<AudioBuffer> &processingBuffer, |
| 23 | + int framesToProcess) override; |
| 24 | + |
| 25 | + private: |
| 26 | + std::shared_ptr<AudioParam> gainParam_; |
| 27 | +}; |
| 28 | +``` |
| 29 | +
|
| 30 | +- Host Object (HO) |
| 31 | +
|
| 32 | + Interop class between C++ and JS, implemented on C++ side. HO is returned from C++ to JS from BaseAudioContext factory methods. JS has its own interfaces that works as a counterpart of C++ HO. There is no strong typing mechanism between C++ and JS. Implementation is based on the alignment between C++ HO and JS interface. |
| 33 | +
|
| 34 | +```cpp |
| 35 | +class GainNodeHostObject : public AudioNodeHostObject { |
| 36 | + public: |
| 37 | + explicit GainNodeHostObject( |
| 38 | + const std::shared_ptr<BaseAudioContext> &context, |
| 39 | + const GainOptions &options); |
| 40 | +
|
| 41 | + JSI_PROPERTY_GETTER_DECL(gain); |
| 42 | +
|
| 43 | + private: |
| 44 | + std::shared_ptr<AudioParamHostObject> gainParam_; |
| 45 | +}; |
| 46 | +``` |
| 47 | +```ts |
| 48 | +export interface IGainNode extends IAudioNode { |
| 49 | + readonly gain: IAudioParam; |
| 50 | +} |
| 51 | +``` |
| 52 | + |
| 53 | +- Typescript (JS) |
| 54 | + |
| 55 | + Elegant typescript wrapper around JS HO interface. |
| 56 | + |
| 57 | +```ts |
| 58 | +class GainNode extends AudioNode { |
| 59 | + readonly gain: AudioParam; |
| 60 | + |
| 61 | + constructor(context: BaseAudioContext, options?: TGainOptions) { |
| 62 | + const gainNode: IGainNode = context.context.createGain(options || {}); // context.context is C++ HO |
| 63 | + super(context, gainNode); |
| 64 | + this.gain = new AudioParam(gainNode.gain, context); |
| 65 | + } |
| 66 | +} |
| 67 | +``` |
| 68 | + |
| 69 | +## Core (C++) implementation |
| 70 | + |
| 71 | +Each AudioNode should implement one virtual method: |
| 72 | +```cpp |
| 73 | +std::shared_ptr<AudioBuffer> processNode( |
| 74 | + const std::shared_ptr<AudioBuffer> &processingBuffer, |
| 75 | + int framesToProcess) |
| 76 | +``` |
| 77 | +
|
| 78 | +It is responsible for AudioNode's processing logic. It gets input buffer as argument - `processingBus` and should return processed buffer. |
| 79 | +
|
| 80 | +```cpp |
| 81 | +std::shared_ptr<AudioBuffer> GainNode::processNode( |
| 82 | + const std::shared_ptr<AudioBuffer> &processingBuffer, |
| 83 | + int framesToProcess) { |
| 84 | + std::shared_ptr<BaseAudioContext> context = context_.lock(); |
| 85 | + if (context == nullptr) |
| 86 | + return processingBuffer; |
| 87 | + double time = context->getCurrentTime(); |
| 88 | + auto gainParamValues = gainParam_->processARateParam(framesToProcess, time); |
| 89 | + auto gainValues = gainParamValues->getChannel(0); |
| 90 | +
|
| 91 | + for (size_t i = 0; i < processingBuffer->getNumberOfChannels(); i++) { |
| 92 | + auto channel = processingBuffer->getChannel(i); |
| 93 | + channel->multiply(*gainValues, framesToProcess); |
| 94 | + } |
| 95 | +
|
| 96 | + return processingBuffer; |
| 97 | +} |
| 98 | +``` |
| 99 | + |
| 100 | +There are a few rules that should be followed when implementing C++ AudioNode core. |
| 101 | + |
| 102 | +- **Thread safety**: Each AudioNode should be created in thread safe manner. |
| 103 | + |
| 104 | +- **Heap allocations**: Heap allocations are not allowed on the Audio Thread, so all necessary data should be allocated in constructor, or pre-allocated on other thread and passed to AudioNode. |
| 105 | + |
| 106 | +- **Destructions** No destructions are allowed to happen on the Audio Thread. AudioNode destruction are handled by already active AudioDestructor. If you need to perform some cleanup, you have to delegate it to AudioDestructor. |
| 107 | + |
| 108 | +- **No locks on Audio Thread**: Locks are not allowed on the Audio Thread. Audio procssing have to be highly performant and efficient. |
| 109 | + |
| 110 | +- **No syscalls** Syscalls are not allowed on the Audio Thread, so if you need to perform some work that requires syscalls, you have to delegate it to other thread. |
| 111 | + |
| 112 | +## HostObject implementation |
| 113 | + |
| 114 | +We can distinguish three types of AudioNode's JS methods: |
| 115 | + |
| 116 | +1. **getter** |
| 117 | + |
| 118 | +```ts |
| 119 | +const fftSize = analyserNode.fftSize; |
| 120 | +``` |
| 121 | + |
| 122 | +C++ counter part is `JSI_PROPERTY_GETTER`. It just returns some value. |
| 123 | + |
| 124 | +```cpp |
| 125 | +JSI_PROPERTY_GETTER_IMPL(AnalyserNodeHostObject, fftSize) { |
| 126 | + return {fftSize_}; |
| 127 | +} |
| 128 | +``` |
| 129 | +
|
| 130 | +2. **setter** |
| 131 | +
|
| 132 | +C++ counterpart is `JSI_PROPERTY_SETTER`. It just receives some value. |
| 133 | +
|
| 134 | +```ts |
| 135 | +analyserNode.fftSize = 2048; |
| 136 | +``` |
| 137 | + |
| 138 | +```cpp |
| 139 | +JSI_PROPERTY_SETTER_IMPL(AnalyserNodeHostObject, minDecibels) { |
| 140 | + auto analyserNode = std::static_pointer_cast<AnalyserNode>(node_); |
| 141 | + auto minDecibels = static_cast<float>(value.getNumber()); |
| 142 | + auto event = [analyserNode, minDecibels](BaseAudioContext&) { |
| 143 | + analyserNode->setMinDecibels(minDecibels); |
| 144 | + }; |
| 145 | + analyserNode->scheduleAudioEvent(std::move(event)); |
| 146 | + minDecibels_ = minDecibels; |
| 147 | +} |
| 148 | +``` |
| 149 | +
|
| 150 | +
|
| 151 | +3. **function** |
| 152 | +
|
| 153 | +C++ counterpart is `JSI_HOST_FUNCTION`. It is a common function that can receive arguments and return some value. |
| 154 | +
|
| 155 | +```ts |
| 156 | +const fftOutput = new Uint8Array(analyser.frequencyBinCount); |
| 157 | +analyserNode.getByteFrequencyData(fftOutput); |
| 158 | +``` |
| 159 | + |
| 160 | +```cpp |
| 161 | +JSI_HOST_FUNCTION_IMPL(AnalyserNodeHostObject, getByteFrequencyData) { |
| 162 | + auto arrayBuffer = |
| 163 | + args[0].getObject(runtime).getPropertyAsObject(runtime, "buffer").getArrayBuffer(runtime); |
| 164 | + auto data = arrayBuffer.data(runtime); |
| 165 | + auto length = static_cast<int>(arrayBuffer.size(runtime)); |
| 166 | + |
| 167 | + auto analyserNode = std::static_pointer_cast<AnalyserNode>(node_); |
| 168 | + analyserNode->getByteFrequencyData(data, length); |
| 169 | + |
| 170 | + return jsi::Value::undefined(); |
| 171 | +} |
| 172 | +``` |
| 173 | +
|
| 174 | +All methods should be registerd in C++ HO constructor: |
| 175 | +
|
| 176 | +```cpp |
| 177 | +AnalyserNodeHostObject::AnalyserNodeHostObject(const std::shared_ptr<BaseAudioContext>& context, const AnalyserOptions &options) |
| 178 | + : /* ... */ { |
| 179 | + addGetters(JSI_EXPORT_PROPERTY_GETTER(AnalyserNodeHostObject, fftSize)); |
| 180 | + addSetters(JSI_EXPORT_PROPERTY_SETTER(AnalyserNodeHostObject, fftSize)); |
| 181 | + addFunctions(JSI_EXPORT_FUNCTION(AnalyserNodeHostObject, getByteFrequencyData),); |
| 182 | +} |
| 183 | +``` |
| 184 | + |
| 185 | +#### Shadow state (C++) |
| 186 | + |
| 187 | +Shadow state is a mechanism introduced in order to make communication between JS and the Audio Thread lock-free. AudioNodeHostObject stores the set of properties, which are modified only by JS thread (the same set C++ AudioNode has). Everytime we want to access some property from JS, we can just return property from shadow state, when we modify some property we have to update shadow state and schedule update event on Audio Event Loop (SPSC). By following that manner we can skip accessing AudioNode state, that is also accessed by the Audio Thread - no need to lock or use atomic variables. |
| 188 | + |
| 189 | +```cpp |
| 190 | +class OscillatorNodeHostObject : public AudioScheduledSourceNodeHostObject { |
| 191 | + public: |
| 192 | + /* ... */ |
| 193 | + |
| 194 | + JSI_PROPERTY_GETTER_DECL(type); |
| 195 | + JSI_PROPERTY_SETTER_DECL(type); |
| 196 | + |
| 197 | + private: |
| 198 | + /* ... */ |
| 199 | + OscillatorType type_; |
| 200 | +}; |
| 201 | +``` |
| 202 | +
|
| 203 | +```cpp |
| 204 | +JSI_PROPERTY_GETTER_IMPL(OscillatorNodeHostObject, type) { |
| 205 | + return jsi::String::createFromUtf8(runtime, js_enum_parser::oscillatorTypeToString(type_)); |
| 206 | +} |
| 207 | +
|
| 208 | +JSI_PROPERTY_SETTER_IMPL(OscillatorNodeHostObject, type) { |
| 209 | + auto oscillatorNode = std::static_pointer_cast<OscillatorNode>(node_); |
| 210 | + auto type = js_enum_parser::oscillatorTypeFromString(value.asString(runtime).utf8(runtime)); |
| 211 | +
|
| 212 | + auto event = [oscillatorNode, type](BaseAudioContext &) { |
| 213 | + oscillatorNode->setType(type); |
| 214 | + }; |
| 215 | + type_ = type; |
| 216 | +
|
| 217 | + oscillatorNode->scheduleAudioEvent(std::move(event)); |
| 218 | +} |
| 219 | +``` |
| 220 | + |
| 221 | +#### Communication between JS Thread and Audio Thread |
| 222 | + |
| 223 | +**getters** and **setters** |
| 224 | + |
| 225 | +1. Property is primitive and is not modified by the Audio Thread. |
| 226 | + |
| 227 | + Shadow state design pattern should be followed. |
| 228 | + |
| 229 | +2. Property is not primitive and is not modified by the Audio Thread. |
| 230 | + |
| 231 | + It should be stored in TS layer and copied to AudioNode |
| 232 | + |
| 233 | +3. Property is primitive and can be modified by the Audio Thread. |
| 234 | + |
| 235 | + In C++ core it should be an atomic variable that allows to access it in thread-safe manner from both threads. |
| 236 | + |
| 237 | +```cpp |
| 238 | +class AudioParam { |
| 239 | +public: |
| 240 | + /* ... */ |
| 241 | + |
| 242 | + [[nodiscard]] inline float getValue() const noexcept { |
| 243 | + return value_.load(std::memory_order_relaxed); |
| 244 | + } |
| 245 | + |
| 246 | + inline void setValue(float value) { |
| 247 | + value_.store(std::clamp(value, minValue_, maxValue_), std::memory_order_release); |
| 248 | + } |
| 249 | + |
| 250 | + /* ... */ |
| 251 | + |
| 252 | +private: |
| 253 | + std::atomic<float> value_; |
| 254 | + |
| 255 | + /* ... */ |
| 256 | +}; |
| 257 | +``` |
| 258 | +
|
| 259 | +```cpp |
| 260 | +JSI_PROPERTY_GETTER_IMPL(AudioParamHostObject, value) { |
| 261 | + return {param_->getValue()}; |
| 262 | +} |
| 263 | +
|
| 264 | +JSI_PROPERTY_SETTER_IMPL(AudioParamHostObject, value) { |
| 265 | + auto event = [param = param_, value = static_cast<float>(value.getNumber())](BaseAudioContext &) { |
| 266 | + param->setValue(value); |
| 267 | + }; |
| 268 | +
|
| 269 | + param_->scheduleAudioEvent(std::move(event)); |
| 270 | +} |
| 271 | +``` |
| 272 | + |
| 273 | +4. Property is not primitive and can be modified by the Audio Thread. |
| 274 | + In C++ core triple buffer pattern should be followed. It allows to have one copy of property for reader, one for writer and one for pending update. On each update we just swap pending update with writer, and on each read we just read from reader. In that manner we can skip locks and just operate on atomic indices. |
| 275 | + |
| 276 | + Check AnalyserNode implementation for example or this [article](https://medium.com/@sgn00/triple-buffer-lock-free-concurrency-primitive-611848627a1e) for more details. |
| 277 | + |
| 278 | +**functions** |
| 279 | + |
| 280 | +Function's should follow the same thread-safe lock-free patterns as getters/setters. Set of properties read/write by the function determines mechanisms that should be used in implementation. |
0 commit comments