[WARP] More documentation updates and python examples

Author: emesare

docs/guide/warp.md

@@ -24,12 +24,12 @@ When a function matches, we will apply the following information:
 - Symbol
     - Name
     - Demangled type
-- User defined type
+- User-defined type
     - Calling convention
     - Parameter names
     - Parameter types
     - Return type
-- User defined variables
+- User-defined variables
     - Name
     - Type
 - Comments
@@ -52,8 +52,8 @@ Files are automatically loaded from two locations when Binary Ninja starts:
 ### Manually
 
 Aside from using the signature directory you can load any WARP file manually using the command `WARP\\Load File` or via
-the UI sidebar, they both do the same thing. Once the file is loaded, you do not need to load it for every view, it is
-available globally.
+the UI sidebar, they both do the same thing. Once the file is loaded, you do not need to load it for every view, any view
+that performs function matching will have access to the loaded file.
 
 ???+ Info "Tip"
     When loading signatures you may encounter a dialog asking to "Override file target?" this happens when your file
@@ -62,35 +62,20 @@ available globally.
 
 ## Creating WARP Files
 
-Before you actually can create these WARP files, you must identify the binary files relevant to the target binary. Once
-that is done, you can determine which of the following procedures is best:
+Before you actually can create these WARP files, you must identify the binary files relevant to the target binary. This
+can differ depending on the type of binary you are working with, but once you have those files you can create the files
+using the command `WARP\\Process` or via the UI sidebar.
 
-=== "From the current view"
-    You can create signatures for the current view by running the
-    command `WARP\\Create\\From View` or by hitting the associated
-    button in the WARP sidebar.
+![Processor Dialog](../img/warp/processor_dialog.png "Processor Dialog"){ width="600" }
 
-    ![From View](../img/warp/create_from_view.png "From View")
-
-=== "From a file"
-    Using the command `WARP\\Create\\From File(s)` you can create
-    a signature file for an external file. This is useful if you
-    are generating signatures for library files (`.a`, `.lib`, `.rlib`).
-
-    ![From File](../img/warp/create_from_file.png "From File")
-
-=== "From the current project"
-    When dealing with a large dataset, you will want a way to process
-    files in parallel, using the `WARP\\Create\\From Project` command
-    you can generate signatures for any set of files in a project, this
-    includes archive formats like library files (`.a`, `.lib`, `.rlib`).
-
-    ![From Project](../img/warp/create_from_project.png "From Project")
+The processor dialog will allow you to select the files you want to process, including directories and project files. To
+add more files, you can use the "+" button and select the files you want to add. If you have more than one file to process 
+the worker count will control how many entries are processed in parallel.
 
 ???+ Info "Tip"
     You can also create signature files using the provided API, see the [API section](#api) for more details.
 
-Both the file and project signature creation support loading the following file formats:
+The following file formats are supported:
 
 - Binary files (`.exe`, `.so`, `.dylib`)
 - WARP files (`.warp`)
@@ -100,6 +85,14 @@ Both the file and project signature creation support loading the following file
 - Archive files (`.a`, `.lib`, `.rlib`)
     - The archive entry files will be extracted to a temporary directory for processing.
 
+After processing is complete, you will be shown a dialog with the results, from here you can save the file to disk or commit
+the data to the server using the commit dialog.
+
+![Report Dialog](../img/warp/report_dialog.png "Report Dialog"){ width="600" }
+
+If you are trying to commit the data to the server, make sure you have [connected](#connecting) to the server first with
+a valid API key, otherwise you will get an error when commiting.
+
 ### Including specific functions
 
 Sometimes you may not want to include every function but a subset, in that case you can tag functions to include with
@@ -109,9 +102,8 @@ will be prompted whether you want to keep the existing data, you will want to sa
 
 ### File size
 
-Information in the WARP file will be deduplicated across all processed files automatically.
-If your files are too large, try and adjust the file data to something like "Symbols" only, and if you are looking to
-make the files load quicker, turn off compression.
+Information in the WARP file will be deduplicated across all processed files automatically. If your files are too large, 
+try and adjust the file data to something like "Symbols" only.
 
 ## Networked Functionality
 
@@ -127,7 +119,7 @@ and the default primary server is https://warp.binary.ninja. You can also give i
 in as your user, and have access to push data to your sources using `warp.container.serverApiKey`.
 
 Once restarted, you should see a log message indicating you have connected. You can also verify connections in the WARP 
-sidebar under the "Containers" tab which should list the provided WARP server(s) alongside any of your sources you have created.
+sidebar under the "Containers" tab, which should list the provided WARP server(s) alongside any of your sources you have created.
 
 ### Pulling Networked Data
 
@@ -140,7 +132,7 @@ globally by modifying the setting `warp.fetcher.allowedSourceTags` as a comma se
 from within the server UI, either by source users or the server admin, the tags "official" and "trusted" may only be added
 or removed by the server admin.
 
-![Fetch Dialog](../img/warp/fetch_dialog.png "Fetch Dialog")
+![Fetch Dialog](../img/warp/fetch_dialog.png "Fetch Dialog"){ width="600" }
 
 ???+ Info "Tip"
     Fetching of function information from the server will also be done on demand when navigating to a function for the first time
@@ -164,11 +156,18 @@ To get an API key using the website:
 
 Once restarted, you should see a log message indicating you have connected as the associated user.
 
+> [WARP.Plugin] Server 'https://warp.binary.ninja' connected, logged in as user 'binary-dog-4213'
+
 After logging in, you can create a new source on the server by right-clicking in the container sources tab and selecting
-"Add Source" you can also do this via the website. 
+"Add Source" you can also do this via the website or in the processor dialog (see below). 
+
+Once you have created a source, you can start pushing your information to the server by invoking the processor dialog
+and opening the commit dialog after processing of the data has completed.
+
+![Commit Dialog](../img/warp/commit_dialog.png "Commit Dialog"){ width="600" }
 
-Once you have created a source, you can start pushing your WARP files to them using the command `WARP\\Commit File` or 
-using the ⬆ button within the WARP sidebar.
+Each source operates as its own collection of function and type information, creating a new source is as simple as clicking
+the "+" button and giving it a name. The sources are synced on the server and can be managed from the website.
 
 ## Overwriting Matched Functions
 
@@ -189,23 +188,32 @@ from being matched automatically in the future.
 
 ## API
 
-To create, query and load WARP data programmatically, we provide a [Python API]. For those looking to interact with WARP
+To create, query, and load WARP data programmatically, we provide a [Python API]. For those looking to interact with WARP
 from Rust because the plugin is open source, you can depend _directly_ on the [Rust plugin], skipping the FFI entirely.
 
 ### Rust example (recommended)
 
-This example will use the Rust API directly to generate WARP files from given inputs, it will automatically parallelize
-the work, so it will be much faster than the Python example.
+This example will use the Rust API directly to generate WARP files from given inputs, this operates with the same processor
+as the UI and supports all the same options.
 
 Find the example [here](https://github.com/Vector35/binaryninja-api/tree/dev/plugins/warp/examples/headless).
 
 ### Python example
 
+This example will open a binary in Binary Ninja then output a WARP signature file using the core processor API. This is the
+easiest way to get started with the API, as it will use the same processor as the UI and support all the same options.
+
+Find the example [here](https://github.com/Vector35/binaryninja-api/tree/dev/plugins/warp/examples/create_signatures.py).
+
+### Python example (advanced)
+
 This example will open a binary in Binary Ninja then output a WARP signature file.
 
 The flexibility of the API allows you to include or exclude any functions you want from the creation of the signature file.
+The cost is that it does not use the same processor as the UI and you will need to implement the same logic yourself for
+selecting the functions and processing in parallel.
 
-Find the example [here](https://github.com/Vector35/binaryninja-api/tree/dev/plugins/warp/examples/create_signatures.py).
+Find the example [here](https://github.com/Vector35/binaryninja-api/tree/dev/plugins/warp/examples/create_signatures_advanced.py).
 
 ## Troubleshooting
 
@@ -230,20 +238,34 @@ The function GUID will differ if the instruction highlights are not exactly the
 
 When running the matcher manually, you may get a warning about no relocatable regions found; this means you have no defined
 sections or segments in your view. For WARP to work we must have some range of address space to work with, without it the
-function GUIDs will likely be inconsistent if the functions can be based at different addresses.
+function GUIDs will likely be inconsistent if the functions can be based at different addresses. Once you have updated the sections,
+or segments, you should [regenerate the function GUIDs](#regenerating-the-function-guids).
 
 ### "Relocatable region has a low start-address" warning
 
 WARP uses relocatable regions to determine relocatable addresses encoded in instructions, if you have a relocatable region
-that covers a low address space, WARP may mask regular constants and other irrelevant instructions. This warning mostly
+that covers a low-address space, WARP may mask regular constants and other irrelevant instructions. This warning mostly
 affects firmware binaries (or other mapped views), if you have not rebased the view to the correct image base, then you
-should as it will fix this issue
+should as it will fix this issue. Once you have rebased the view, you should [regenerate the function GUIDs](#regenerating-the-function-guids).
 
 ### Failed to connect to the server
 
 If you fail to connect to a WARP server, you will receive an error in the log. Outside typical network connectivity issues 
 it is possible the provided server URL is malformed, verify that the URL looks similar to the default server URL: `https://warp.binary.ninja`
 
+### Regenerating the function GUIDs
+
+After updating the binary with new sections, segments, or base address, you will need to regenerate the function GUIDs to ensure 
+they are accurate and up to date. Unfortunately, this is currently not something we can automatically do for you. To do this
+you will need to run something like this:
+
+```python
+from binaryninja.warp import *
+for func in bv.functions:
+    func.remove_metadata('warp_function_guid')
+    get_function_guid(func)
+```
+
 ## Glossary
 
 Here is a list of terms used and a simplified description, please see the [WARP] spec repository for a more detailed description.
@@ -252,17 +274,17 @@ Here is a list of terms used and a simplified description, please see the [WARP]
 A **Target** defines platform-specific information needed to filter out irrelevant WARP information.
 
 ### Container
-A **Container** stores and manages WARP data, whether in memory, on disk or over the network. Each container has its own collection of sources.
+A **Container** stores and manages WARP data, whether in memory, on disk, or over the network. Each container has its own collection of sources.
 
 ### Source
 A **Source** is a collection of WARP data within a container, like a file containing function and type information.
 
 ### Source Tag
-A **Source Tag** is an arbitrary string that is used for filtering fetched function data from containers, useful when dealing with larger potentially
-untrusted datasets.
+A **Source Tag** is an arbitrary string, which is used for filtering fetched function data from containers, useful when dealing with 
+larger and potentially untrusted datasets.
 
 ### Function
-A **Function** in WARP represents the collection of metadata that we wish to transfer, such as the symbol, comments and types.
+A **Function** in WARP represents the collection of metadata that we wish to transfer, such as the symbol, comments, and types.
 
 ### Function GUID
 A **Function GUID** is a unique ID derived from the contents of the function, allowing matching across different binaries.
@@ -271,6 +293,13 @@ A **Function GUID** is a unique ID derived from the contents of the function, al
 A **Constraint** helps ensure accurate function matching by verifying specific properties are shared between functions.
 For example, a referenced function would be used as a constraint.
 
+### File
+A **File** in WARP represents a collection of chunks, which are use to store function and type information. It is exposed 
+via the API as a way to pass around the data to different parts of the application, such as the processor and the container.
+
+### Chunk
+A **Chunk** is a collection of either function or type information, stored as a flatbuffer.
+
 [WARP]: https://github.com/vector35/warp
 [Install Directory]: https://docs.binary.ninja/guide/#binary-folder
 [User Directory]: https://docs.binary.ninja/guide/#user-folder

docs/img/warp/commit_dialog.png

@@ -2,41 +2,27 @@
 
 import sys
 from pathlib import Path
-from binaryninja import load
-from binaryninja.warp import WarpContainer, WarpFunction, WarpTarget
+from binaryninja.warp import *
 
 def process_binary(input_file: str, output_dir: str) -> None:
     input_path = Path(input_file)
     output_dir = Path(output_dir)
     output_dir.mkdir(parents=True, exist_ok=True)
-    bv = load(input_path)
-    bv.update_analysis_and_wait()
-    if not bv:
+    output_file = output_dir / f"{input_path.stem}.warp"
+    processor = WarpProcessor()
+    processor.add_path(input_file)
+    file = processor.start()
+    if not file:
         return
+    buffer = file.to_data_buffer()
+    with open(output_file, 'wb') as f:
+        f.write(bytes(buffer))
+    print(f"Wrote {len(buffer)} bytes to {output_file}")
 
-    # For the sake of this example we are going to assume the container "User"
-    # is available, in the future we might want to make containers constructable
-    container = WarpContainer.by_name("User")
-    output_file = output_dir / f"{input_path.stem}_analysis.warp"
-    # Add the source so we can add functions to it and then commit it (write to disk)
-    source = container.add_source(str(output_file))
-    if source is None:
-        print(f"Failed to create source {str(output_file)}")
-        return
-
-    # NOTE: You probably want to pull the platform from the function, but for this example it's fine.
-    target = WarpTarget(bv.platform)
-    # NOTE: You probably want to filter for functions with actual annotations, no point to signature a function with no symbol.
-    functions_to_warp = [WarpFunction(func) for func in bv.functions]
-    container.add_functions(target, source, functions_to_warp)
-
-    # Actually write the warp file to disk.
-    container.commit_source(source)
-    bv.file.close()
-    print(f"committed {len(functions_to_warp)} functions to {output_file}")
 
 if __name__ == "__main__":
     if len(sys.argv) != 3:
         print(f"Usage: {sys.argv[0]} <input_binary> <output_directory>")
         sys.exit(1)
+    binaryninja._init_plugins()
     process_binary(sys.argv[1], sys.argv[2])

docs/img/warp/create_from_file.png

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+
+import sys
+from pathlib import Path
+from binaryninja import load
+from binaryninja.warp import WarpContainer, WarpFunction, WarpTarget
+
+def process_binary(input_file: str, output_dir: str) -> None:
+    input_path = Path(input_file)
+    output_dir = Path(output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+    bv = load(input_path)
+    bv.update_analysis_and_wait()
+    if not bv:
+        return
+
+    container = WarpContainer.add("Python")
+    output_file = output_dir / f"{input_path.stem}_analysis.warp"
+    # Add the source so we can add functions to it and then commit it (write to disk)
+    source = container.add_source(str(output_file))
+    if source is None:
+        print(f"Failed to create source {str(output_file)}")
+        return
+
+    # NOTE: You probably want to pull the platform from the function, but for this example it's fine.
+    target = WarpTarget(bv.platform)
+    # NOTE: You probably want to filter for functions with actual annotations, no point to signature a function with no symbol.
+    functions_to_warp = [WarpFunction(func) for func in bv.functions]
+    container.add_functions(target, source, functions_to_warp)
+
+    # Actually write the warp file to disk.
+    container.commit_source(source)
+    bv.file.close()
+    print(f"committed {len(functions_to_warp)} functions to {output_file}")
+
+if __name__ == "__main__":
+    if len(sys.argv) != 3:
+        print(f"Usage: {sys.argv[0]} <input_binary> <output_directory>")
+        sys.exit(1)
+    process_binary(sys.argv[1], sys.argv[2])