Respond to feedback

Signed-off-by: James Sturtevant <jsturtevant@gmail.com>

Author: jsturtevant

docs/paging-development-notes.md

@@ -7,46 +7,15 @@ design in which the guest is aware of a readonly snapshot from
 which it is being run, and manages its own copy-on-write.
 
 Because of this, there are two very fundamental regions of the guest
-physical address space, which are always populated: one, near the
-bottom of memory (starting at GPA `0x1000`), is a
-(hypervisor-enforced) readonly mapping of the base snapshot from which
-this guest is being evolved. Another, at the top of memory, is simply
+physical address space, which are always populated: one, at the very
+bottom of memory, is a (hypervisor-enforced) readonly mapping of the
+base snapshot from which this guest is being evolved. Another, at the top of memory, is simply
 a large bag of blank pages: scratch memory into which this VM can
 write.
 
-```
-  Guest Physical Address Space (GPA)
-
-  +-------------------------------+ MAX_GPA
-  |    Exn Stack, Bookkeeping     |
-  |    (scratch size, allocator   |
-  |     state, reserved PT slot)  |
-  +-------------------------------+
-  |      Free Scratch Memory      |
-  +-------------------------------+
-  |         Output Data           |
-  +-------------------------------+
-  |         Input Data            |
-  +-------------------------------+
-  |                               |
-  |     (unmapped — no RAM        |
-  |      backing these addrs)     |
-  |                               |
-  +-------------------------------+
-  |                               |
-  |  Snapshot (RO / CoW on write) |
-  |    Guest Page Tables          |
-  |    Init Data                  |
-  |    Guest Heap                 |
-  |    PEB                        |
-  |    Guest Binary               |
-  |                               |
-  +-------------------------------+ 0x1000
-  |    (null guard page)          |
-  +-------------------------------+ 0x0000
-```
-
-
+For the detailed layout of each region, including field offsets, see
+the diagrams and comments in [`src/hyperlight_host/src/mem/layout.rs`](../src/hyperlight_host/src/mem/layout.rs)
+and the constants in [`hyperlight_common::layout`](../src/hyperlight_common/src/layout.rs).
 
 ## The scratch map
 
@@ -92,63 +61,56 @@ original virtual address to point to the new page.
     Snapshot page at GPA 0x5000 is untouched.
 ```
 
-The page table
-entries to do this will likely need to be copied themselves, and so a
+The page table entries to do this will likely need to be copied themselves, and so a
 ready supply of already-mapped scratch pages to use for replacement
-page tables is set up by the Host. The guest keeps a mapping of the entire scratch
-physical memory into virtual memory at a fixed offset
-(`scratch_base_gva - scratch_base_gpa`), so that any scratch physical
-address can be accessed by adding this offset.
+page tables is needed. Currently, the guest accomplishes this by
+keeping an identity mapping of the entire scratch memory around.
 
 The host and the guest need to agree on the location of this mapping,
 so that (a) the host can create it when first setting up a blank guest
 and (b) the host can ignore it when taking a snapshot (see below).
 
-The host creates the scratch map at the top of virtual memory
-(`MAX_GVA - scratch_size + 1`) and at the top of physical memory
-(`MAX_GPA - scratch_size + 1`). In the future, we may add support for a guest to
+Currently, the host always creates the scratch map at the top of
+virtual memory.  In the future, we may add support for a guest to
 request that it be moved.
 
 ## The snapshot mapping
 
 The snapshot page tables must be mapped at some virtual address so
-that the guest can read and copy them during CoW operations.
-Today, the host simply
-copies the page tables into scratch when restoring a sandbox, and the
-guest works on those scratch copies directly. 
+that the guest can read and copy them during CoW operations. The
+preferred approach is to map the snapshot page tables directly from
+the snapshot region into the guest's virtual address space.
 
-## Top-of-scratch metadata layout
+However, on amd64, this is complicated by architectural constraints.
+Currently, the host simply copies the page tables into scratch when
+restoring a sandbox, and the guest works on those scratch copies
+directly. In the near future, we expect to be able to use the
+preferred approach on aarch64, and with some minor hypervisor changes,
+on amd64 as well.
 
-The top page of the scratch region contains structured metadata at
-fixed offsets down from the top:
-
-| Offset from top | Field                                |
-|-----------------|--------------------------------------|
-| `0x08`          | Scratch size (`u64`)                 |
-| `0x10`          | Allocator state (`u64`)              |
-| `0x18`          | Reserved snapshot PT base (`u64`)    |
-| `0x20`          | Exception stack starts here          |
+## Top-of-scratch metadata layout
 
+The top of the scratch region contains structured metadata at fixed
+offsets such as the scratch size, allocator state and where the excpetions starts.
 These offsets are defined as `SCRATCH_TOP_*` constants in
-`hyperlight_common::layout`.
+[`hyperlight_common::layout`](../src/hyperlight_common/src/layout.rs), which has detailed comments on each
+field.
 
 ## The physical page allocator
 
 The host needs to be able to reset the state of the physical page
-allocator when resuming from a snapshot. We use a simple bump
-allocator as a physical page allocator, with no support for free,
+allocator when resuming from a snapshot. Currently, we use a simple
+bump allocator as a physical page allocator, with no support for free,
 since pages not in use will automatically be omitted from a snapshot.
 The allocator state is a single `u64` tracking the address of the
-first free page, located at offset `0x10` from the top of scratch
-(see layout above). The guest advances it atomically via `lock xadd`.
+first free page, located below the metadata at the top of scratch.
+The guest advances it atomically.
 
 ## The guest exception stack
 
 Similarly, the guest needs a stack that is always writable, in order
-to be able to take exceptions to it. The exception stack begins at
-offset `0x20` from the top of the scratch region (below the metadata
-fields described above) and grows downward through the remainder of
-the top page.
+to be able to take exceptions to it. The exception stack begins below
+the metadata at the top of the scratch region and grows downward.
 
 ## Taking a snapshot
 
@@ -177,26 +139,17 @@ calls, i.e. there may be no calls in flight at the time of
 snapshotting. This is not enforced, but odd things may happen if it is
 violated.
 
-I/O buffers are statically allocated at the bottom of the scratch
-region:
+Buffer management between the host and guest is needed to pass call
+arguments and return values. Ideally, buffers would be dynamically
+allocated from the scratch region as needed.
 
-```
-+-------------------------------------------+ (top of scratch)
-|       Exn Stack, Bookkeeping              |
-|  (scratch size, allocator state,          |
-|   reserved PT base)                       |
-+-------------------------------------------+
-|           Free Scratch Memory             |
-+-------------------------------------------+
-|                Output Data                |
-+-------------------------------------------+
-|                Input Data                 |
-+-------------------------------------------+ (scratch base)
-```
+Currently, I/O buffers are statically allocated at the bottom of the
+scratch region. This is a stopgap pending improved
+physical allocation and buffer management.
 
-The minimum scratch size (`min_scratch_size()`) accounts for these
-buffers plus overhead for the Task State Segment (TSS), Interrupt Descriptor Table (IDT), page table CoW, a minimal
-non-exception stack, and the exception stack and metadata.
+The minimum scratch size is calculated by `min_scratch_size()` in the
+architecture-specific layout modules under `hyperlight_common`; see
+that function for the detailed breakdown of required overhead.
 
 ## Creating a fresh guest
 
@@ -207,9 +160,11 @@ virtual memory.  If the ELF has segments whose virtual addresses
 overlap with the scratch map, an error will be returned.
 
 In the current startup path, the host enters the guest with
-`RSP` pointing to the exception stack. Early guest init then
-allocates the main stack at `MAIN_STACK_TOP_GVA`, switches to it,
-and continues generic initialization.
+the stack pointer pointing to the exception stack. Early guest init
+then allocates the main stack at `MAIN_STACK_TOP_GVA`, switches to
+it, and continues generic initialization. Note that exception stack
+overflows can be difficult to detect, since there is no guard page
+below the exception stack within the scratch region.
 
 # Architecture-specific details of virtual memory setup