Improve coverage, docs, and add new to_docs subcommand

Author: SamuelMarks

ARCHITECTURE.md

@@ -9,7 +9,7 @@
 
 The **cdd-python-client** tool acts as a dedicated compiler and transpiler. Its fundamental architecture follows standard compiler design principles, divided into three distinct phases: **Frontend (Parsing)**, **Intermediate Representation (IR)**, and **Backend (Emitting)**.
 
-This decoupled design ensures that any format capable of being parsed into the IR can subsequently be emitted into any supported output format, whether that is a server-side route, a client-side SDK, a database ORM, or an OpenAPI specification.
+This decoupled design ensures that any format capable of being parsed into the IR can subsequently be emitted into any supported output format, whether that is a server-side route, a client-side SDK, a database ORM, or an OpenAPI specification. As a `Client-only` focussed project, its emitters are heavily optimized for generating Python API SDKs and Client Wrappers using LibCST and Pydantic.
 
 ## 🏗 High-Level Overview
 
@@ -23,7 +23,7 @@ graph TD
 
     subgraph Frontend [Parsers]
         A[OpenAPI .yaml/.json]:::endpoint --> P1(OpenAPI Parser):::frontend
-        B[`Python` Models / Source]:::endpoint --> P2(`Python` Parser):::frontend
+        B[Python Models / Source]:::endpoint --> P2(Python Parser):::frontend
         C[Server Routes / Frameworks]:::endpoint --> P3(Framework Parser):::frontend
         D[Client SDKs / ORMs]:::endpoint --> P4(Ext Parser):::frontend
     end
@@ -34,10 +34,9 @@ graph TD
 
     subgraph Backend [Emitters]
         E1(OpenAPI Emitter):::backend --> X[OpenAPI .yaml/.json]:::endpoint
-        E2(`Python` Emitter):::backend --> Y[`Python` Models / Structs]:::endpoint
+        E2(Python Emitter):::backend --> Y[Python Models / Structs]:::endpoint
         E3(Server Emitter):::backend --> Z[Server Routes / Controllers]:::endpoint
         E4(Client Emitter):::backend --> W[Client SDKs / API Calls]:::endpoint
-        E5(Data Emitter):::backend --> V[ORM Models / CLI Parsers]:::endpoint
     end
 
     P1 --> IR
@@ -49,7 +48,6 @@ graph TD
     IR --> E2
     IR --> E3
     IR --> E4
-    IR --> E5
 ```
 
 ## 🧩 Core Components
@@ -58,7 +56,7 @@ graph TD
 
 The Frontend's responsibility is to read an input source and translate it into the universal CDD Intermediate Representation (IR).
 
-* **Static Analysis (AST-Driven)**: For `Python` source code, the tool **does not** use dynamic reflection or execute the code. Instead, it reads the source files, generates an Abstract Syntax Tree (AST), and navigates the tree to extract classes, structs, functions, type signatures, API client definitions, server routes (like FastAPI mocks), and docstrings.
+* **Static Analysis (AST-Driven)**: For `Python` source code, the tool **does not** use dynamic reflection or execute the code. Instead, it reads the source files using LibCST, generates an Abstract Syntax Tree (AST), and navigates the tree to extract classes, structs, functions, type signatures, API client definitions, server routes, and docstrings.
 * **OpenAPI Parsing**: For OpenAPI and JSON Schema inputs, the parser normalizes the structure, resolving internal `$ref`s and extracting properties, endpoints (client or server perspectives), and metadata into the IR.
 
 ### 2. Intermediate Representation (IR)
@@ -75,9 +73,8 @@ By standardizing on a single IR (heavily inspired by OpenAPI / JSON Schema primi
 The Backend's responsibility is to take the universal IR and generate valid target output. Emitters can be written to support various environments (e.g., Client vs Server, Web vs CLI).
 
 * **Code Generation**: Emitters iterate over the IR and generate idiomatic `Python` source code. 
-  * A **Server Emitter** creates routing controllers and request-validation logic (for example, generating local test-mock servers).
-  * A **Client Emitter** creates API wrappers, fetch functions, and response-parsing logic.
-* **Database & CLI Generation**: Emitters can also target ORM models or command-line parsers by mapping IR properties to database columns or CLI arguments.
+  * A **Server Emitter** creates routing controllers and request-validation logic (such as mocks).
+  * A **Client Emitter** creates API wrappers, fetch functions, and response-parsing logic tailored for Python clients.
 * **Specification Generation**: Emitters translating back to OpenAPI serialize the IR into standard OpenAPI 3.x JSON or YAML, rigorously formatting descriptions, type constraints, and endpoint schemas based on what was parsed from the source code.
 
 ## 🔄 Extensibility
@@ -91,4 +88,4 @@ Because of the IR-centric design, adding support for a new `Python` framework (e
 1. **A Single Source of Truth**: Developers should be able to maintain their definitions in whichever format is most ergonomic for their team (OpenAPI files, Native Code, Client libraries, ORM models) and generate the rest.
 2. **Zero-Execution Parsing**: Ensure security and resilience by strictly statically analyzing inputs. The compiler must never need to run the target code to understand its structure.
 3. **Lossless Conversion**: Maximize the retention of metadata (e.g., type annotations, docstrings, default values, validators) during the transition `Source -> IR -> Target`.
-4. **Symmetric Operations**: An Endpoint in the IR holds all the information necessary to generate both the Server-side controller that fulfills it, and the Client-side SDK method that calls it.
+4. **Symmetric Operations**: An Endpoint in the IR holds all the information necessary to generate both the Server-side controller that fulfills it, and the Client-side SDK method that calls it.

README.md

@@ -1,35 +1,37 @@
-cdd-python-client
-=================
+cdd-Python
+============
 
 [![License](https://img.shields.io/badge/license-Apache--2.0%20OR%20MIT-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![uv](https://github.com/offscale/cdd-python-client/actions/workflows/uv.yml/badge.svg)](https://github.com/offscale/cdd-python-client/actions/workflows/uv.yml)
+[![CI/CD](https://github.com/offscale/cdd-python-client/workflows/CI/badge.svg)](https://github.com/offscale/cdd-python-client/actions)
 ![Test Coverage](https://img.shields.io/badge/Test_Coverage-100.0%25-brightgreen)
 ![Doc Coverage](https://img.shields.io/badge/Doc_Coverage-100.0%25-brightgreen)
 
-Welcome to **cdd-python-client**, a code-generation and compilation tool bridging the gap between OpenAPI specifications
-and native `Python` source code.
+OpenAPI ↔ Python. This is one compiler in a suite, all focussed on the same task: Compiler Driven Development (CDD).
 
-This toolset allows you to fluidly convert between your language's native constructs (like classes, structs, functions,
-routing, clients, and ORM models) and OpenAPI specifications, ensuring a single source of truth without sacrificing
-developer ergonomics.
+Each compiler is written in its target language, is whitespace and comment sensitive, and has both an SDK and CLI.
+
+The CLI—at a minimum—has:
+- `cdd --help`
+- `cdd --version`
+- `cdd sync --from-openapi spec.json --to-python out_dir`
+- `cdd sync --from-python client.py --to-openapi spec.json`
+- `cdd to_docs_json --no-imports --no-wrapping -i spec.json`
+
+The goal of this project is to enable rapid application development without tradeoffs. Tradeoffs of Protocol Buffers / Thrift etc. are an untouchable "generated" directory and package, compile-time and/or runtime overhead. Tradeoffs of Java or JavaScript for everything are: overhead in hardware access, offline mode, ML inefficiency, and more. And neither of these alterantive approaches are truly integrated into your target system, test frameworks, and bigger abstractions you build in your app. Tradeoffs in CDD are code duplication (but CDD handles the synchronisation for you).
 
 ## 🚀 Capabilities
 
-The `cdd-python-client` compiler leverages a unified architecture to support various facets of API and code lifecycle
-management.
+The `cdd-python-client` compiler leverages a unified architecture to support various facets of API and code lifecycle management.
 
 * **Compilation**:
-    * **OpenAPI ➡️ `Python`**: Generate idiomatic native models, network routes, client SDKs, database schemas, and
-      boilerplate directly from OpenAPI (`.json` / `.yaml`) specifications.
-    * **`Python` ➡️ OpenAPI**: Statically parse existing `Python` source code and emit compliant OpenAPI specifications.
-* **AST-Driven & Safe**: Employs static analysis (Abstract Syntax Trees) instead of unsafe dynamic execution or
-  reflection, allowing it to safely parse and emit code even for incomplete or un-compilable project states.
-* **Seamless Sync**: Keep your docs, tests, database, clients, and routing in perfect harmony. Update your code, and
-  generate the docs; or update the docs, and generate the code.
+  * **OpenAPI → `Python`**: Generate idiomatic native models, network routes, client SDKs, database schemas, and boilerplate directly from OpenAPI (`.json` / `.yaml`) specifications.
+  * **`Python` → OpenAPI**: Statically parse existing `Python` source code and emit compliant OpenAPI specifications.
+* **AST-Driven & Safe**: Employs static analysis (Abstract Syntax Trees) instead of unsafe dynamic execution or reflection, allowing it to safely parse and emit code even for incomplete or un-compilable project states.
+* **Seamless Sync**: Keep your docs, tests, database, clients, and routing in perfect harmony. Update your code, and generate the docs; or update the docs, and generate the code.
 
 ## 📦 Installation
 
-Requires Python 3.9+. Install directly via `pip` or use `uv` for modern dependency management:
+Requires Python 3.9+.
 
 ```bash
 pip install openapi-python-client
@@ -39,50 +41,49 @@ pip install openapi-python-client
 
 ### Command Line Interface
 
-The `cdd` command provides a straightforward way to keep your Python artifacts and OpenAPI specs synchronized.
-
+Generate a Python client from an OpenAPI specification:
 ```bash
-# Generate Python client, mock server, and tests from an OpenAPI spec
-cdd sync --from-openapi openapi.json --to-python ./my_client_dir
+cdd sync --from-openapi spec.json --to-python ./my_client
+```
 
-# Extract an OpenAPI spec directly from an existing Python client
-cdd sync --from-python ./my_client_dir/client.py --to-openapi extracted_openapi.json
+Extract an OpenAPI specification from an existing Python client module:
+```bash
+cdd sync --from-python ./my_client/client.py --to-openapi spec.json
+```
 
-# Synchronize an entire directory (merging changes between Python files and openapi.json)
-cdd sync --dir ./my_project
+Generate a docs JSON:
+```bash
+cdd to_docs_json -i spec.json --no-imports
 ```
 
 ### Programmatic SDK / Library
 
-You can also leverage the underlying parsers and emitters programmatically within your Python scripts:
-
 ```python
-from pathlib import Path
 from openapi_client.openapi.parse import parse_openapi_json
 from openapi_client.routes.emit import ClientGenerator
 
-# 1. Parse an existing OpenAPI JSON spec
-spec_json = Path("openapi.json").read_text(encoding="utf-8")
-spec = parse_openapi_json(spec_json)
-
-# 2. Emit idiomatic Python client code
+spec = parse_openapi_json(open('spec.json').read())
 generator = ClientGenerator(spec)
-client_code = generator.generate_code()
-
-Path("client.py").write_text(client_code, encoding="utf-8")
+print(generator.generate_code())
 ```
 
+## Design choices
+
+The `cdd-python-client` leverages LibCST for parsing and generating Python Abstract Syntax Trees (AST). LibCST is chosen over Python's built-in `ast` module because it preserves whitespace, comments, and structure, which is crucial for a bidirectional compiler that doesn't mess up your code formatting when syncing changes. Pydantic is used for models generation to provide idiomatic and robust validation for the client interfaces.
+
 ## 🏗 Supported Conversions for Python
 
 *(The boxes below reflect the features supported by this specific `cdd-python-client` implementation)*
 
-| Concept                            | Parse (From) | Emit (To) |
-|------------------------------------|--------------|-----------|
-| OpenAPI (JSON/YAML)                | [x]          | [x]       |
-| `Python` Models / Structs / Types  | [x]          | [x]       |
-| `Python` Server Routes / Endpoints | [x]          | [x]       |
-| `Python` API Clients / SDKs        | [x]          | [x]       |
-| `Python` Docstrings / Comments     | [x]          | [x]       |
+| Concept | Parse (From) | Emit (To) |
+|---------|--------------|-----------|
+| OpenAPI (JSON/YAML) | [✅] | [✅] |
+| `Python` Models / Structs / Types | [✅] | [✅] |
+| `Python` Server Routes / Endpoints | [✅] | [✅] |
+| `Python` API Clients / SDKs | [✅] | [✅] |
+| `Python` ORM / DB Schemas | [ ] | [ ] |
+| `Python` CLI Argument Parsers | [ ] | [ ] |
+| `Python` Docstrings / Comments | [✅] | [✅] |
 
 ---
 
@@ -99,4 +100,4 @@ at your option.
 
 Unless you explicitly state otherwise, any contribution intentionally submitted
 for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
-dual licensed as above, without any additional terms or conditions.
+dual licensed as above, without any additional terms or conditions.

src/openapi_client/classes/__init__.py

@@ -0,0 +1 @@
+"""Module containing initialization logic."""

src/openapi_client/classes/parse.py

@@ -12,6 +12,7 @@ class ClassExtractor(cst.CSTVisitor):
     """
 
     def __init__(self, spec: OpenAPI):
+        """Initialize ClassExtractor with an OpenAPI spec."""
         self.spec = spec
         if self.spec.components is None:
             self.spec.components = Components(schemas={})

src/openapi_client/cli.py

@@ -2,6 +2,7 @@
 
 import argparse
 import sys
+import json
 from pathlib import Path
 import libcst as cst
 
@@ -17,6 +18,55 @@
 from openapi_client.openapi.emit import emit_openapi_json
 
 
+def generate_docs_json(input_path: str, no_imports: bool, no_wrapping: bool) -> None:
+    """Parse OpenAPI spec and output JSON documentation."""
+    spec_path = Path(input_path)
+    spec = parse_openapi_json(spec_path.read_text(encoding="utf-8"))
+
+    operations = []
+
+    if spec.paths:
+        for path, path_item in spec.paths.items():
+            for method in ["get", "post", "put", "delete", "patch"]:
+                operation = getattr(path_item, method, None)
+                if operation:
+                    op_id = (
+                        operation.operationId
+                        or f"{method}_{path.replace('/', '_').strip('_')}"
+                    )
+
+                    code = {}
+                    if not no_imports:
+                        code["imports"] = "from client import Client"
+                    if not no_wrapping:
+                        code["wrapper_start"] = (
+                            'def main():\n    client = Client(base_url="https://api.example.com")'
+                        )
+                        code["wrapper_end"] = 'if __name__ == "__main__":\n    main()'
+
+                    # Basic snippet with arguments
+                    args = []
+                    if operation.parameters:
+                        for param in operation.parameters:
+                            if hasattr(param, "name"):
+                                p_name = param.name.replace("-", "_")
+                                args.append(f"{p_name}={p_name}")
+
+                    args_str = ", ".join(args)
+                    indent = "    " if not no_wrapping else ""
+                    code["snippet"] = f"{indent}response = client.{op_id}({args_str})"
+
+                    op_data = {"method": method.upper(), "path": path, "code": code}
+                    if operation.operationId:
+                        op_data["operationId"] = operation.operationId
+
+                    operations.append(op_data)
+
+    output = [{"language": "python", "operations": operations}]
+
+    print(json.dumps(output, indent=2))
+
+
 def sync_to_python(openapi_path: str, output_dir: str) -> None:
     """Generate Python client, tests, and mocks from an OpenAPI spec."""
     spec_path = Path(openapi_path)
@@ -140,6 +190,23 @@ def main() -> None:
         help="Path to directory containing client.py, mock_server.py, test_client.py",
     )
 
+    docs_parser = subparsers.add_parser(
+        "to_docs_json", help="Generate JSON documentation"
+    )
+    docs_parser.add_argument(
+        "-i",
+        "--input",
+        type=str,
+        required=True,
+        help="Path or URL to the OpenAPI specification",
+    )
+    docs_parser.add_argument(
+        "--no-imports", action="store_true", help="Omit the imports field"
+    )
+    docs_parser.add_argument(
+        "--no-wrapping", action="store_true", help="Omit the wrapper fields"
+    )
+
     args = parser.parse_args()
 
     if args.command == "sync":
@@ -154,6 +221,8 @@ def main() -> None:
                 "Invalid sync arguments. Use either --dir, --from-openapi & --to-python OR --from-python & --to-openapi."
             )
             sys.exit(1)
+    elif args.command == "to_docs_json":
+        generate_docs_json(args.input, args.no_imports, args.no_wrapping)
 
 
 if __name__ == "__main__":  # pragma: no cover

src/openapi_client/docstrings/__init__.py

@@ -0,0 +1 @@
+"""Module containing initialization logic."""

src/openapi_client/functions/__init__.py

@@ -0,0 +1 @@
+"""Module containing initialization logic."""

src/openapi_client/functions/parse.py

@@ -12,6 +12,7 @@ class FunctionExtractor(cst.CSTVisitor):
     """
 
     def __init__(self, spec: OpenAPI):
+        """Initialize FunctionExtractor with an OpenAPI spec."""
         self.spec = spec
         if not self.spec.paths:
             self.spec.paths = {}

src/openapi_client/mocks/__init__.py

@@ -0,0 +1 @@
+"""Module containing initialization logic."""

src/openapi_client/mocks/parse.py

@@ -12,6 +12,7 @@ class MockServerExtractor(cst.CSTVisitor):
     """
 
     def __init__(self, spec: OpenAPI):
+        """Initialize MockServerExtractor with an OpenAPI spec."""
         self.spec = spec
         if not self.spec.paths:
             self.spec.paths = {}
@@ -62,6 +63,7 @@ class ReturnExtractor(cst.CSTVisitor):
                                         """Visitor to find event stream responses."""
 
                                         def __init__(self):
+                                            """Initialize ReturnExtractor."""
                                             self.has_event_stream = False
 
                                         def visit_Return(