Merge branch 'main' into feature/keylist-webhooks

Author: dbluhm

CHANGELOG.md

@@ -18,6 +18,7 @@ See the [README](README.md) for details about this repository and information ab
 - [Developing](#developing)
   - [Prerequisites](#prerequisites)
   - [Running Locally](#running-locally)
+  - [Logging](#logging)
   - [Running Tests](#running-tests)
   - [Running Aries Agent Test Harness Tests](#running-aries-agent-test-harness-tests)
 - [Development Workflow](#development-workflow)

DevReadMe.md

@@ -78,6 +78,19 @@ An ACA-Py instance puts together an OpenAPI-documented REST interface based on t
 
 Technical note: the administrative API exposed by the agent for the controller to use must be protected with an API key (using the --admin-api-key command line arg) or deliberately left unsecured using the --admin-insecure-mode command line arg. The latter should not be used other than in development if the API is not otherwise secured.
 
+## Troubleshooting
+
+There are a number of resources for getting help with ACA-Py and troubleshooting
+any problems you might run into. The [Troubleshooting](Troubleshooting.md)
+document contains some guidance about issues that have been experienced in the
+past. Feel free to submit PRs to supplement the troubleshooting document!
+Searching the [ACA-Py GitHub
+issues](https://github.com/hyperledger/aries-cloudagent-python/issues) will
+often uncover challenges that others have experienced, often with answers to
+solving those challenges. As well, there is the "aries-cloudagent-python"
+channel on the Hyperledger Discord chat server ([invitation
+here](https://discord.gg/hyperledger)).
+
 ## Credit
 
 The initial implementation of ACA-Py was developed by the Government of British Columbia’s Digital Trust Team in Canada. To learn more about what’s happening with decentralized identity and digital trust in British Columbia, a new website will be launching and the link will be made available here.

README.md

@@ -0,0 +1,109 @@
+# Troubleshooting Aries Cloud Agent Python <!-- omit in toc -->
+
+This document contains some troubleshooting information that contributors to the
+community think may be helpful. Most of the content here assumes the reader has
+gotten started with ACA-Py and has arrived here because of an issue that came up
+in their use of ACA-Py.
+
+Contributions (via pull request) to this document are welcome. Topics added here
+will mostly come from reported issues that contributors think would be helpful
+to the larger community.
+
+## Table of Contents <!-- omit in toc -->
+
+- [Unable to Connect to Ledger](#unable-to-connect-to-ledger)
+  - [Local ledger running?](#local-ledger-running)
+  - [Any Firewalls](#any-firewalls)
+- [Damaged, Unpublishable Revocation Registry](#damaged-unpublishable-revocation-registry)
+
+## Unable to Connect to Ledger
+
+The most common issue hit by first time users is getting an error on startup "unable to connect to ledger". Here are a list of things to check when you see that error.
+
+### Local ledger running?
+
+Unless you specify via startup parameters or environment variables that you are using a public Hyperledger Indy ledger, ACA-Py assumes that you are running a local ledger -- an instance of [von-network](https://github.com/bcgov/von-network).
+If that is the cause -- have you started your local ledger, and did it startup properly.  Things to check:
+
+- Any errors in the startup of von-network?
+- Is the von-network webserver (usually at `https:/localhost:9000`) accessible? If so, can you click on and see the Genesis File?
+- Do you even need a local ledger? If not, you can use a public sandbox ledger,
+  such as the [Dev Greenlight ledger](), likely by just prefacing your ACA-Py
+  command with `LEDGER_URL=http://dev.greenlight.bcovrin.vonx.io`. For example,
+  when running the Alice-Faber demo in the [demo](demo) folder, you can run (for
+  example), the Faber agent using the command:
+  `LEDGER_URL=http://dev.greenlight.bcovrin.vonx.io ./run_demo faber`
+
+### Any Firewalls
+
+Do you have any firewalls in play that might be blocking the ports that are used by the ledger, notably 9701-9708? To access a ledger
+the ACA-Py instance must be able to get to those ports of the ledger, regardless if the ledger is local or remote.
+
+## Damaged, Unpublishable Revocation Registry
+
+We have discovered that in the ACA-Py AnonCreds implementation, it is possible
+to get into a state where the publishing of updates to a Revocation Registry
+(RevReg) is impossible. This can happen where ACA-Py starts to publish an update
+to the RevReg, but the write transaction to the Hyperledger Indy ledger fails
+for some reason. When a credential revocation is published, aca-py (via indy-sdk
+or askar/credx) updates the revocation state in the wallet as well as on the
+ledger.  The revocation state is dependant on whatever the previous revocation
+state is/was, so if the ledger and wallet are mis-matched the publish will fail.
+(Andrew/s PR # 1804 (merged) should mitigate but probably won't completely
+eliminate this from happening).
+
+For example, in case we've seen, the write RevRegEntry transaction failed at the
+ledger because there was a problem with accepting the TAA (Transaction Author
+Agreement). Once the error occurred, the RevReg state held by the ACA-Py agent,
+and the RevReg state on the ledger were different. Even after the ability to
+write to the ledger was restored, the RevReg could still not be published
+because of the differences in the RevReg state. Such a situation can now be
+corrected, as follows:
+
+To address this issue, some new endpoints were added to ACA-Py in Release 0.7.4,
+as follows:
+
+- GET `/revocation/registry/<id>/issued` - counts of the number of issued/revoked
+  within a registry
+- GET `/revocation/registry/<id>/issued/details` - details of all credentials
+  issued/revoked within a registry
+- GET `/revocation/registry/<id>/issued/indy_recs` - calculated rev_reg_delta from
+  the ledger
+  - This is used to compare ledger revoked vs wallet revoked credentials, which
+    is essentially the state of the RevReg on the ledger and in ACA-Py. Where
+    there is a difference, we have an error.
+- PUT `/revocation/registry/<id>/fix-revocation-entry-state` - publish an update
+  to the RevReg state on the ledger to bring it into alignment with what is in
+  the ACA-Py instance.
+  - There is a boolean parameter (`apply_ledger_update`) to control whether the
+    ledger entry actually gets published so, if you are so inclined, you can
+    call the endpoint to see what the transaction would be, before you actually
+    try to do a ledger update.  This will return:
+    - `rev_reg_delta` - same as the ".../indy_recs" endpoint
+    - `accum_calculated` - transaction to write to ledger
+    - `accum_fixed` - If `apply_ledger_update`, the transaction actually written
+      to the ledger
+
+Note that there is (currently) a backlog item to prevent the wallet and ledger
+from getting out of sync (e.g. don't update the ACA-Py RevReg state if the
+ledger write fails), but even after that change is made, having this ability
+will be retained for use if needed.
+
+We originally ran into this due to the TAA acceptance getting lost when
+switching to multi-ledger (as described
+[here](https://github.com/hyperledger/aries-cloudagent-python/blob/main/Multiledger.md#a-special-warning-for-taa-acceptance).
+Note that this is one reason how this "out of sync" scenario can occur, but
+there may be others.
+
+We add an integration test that demonstrates/tests this issue [here](https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/features/taa-txn-author-acceptance.feature#L67).
+
+To run the scenario either manually or using the integration tests, you can do the following:
+
+- Start von-network in TAA mode:
+  - `./manage start --taa-sample --logs`
+- Start the tails server as usual:
+  - `./manage start --logs`
+- To run the scenario manually, start faber and let the agent know it needs to TAA-accept before doing any ledger writes:
+  - `./run_demo faber --revocation --taa-accept`, and then you can run through all the transactions using the Swagger page.
+- To run the scenario via an integration test, run:
+  - `./run_bdd -t @taa_required`

Troubleshooting.md

@@ -113,47 +113,53 @@ def print_banner(
 
         # Inbound transports
         banner.print_subtitle("Inbound Transports")
-        banner.print_spacer()
-        banner.print_list(
-            [
-                f"{transport.scheme}://{transport.host}:{transport.port}"
-                for transport in inbound_transports.values()
-            ]
-        )
-        banner.print_spacer()
-
-        external_in_transports = set().union(
-            *(
-                transport
-                for transport in inbound_transports.values()
-                if transport.is_external
-            )
-        )
+        internal_in_transports = [
+            f"{transport.scheme}://{transport.host}:{transport.port}"
+            for transport in inbound_transports.values()
+            if not transport.is_external
+        ]
+        if internal_in_transports:
+            banner.print_spacer()
+            banner.print_list(internal_in_transports)
+            banner.print_spacer()
+        external_in_transports = [
+            f"{transport.scheme}://{transport.host}:{transport.port}"
+            for transport in inbound_transports.values()
+            if transport.is_external
+        ]
         if external_in_transports:
             banner.print_spacer()
-            banner.print_list([f"{external_in_transports}"])
+            banner.print_subtitle("  External Plugin")
+            banner.print_spacer()
+            banner.print_list(external_in_transports)
             banner.print_spacer()
 
         # Outbound transports
-        schemes = set().union(
-            *(transport.schemes for transport in outbound_transports.values())
+        banner.print_subtitle("Outbound Transports")
+        internal_schemes = set().union(
+            *(
+                transport.schemes
+                for transport in outbound_transports.values()
+                if not transport.is_external
+            )
         )
-        if schemes:
-            banner.print_subtitle("Outbound Transports")
+        if internal_schemes:
             banner.print_spacer()
-            banner.print_list([f"{scheme}" for scheme in sorted(schemes)])
+            banner.print_list([f"{scheme}" for scheme in sorted(internal_schemes)])
             banner.print_spacer()
 
-        external_out_transports = set().union(
+        external_schemes = set().union(
             *(
-                transport
+                transport.schemes
                 for transport in outbound_transports.values()
                 if transport.is_external
             )
         )
-        if external_out_transports:
+        if external_schemes:
+            banner.print_spacer()
+            banner.print_subtitle("  External Plugin")
             banner.print_spacer()
-            banner.print_list([f"{external_out_transports}"])
+            banner.print_list([f"{scheme}" for scheme in sorted(external_schemes)])
             banner.print_spacer()
 
         # DID info

aries_cloudagent/config/logging.py

@@ -63,6 +63,7 @@
 from ..vc.ld_proofs.document_loader import DocumentLoader
 from ..version import __version__, RECORD_TYPE_ACAPY_VERSION
 from ..wallet.did_info import DIDInfo
+from .oob_processor import OobMessageProcessor
 
 from .dispatcher import Dispatcher
 from .util import STARTUP_EVENT_TOPIC, SHUTDOWN_EVENT_TOPIC
@@ -203,6 +204,13 @@ async def setup(self):
                 BaseMultitenantManager, MultitenantManagerProvider(self.root_profile)
             )
 
+        # Bind oob message processor to be able to receive and process un-encrypted
+        # messages
+        context.injector.bind_instance(
+            OobMessageProcessor,
+            OobMessageProcessor(inbound_message_router=self.inbound_message_router),
+        )
+
         # Bind default PyLD document loader
         context.injector.bind_instance(
             DocumentLoader, DocumentLoader(self.root_profile)
@@ -636,8 +644,10 @@ async def queue_outbound(
             message: An outbound message to be sent
             inbound: The inbound message that produced this response, if available
         """
+        has_target = outbound.target or outbound.target_list
+
         # populate connection target(s)
-        if not outbound.target and not outbound.target_list and outbound.connection_id:
+        if not has_target and outbound.connection_id:
             conn_mgr = ConnectionManager(profile)
             try:
                 outbound.target_list = await self.dispatcher.run_task(
@@ -654,6 +664,15 @@ async def queue_outbound(
                     self.admin_server.notify_fatal_error()
                 raise
             del conn_mgr
+        # Find oob/connectionless target we can send the message to
+        elif not has_target and outbound.reply_thread_id:
+            message_processor = profile.inject(OobMessageProcessor)
+            outbound.target = await self.dispatcher.run_task(
+                message_processor.find_oob_target_for_outbound_message(
+                    profile, outbound
+                )
+            )
+
         return await self._queue_message(profile, outbound)
 
     async def _queue_message(

aries_cloudagent/core/conductor.py

@@ -14,6 +14,8 @@
 
 from aiohttp.web import HTTPException
 
+
+from ..connections.models.conn_record import ConnRecord
 from ..core.profile import Profile
 from ..messaging.agent_message import AgentMessage
 from ..messaging.base_message import BaseMessage
@@ -173,11 +175,20 @@ async def handle_message(
 
         context.injector.bind_instance(BaseResponder, responder)
 
-        connection_mgr = ConnectionManager(profile)
-        connection = await connection_mgr.find_inbound_connection(
-            inbound_message.receipt
-        )
-        del connection_mgr
+        # When processing oob attach message we supply the connection id
+        # associated with the inbound message
+        if inbound_message.connection_id:
+            async with self.profile.session() as session:
+                connection = await ConnRecord.retrieve_by_id(
+                    session, inbound_message.connection_id
+                )
+        else:
+            connection_mgr = ConnectionManager(profile)
+            connection = await connection_mgr.find_inbound_connection(
+                inbound_message.receipt
+            )
+            del connection_mgr
+
         if connection:
             inbound_message.connection_id = connection.connection_id