Skip to content

Swarming: Patches Auth default credentials at startup#5245

Draft
IvanBM18 wants to merge 2 commits intomasterfrom
fix/swarming/patches_auth_at_startup
Draft

Swarming: Patches Auth default credentials at startup#5245
IvanBM18 wants to merge 2 commits intomasterfrom
fix/swarming/patches_auth_at_startup

Conversation

@IvanBM18
Copy link
Copy Markdown
Collaborator

Fuzzing tasks ins warming have an intermittent error in which startup of any gcp lirbary or module fails with:

  File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/ndb_init.py", line 39, in _client
    _ndb_client = ndb.Client(project=utils.get_application_id())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 145, in __init__
    super(Client, self).__init__(
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/client/__init__.py", line 320, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/client/__init__.py", line 268, in __init__
    project = self._determine_default(project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 249, in _determine_default
    return _determine_default_project(project)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 70, in _determine_default_project
    project = _helpers._determine_default_project(project=project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/_helpers/__init__.py", line 152, in _determine_default_project
    _, project = google.auth.default()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/auth/_default.py", line 692, in default
    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

This happens because the timeout for calling the default credentials times out because credentials coming from luci-auth doesn't always comes fast enough for the internal gcp libraries.

To fix this, we are manually patching the google.auth.default module to return some credentials that do come from the swarming bot.

Note: This is a draft since dev testing is still in progress, will open the PR when dev testing is finished.


def patched_default_credentials(
*args, **kwargs) -> tuple[compute_engine.Credentials, str] | None:
# pylint: disable=unused-argument
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: We have the args and kwargs in the method signature because otherwise if we call the default() method with any args an exception would be thrown because the patch didn't matched the expected signature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant