Skip to content

Build: [AEA-0000] - Add retry logic to the deploy-api stage #2986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wildjames wants to merge 1 commit into from
Closed

Build: [AEA-0000] - Add retry logic to the deploy-api stage #2986

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 33 additions & 29 deletions .github/scripts/deploy_api.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,33 @@
#!/usr/bin/env bash
set -eu pipefail

invoke_lambda_with_retry() {
local function_name="$1"
local payload_file="$2"
local max_attempts=3
local delay=30

for attempt in $(seq 1 "$max_attempts"); do
echo "Invoking ${function_name} (attempt ${attempt} of ${max_attempts})"
aws lambda invoke --function-name "${function_name}" --cli-binary-format raw-in-base64-out --payload "file://${payload_file}" out.txt > response.json

if ! eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
return 0
Comment thread
wildjames marked this conversation as resolved.
fi

echo "Error calling lambda (attempt ${attempt} of ${max_attempts})"
cat out.txt
Comment on lines +9 to +19
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

out.txt and response.json are overwritten on every attempt and on every lambda call, which makes post-failure debugging harder (especially with retries). Consider incorporating the function name/attempt into the filenames, or at least printing both the lambda payload output and the invoke metadata for each failed attempt.

Suggested change
for attempt in $(seq 1 "$max_attempts"); do
echo "Invoking ${function_name} (attempt ${attempt} of ${max_attempts})"
aws lambda invoke --function-name "${function_name}" --cli-binary-format raw-in-base64-out --payload "file://${payload_file}" out.txt > response.json
if ! eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
return 0
fi
echo "Error calling lambda (attempt ${attempt} of ${max_attempts})"
cat out.txt
local safe_function_name
safe_function_name=$(printf '%s' "${function_name}" | tr -c '[:alnum:]._-' '_')
for attempt in $(seq 1 "$max_attempts"); do
local payload_output_file="${safe_function_name}.attempt_${attempt}.out.txt"
local invoke_metadata_file="${safe_function_name}.attempt_${attempt}.response.json"
echo "Invoking ${function_name} (attempt ${attempt} of ${max_attempts})"
aws lambda invoke --function-name "${function_name}" --cli-binary-format raw-in-base64-out --payload "file://${payload_file}" "${payload_output_file}" > "${invoke_metadata_file}"
if ! jq -e '.FunctionError' "${invoke_metadata_file}" >/dev/null; then
return 0
fi
echo "Error calling lambda (attempt ${attempt} of ${max_attempts})"
echo "Lambda payload output (${payload_output_file}):"
cat "${payload_output_file}"
echo "Lambda invoke metadata (${invoke_metadata_file}):"
cat "${invoke_metadata_file}"

Copilot uses AI. Check for mistakes.

Comment thread
wildjames marked this conversation as resolved.
if [ "$attempt" -lt "$max_attempts" ]; then
echo "Retrying in ${delay} seconds..."
sleep "$delay"
fi
done

echo "Lambda ${function_name} failed after ${max_attempts} attempts"
return 1
}

echo "API type: ${API_TYPE}"
echo "Specification path: ${SPEC_PATH}"
echo "Specification version: ${VERSION_NUMBER}"
Expand Down Expand Up @@ -31,7 +58,7 @@ put_secret_lambda=lambda-resources-ProxygenPTLMTLSSecretPut
instance_put_lambda=lambda-resources-ProxygenPTLInstancePut
spec_publish_lambda=lambda-resources-ProxygenPTLSpecPublish

if [[ "$APIGEE_ENVIRONMENT" =~ ^(int|sandbox|prod)$ ]]; then
if [[ "$APIGEE_ENVIRONMENT" =~ ^(int|sandbox|prod)$ ]]; then
put_secret_lambda=lambda-resources-ProxygenProdMTLSSecretPut
instance_put_lambda=lambda-resources-ProxygenProdInstancePut
spec_publish_lambda=lambda-resources-ProxygenProdSpecPublish
Expand Down Expand Up @@ -71,7 +98,7 @@ if [[ "${IS_PULL_REQUEST}" == "true" ]]; then
jq '."x-nhsd-apim".temporary = true' "${SPEC_PATH}" > temp.json && mv temp.json "${SPEC_PATH}"
fi

# Find and replace the specification version number
# Find and replace the specification version number
jq --arg version "${VERSION_NUMBER}" '.info.version = $version' "${SPEC_PATH}" > temp.json && mv temp.json "${SPEC_PATH}"

# Find and replace the x-nhsd-apim.target.url value
Expand Down Expand Up @@ -138,12 +165,7 @@ if [[ "${ENABLE_MUTUAL_TLS}" == "true" ]]; then
--arg proxygenSecretName "${proxygen_private_key_arn}" \
'{apiName: $apiName, environment: $environment, secretName: $secretName, secretKey: $secretKey, secretCert: $secretCert, kid, $kid, proxygenSecretName: $proxygenSecretName}' > payload.json
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The jq payload object contains kid, $kid, which is not valid key/value construction and will either set kid to null or cause a jq syntax error. This should be a proper field assignment (e.g. kid: $kid) so the lambda receives the expected KID value.

Suggested change
'{apiName: $apiName, environment: $environment, secretName: $secretName, secretKey: $secretKey, secretCert: $secretCert, kid, $kid, proxygenSecretName: $proxygenSecretName}' > payload.json
'{apiName: $apiName, environment: $environment, secretName: $secretName, secretKey: $secretKey, secretCert: $secretCert, kid: $kid, proxygenSecretName: $proxygenSecretName}' > payload.json

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

@wildjames wildjames Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

god damn, that might be the problem. Why does this work most of the time, then?


aws lambda invoke --function-name "${put_secret_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json
if eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
echo 'Error calling lambda'
cat out.txt
exit 1
fi
invoke_lambda_with_retry "${put_secret_lambda}" payload.json
echo "Secret stored successfully"

else
Expand All @@ -163,13 +185,7 @@ if [[ "${DRY_RUN}" == "false" ]]; then
--arg proxygenSecretName "${proxygen_private_key_arn}" \
'{apiName: $apiName, environment: $environment, specDefinition: $spec, instance: $instance, kid: $kid, proxygenSecretName: $proxygenSecretName}' > payload.json

aws lambda invoke --function-name "${instance_put_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json

if eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
echo 'Error calling lambda'
cat out.txt
exit 1
fi
invoke_lambda_with_retry "${instance_put_lambda}" payload.json
echo "Instance deployed"
else
echo "Would call ${instance_put_lambda}"
Expand All @@ -187,13 +203,7 @@ if [[ "${APIGEE_ENVIRONMENT}" == "int" ]]; then
--arg proxygenSecretName "${proxygen_private_key_arn}" \
'{apiName: $apiName, environment: $environment, specDefinition: $spec, instance: $instance, kid: $kid, proxygenSecretName: $proxygenSecretName}' > payload.json

aws lambda invoke --function-name "${spec_publish_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json

if eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
echo 'Error calling lambda'
cat out.txt
exit 1
fi
invoke_lambda_with_retry "${spec_publish_lambda}" payload.json
echo "Spec deployed"
else
echo "Would call ${spec_publish_lambda}"
Expand All @@ -212,13 +222,7 @@ if [[ "${APIGEE_ENVIRONMENT}" == "internal-dev" && "${IS_PULL_REQUEST}" == "fals
--arg proxygenSecretName "${proxygen_private_key_arn}" \
'{apiName: $apiName, environment: $environment, specDefinition: $spec, instance: $instance, kid: $kid, proxygenSecretName: $proxygenSecretName}' > payload.json

aws lambda invoke --function-name "${spec_publish_lambda}" --cli-binary-format raw-in-base64-out --payload file://payload.json out.txt > response.json

if eval "cat response.json | jq -e '.FunctionError' >/dev/null"; then
echo 'Error calling lambda'
cat out.txt
exit 1
fi
invoke_lambda_with_retry "${spec_publish_lambda}" payload.json
echo "Spec deployed"
else
echo "Would call ${spec_publish_lambda}"
Expand Down