Skip to content

fix: truncate long reference_id to avoid DB DataError #2248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Sanjay-VK07 wants to merge 3 commits into
base: main
Choose a base branch
from
+33 −1
Open

fix: truncate long reference_id to avoid DB DataError #2248

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a9c1396
fix: truncate long reference_id to avoid DB DataError
Sanjay-VK07 Apr 2, 2026
2408be8
fix: append ellipsis when truncating reference_id
Sanjay-VK07 Apr 17, 2026
0d3b984
test: add coverage for truncated reference_id
Sanjay-VK07 Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion vulnerabilities/pipelines/enhance_with_exploitdb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,13 +128,16 @@ def add_exploit_references(ref_id, direct_url, path, vul_id, logger):
"direct_url": direct_url,
}

MAX_REF_LEN = 200
safe_ref_id = ref_id[:MAX_REF_LEN] if ref_id else ref_id
Copy link
Copy Markdown
Collaborator

@ziadhany ziadhany Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add "..." at the end of the truncated text to indicate that it has been shortened?

Copy link
Copy Markdown
Author

@Sanjay-VK07 Sanjay-VK07 Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion!

I’ve updated the implementation to append "..." when truncating the reference_id to clearly indicate that it has been shortened.

I also ran the pipeline locally with proper Django setup. The tests are executing correctly.

Test summary:
39 failed, 885 passed, 1 skipped

The remaining failures are related to SQLite limitations as the project expects PostgreSQL.

Please let me know if any further changes are needed.

Copy link
Copy Markdown
Collaborator

@ziadhany ziadhany Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sanjay-VK07 All tests should run successfully before we merge. You can paste the error logs, I might be able to help or do a quick search on the error text on google.

Copy link
Copy Markdown
Author

@Sanjay-VK07 Sanjay-VK07 Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion!

I ran the pipeline locally and here are some sample error logs:

django.db.utils.NotSupportedError: DISTINCT ON fields is not supported by this database backend
django.db.utils.NotSupportedError: SQLite schema editor cannot be used while foreign key constraint checks are enabled
django.db.utils.NotSupportedError: contains lookup is not supported on this database backend

Summary:
39 failed, 885 passed, 1 skipped
These failures seem to be related to SQLite limitations (e.g., DISTINCT ON, schema editor constraints, unsupported lookups), while the project appears to expect PostgreSQL for full test compatibility.

Please let me know if you would like me to run this specifically with PostgreSQL as well.

Copy link
Copy Markdown
Collaborator

@ziadhany ziadhany Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you using SQLite? I think it would be better to switch to PostgreSQL to avoid this error

See:
https://vulnerablecode.readthedocs.io/en/latest/installation.html#database

Copy link
Copy Markdown
Author

@Sanjay-VK07 Sanjay-VK07 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reran the tests in the Docker/Codespaces PostgreSQL (postgres:15) environment as suggested. The SQLite-specific backend errors are gone. The remaining failures are still present and appear unrelated to the database backend.


for key, url in url_map.items():
if url:
try:
ref, created = VulnerabilityReference.objects.update_or_create(
url=url,
defaults={
"reference_id": ref_id,
"reference_id": safe_ref_id,
"reference_type": VulnerabilityReference.EXPLOIT,
Comment on lines +143 to 144
Copy link
Copy Markdown
Collaborator

@ziadhany ziadhany Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test for this change.

Copy link
Copy Markdown
Author

@Sanjay-VK07 Sanjay-VK07 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback. I’ve already added the truncation with ellipsis update, and I’m now adding a dedicated test case for this behavior in the ExploitDB pipeline tests. I’ll push the update shortly.

},
)
Expand Down