Skip to content

Commit 72af9e1

Browse files
api-clients-generation-pipeline[bot]ci.datadog-api-spec
api-clients-generation-pipeline[bot]
and
ci.datadog-api-spec
authored
Add Annotations endpoints (#3203)
Co-authored-by: ci.datadog-api-spec <packages@datadoghq.com>
1 parent 9206cb5 commit 72af9e1

44 files changed

Lines changed: 5663 additions & 3 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.generator/schemas/v2/openapi.yaml‎

Lines changed: 786 additions & 1 deletion
Large diffs are not rendered by default.

‎examples/v2/llm-observability/CreateLLMObsAnnotationQueue.rb‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Create an LLM Observability annotation queue returns "Created" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.create_llm_obs_annotation_queue".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
9+
body = DatadogAPIClient::V2::LLMObsAnnotationQueueRequest.new({
10+
data: DatadogAPIClient::V2::LLMObsAnnotationQueueDataRequest.new({
11+
attributes: DatadogAPIClient::V2::LLMObsAnnotationQueueDataAttributesRequest.new({
12+
description: "Queue for annotating customer support traces",
13+
name: "My annotation queue",
14+
project_id: "a33671aa-24fd-4dcd-9b33-a8ec7dde7751",
15+
}),
16+
type: DatadogAPIClient::V2::LLMObsAnnotationQueueType::QUEUES,
17+
}),
18+
})
19+
p api_instance.create_llm_obs_annotation_queue(body)

‎examples/v2/llm-observability/CreateLLMObsAnnotationQueueInteractions.rb‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Add annotation queue interactions returns "Created" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.create_llm_obs_annotation_queue_interactions".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
9+
body = DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionsRequest.new({
10+
data: DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionsDataRequest.new({
11+
attributes: DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionsDataAttributesRequest.new({
12+
interactions: [
13+
DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionItem.new({
14+
content_id: "trace-abc-123",
15+
type: DatadogAPIClient::V2::LLMObsInteractionType::TRACE,
16+
}),
17+
],
18+
}),
19+
type: DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionsType::INTERACTIONS,
20+
}),
21+
})
22+
p api_instance.create_llm_obs_annotation_queue_interactions("queue_id", body)

‎examples/v2/llm-observability/DeleteLLMObsAnnotationQueue.rb‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Delete an LLM Observability annotation queue returns "No Content" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.delete_llm_obs_annotation_queue".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
api_instance.delete_llm_obs_annotation_queue("queue_id")

‎examples/v2/llm-observability/DeleteLLMObsAnnotationQueueInteractions.rb‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Delete annotation queue interactions returns "No Content" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.delete_llm_obs_annotation_queue_interactions".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
9+
body = DatadogAPIClient::V2::LLMObsDeleteAnnotationQueueInteractionsRequest.new({
10+
data: DatadogAPIClient::V2::LLMObsDeleteAnnotationQueueInteractionsDataRequest.new({
11+
attributes: DatadogAPIClient::V2::LLMObsDeleteAnnotationQueueInteractionsDataAttributesRequest.new({
12+
interaction_ids: [
13+
"00000000-0000-0000-0000-000000000000",
14+
"00000000-0000-0000-0000-000000000001",
15+
],
16+
}),
17+
type: DatadogAPIClient::V2::LLMObsAnnotationQueueInteractionsType::INTERACTIONS,
18+
}),
19+
})
20+
api_instance.delete_llm_obs_annotation_queue_interactions("queue_id", body)

‎examples/v2/llm-observability/GetLLMObsAnnotatedInteractions.rb‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Get annotated queue interactions returns "OK" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.get_llm_obs_annotated_interactions".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
p api_instance.get_llm_obs_annotated_interactions("queue_id")

‎examples/v2/llm-observability/ListLLMObsAnnotationQueues.rb‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# List LLM Observability annotation queues returns "OK" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.list_llm_obs_annotation_queues".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
p api_instance.list_llm_obs_annotation_queues()

‎examples/v2/llm-observability/UpdateLLMObsAnnotationQueue.rb‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Update an LLM Observability annotation queue returns "OK" response
2+
3+
require "datadog_api_client"
4+
DatadogAPIClient.configure do |config|
5+
config.unstable_operations["v2.update_llm_obs_annotation_queue".to_sym] = true
6+
end
7+
api_instance = DatadogAPIClient::V2::LLMObservabilityAPI.new
8+
9+
body = DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateRequest.new({
10+
data: DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateDataRequest.new({
11+
attributes: DatadogAPIClient::V2::LLMObsAnnotationQueueUpdateDataAttributesRequest.new({
12+
description: "Updated description",
13+
name: "Updated queue name",
14+
}),
15+
type: DatadogAPIClient::V2::LLMObsAnnotationQueueType::QUEUES,
16+
}),
17+
})
18+
p api_instance.update_llm_obs_annotation_queue("queue_id", body)

‎features/scenarios_model_mapping.rb‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2751,6 +2751,31 @@
27512751
"v2.UpdateIPAllowlist" => {
27522752
"body" => "IPAllowlistUpdateRequest",
27532753
},
2754+
"v2.ListLLMObsAnnotationQueues" => {
2755+
"project_id" => "String",
2756+
"queue_ids" => "Array<String>",
2757+
},
2758+
"v2.CreateLLMObsAnnotationQueue" => {
2759+
"body" => "LLMObsAnnotationQueueRequest",
2760+
},
2761+
"v2.DeleteLLMObsAnnotationQueue" => {
2762+
"queue_id" => "String",
2763+
},
2764+
"v2.UpdateLLMObsAnnotationQueue" => {
2765+
"queue_id" => "String",
2766+
"body" => "LLMObsAnnotationQueueUpdateRequest",
2767+
},
2768+
"v2.GetLLMObsAnnotatedInteractions" => {
2769+
"queue_id" => "String",
2770+
},
2771+
"v2.CreateLLMObsAnnotationQueueInteractions" => {
2772+
"queue_id" => "String",
2773+
"body" => "LLMObsAnnotationQueueInteractionsRequest",
2774+
},
2775+
"v2.DeleteLLMObsAnnotationQueueInteractions" => {
2776+
"queue_id" => "String",
2777+
"body" => "LLMObsDeleteAnnotationQueueInteractionsRequest",
2778+
},
27542779
"v2.ListLLMObsExperiments" => {
27552780
"filter_project_id" => "String",
27562781
"filter_dataset_id" => "String",

‎features/v2/llm_observability.feature‎

Lines changed: 153 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,40 @@
11
@endpoint(llm-observability) @endpoint(llm-observability-v2)
22
Feature: LLM Observability
3-
Manage LLM Observability projects, datasets, dataset records, and
4-
experiments via the Experiments API.
3+
Manage LLM Observability projects, datasets, dataset records, experiments,
4+
and annotations.
55

66
Background:
77
Given a valid "apiKeyAuth" key in the system
88
And a valid "appKeyAuth" key in the system
99
And an instance of "LLMObservability" API
1010

11+
@generated @skip @team:DataDog/ml-observability
12+
Scenario: Add annotation queue interactions returns "Bad Request" response
13+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
14+
And new "CreateLLMObsAnnotationQueueInteractions" request
15+
And request contains "queue_id" parameter from "REPLACE.ME"
16+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
17+
When the request is sent
18+
Then the response status is 400 Bad Request
19+
20+
@generated @skip @team:DataDog/ml-observability
21+
Scenario: Add annotation queue interactions returns "Created" response
22+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
23+
And new "CreateLLMObsAnnotationQueueInteractions" request
24+
And request contains "queue_id" parameter from "REPLACE.ME"
25+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
26+
When the request is sent
27+
Then the response status is 201 Created
28+
29+
@generated @skip @team:DataDog/ml-observability
30+
Scenario: Add annotation queue interactions returns "Not Found" response
31+
Given operation "CreateLLMObsAnnotationQueueInteractions" enabled
32+
And new "CreateLLMObsAnnotationQueueInteractions" request
33+
And request contains "queue_id" parameter from "REPLACE.ME"
34+
And body with value {"data": {"attributes": {"interactions": [{"content_id": "trace-abc-123", "type": "trace"}]}, "type": "interactions"}}
35+
When the request is sent
36+
Then the response status is 404 Not Found
37+
1138
@generated @skip @team:DataDog/ml-observability
1239
Scenario: Append records to an LLM Observability dataset returns "Bad Request" response
1340
Given operation "CreateLLMObsDatasetRecords" enabled
@@ -48,6 +75,22 @@ Feature: LLM Observability
4875
When the request is sent
4976
Then the response status is 200 OK
5077

78+
@generated @skip @team:DataDog/ml-observability
79+
Scenario: Create an LLM Observability annotation queue returns "Bad Request" response
80+
Given operation "CreateLLMObsAnnotationQueue" enabled
81+
And new "CreateLLMObsAnnotationQueue" request
82+
And body with value {"data": {"attributes": {"description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751"}, "type": "queues"}}
83+
When the request is sent
84+
Then the response status is 400 Bad Request
85+
86+
@generated @skip @team:DataDog/ml-observability
87+
Scenario: Create an LLM Observability annotation queue returns "Created" response
88+
Given operation "CreateLLMObsAnnotationQueue" enabled
89+
And new "CreateLLMObsAnnotationQueue" request
90+
And body with value {"data": {"attributes": {"description": "Queue for annotating customer support traces", "name": "My annotation queue", "project_id": "a33671aa-24fd-4dcd-9b33-a8ec7dde7751"}, "type": "queues"}}
91+
When the request is sent
92+
Then the response status is 201 Created
93+
5194
@generated @skip @team:DataDog/ml-observability
5295
Scenario: Create an LLM Observability dataset returns "Bad Request" response
5396
Given operation "CreateLLMObsDataset" enabled
@@ -221,6 +264,87 @@ Feature: LLM Observability
221264
When the request is sent
222265
Then the response status is 204 No Content
223266

267+
@generated @skip @team:DataDog/ml-observability
268+
Scenario: Delete an LLM Observability annotation queue returns "No Content" response
269+
Given operation "DeleteLLMObsAnnotationQueue" enabled
270+
And new "DeleteLLMObsAnnotationQueue" request
271+
And request contains "queue_id" parameter from "REPLACE.ME"
272+
When the request is sent
273+
Then the response status is 204 No Content
274+
275+
@generated @skip @team:DataDog/ml-observability
276+
Scenario: Delete an LLM Observability annotation queue returns "Not Found" response
277+
Given operation "DeleteLLMObsAnnotationQueue" enabled
278+
And new "DeleteLLMObsAnnotationQueue" request
279+
And request contains "queue_id" parameter from "REPLACE.ME"
280+
When the request is sent
281+
Then the response status is 404 Not Found
282+
283+
@generated @skip @team:DataDog/ml-observability
284+
Scenario: Delete annotation queue interactions returns "Bad Request" response
285+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
286+
And new "DeleteLLMObsAnnotationQueueInteractions" request
287+
And request contains "queue_id" parameter from "REPLACE.ME"
288+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
289+
When the request is sent
290+
Then the response status is 400 Bad Request
291+
292+
@generated @skip @team:DataDog/ml-observability
293+
Scenario: Delete annotation queue interactions returns "No Content" response
294+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
295+
And new "DeleteLLMObsAnnotationQueueInteractions" request
296+
And request contains "queue_id" parameter from "REPLACE.ME"
297+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
298+
When the request is sent
299+
Then the response status is 204 No Content
300+
301+
@generated @skip @team:DataDog/ml-observability
302+
Scenario: Delete annotation queue interactions returns "Not Found" response
303+
Given operation "DeleteLLMObsAnnotationQueueInteractions" enabled
304+
And new "DeleteLLMObsAnnotationQueueInteractions" request
305+
And request contains "queue_id" parameter from "REPLACE.ME"
306+
And body with value {"data": {"attributes": {"interaction_ids": ["00000000-0000-0000-0000-000000000000", "00000000-0000-0000-0000-000000000001"]}, "type": "interactions"}}
307+
When the request is sent
308+
Then the response status is 404 Not Found
309+
310+
@generated @skip @team:DataDog/ml-observability
311+
Scenario: Get annotated queue interactions returns "Bad Request" response
312+
Given operation "GetLLMObsAnnotatedInteractions" enabled
313+
And new "GetLLMObsAnnotatedInteractions" request
314+
And request contains "queue_id" parameter from "REPLACE.ME"
315+
When the request is sent
316+
Then the response status is 400 Bad Request
317+
318+
@generated @skip @team:DataDog/ml-observability
319+
Scenario: Get annotated queue interactions returns "Not Found" response
320+
Given operation "GetLLMObsAnnotatedInteractions" enabled
321+
And new "GetLLMObsAnnotatedInteractions" request
322+
And request contains "queue_id" parameter from "REPLACE.ME"
323+
When the request is sent
324+
Then the response status is 404 Not Found
325+
326+
@generated @skip @team:DataDog/ml-observability
327+
Scenario: Get annotated queue interactions returns "OK" response
328+
Given operation "GetLLMObsAnnotatedInteractions" enabled
329+
And new "GetLLMObsAnnotatedInteractions" request
330+
And request contains "queue_id" parameter from "REPLACE.ME"
331+
When the request is sent
332+
Then the response status is 200 OK
333+
334+
@generated @skip @team:DataDog/ml-observability
335+
Scenario: List LLM Observability annotation queues returns "Bad Request" response
336+
Given operation "ListLLMObsAnnotationQueues" enabled
337+
And new "ListLLMObsAnnotationQueues" request
338+
When the request is sent
339+
Then the response status is 400 Bad Request
340+
341+
@generated @skip @team:DataDog/ml-observability
342+
Scenario: List LLM Observability annotation queues returns "OK" response
343+
Given operation "ListLLMObsAnnotationQueues" enabled
344+
And new "ListLLMObsAnnotationQueues" request
345+
When the request is sent
346+
Then the response status is 200 OK
347+
224348
@generated @skip @team:DataDog/ml-observability
225349
Scenario: List LLM Observability dataset records returns "Bad Request" response
226350
Given operation "ListLLMObsDatasetRecords" enabled
@@ -357,6 +481,33 @@ Feature: LLM Observability
357481
When the request is sent
358482
Then the response status is 200 OK
359483

484+
@generated @skip @team:DataDog/ml-observability
485+
Scenario: Update an LLM Observability annotation queue returns "Bad Request" response
486+
Given operation "UpdateLLMObsAnnotationQueue" enabled
487+
And new "UpdateLLMObsAnnotationQueue" request
488+
And request contains "queue_id" parameter from "REPLACE.ME"
489+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
490+
When the request is sent
491+
Then the response status is 400 Bad Request
492+
493+
@generated @skip @team:DataDog/ml-observability
494+
Scenario: Update an LLM Observability annotation queue returns "Not Found" response
495+
Given operation "UpdateLLMObsAnnotationQueue" enabled
496+
And new "UpdateLLMObsAnnotationQueue" request
497+
And request contains "queue_id" parameter from "REPLACE.ME"
498+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
499+
When the request is sent
500+
Then the response status is 404 Not Found
501+
502+
@generated @skip @team:DataDog/ml-observability
503+
Scenario: Update an LLM Observability annotation queue returns "OK" response
504+
Given operation "UpdateLLMObsAnnotationQueue" enabled
505+
And new "UpdateLLMObsAnnotationQueue" request
506+
And request contains "queue_id" parameter from "REPLACE.ME"
507+
And body with value {"data": {"attributes": {"description": "Updated description", "name": "Updated queue name"}, "type": "queues"}}
508+
When the request is sent
509+
Then the response status is 200 OK
510+
360511
@generated @skip @team:DataDog/ml-observability
361512
Scenario: Update an LLM Observability dataset returns "Bad Request" response
362513
Given operation "UpdateLLMObsDataset" enabled

0 commit comments

Comments
 (0)