Skip to content

Commit 5614694

Browse files
JesusTheHunJesusTheHun
committed
feat(http-client): wait for FTS
1 parent 1d1cfb2 commit 5614694

18 files changed

Lines changed: 726 additions & 7 deletions

‎docs/src/guide/test-with-couchbase.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# How to write tests with Couchbase
2+
3+
When you write tests interacting with Couchbase, it's easy to get fooled by its asynchronicity.
4+
Buckets, scopes, collections, query indexes and search indexes are built asynchronously.
5+
6+
This means a numbers of things to avoid the flakiness of our tests.
7+
8+
## TL;DR - Cheat sheet
9+
10+
| | Check / Solution | Cbjs helper |
11+
|--------------------------------------|---------------------------------------------------------------------------------------|---------------------------------------|
12+
| Write a keyspace inside a bucket | GET `/pools/default/<bucket>` is a `200` | `waitForBucket` |
13+
| Write a keyspace inside a scope | Response of `/pools/default/<bucket>/scopes` includes the scope | `waitForScope` |
14+
| Write a keyspace inside a collection | Response of `/pools/default/<bucket>/scopes` includes the collection | `waitForCollection` |
15+
| Write a document in the bucket | The bucket stats (ram, vBuckets) are not empty | `waitForBucket` |
16+
| Write a document in the scope | Bucket is writable and scope is included in `/pools/default/<bucket>/scopes` | `waitForScope` |
17+
| Write a document in the collection | Bucket is writable and collection is included in `/pools/default/<bucket>/scopes` | `waitForCollection` |
18+
| Create a query index using a bucket | `SELECT RAW name FROM system:keyspaces` | `getQueryBuckets` |
19+
| Query fresh documents | Query with `REQUEST_PLUS` consistency on the indexes involved before our actual query | Too business specific to get a helper |
20+
| Full Text Search fresh documents | Query for docIds and wait for the result to include our documents | `waitForDocumentsInSearchIndex` |
21+
22+
23+
## Keyspaces (buckets, scopes and collections)
24+
25+
When we create a keyspace object, the API will return immediately, but the creation is not instantaneous.
26+
Each service of each node will be notified asynchronously about the new keyspace.
27+
28+
We will need to wait for the keyspace to be known and ready in order to :
29+
30+
1. Create a keyspace object within it
31+
2. Write a document within it
32+
3. Create an index that references it
33+
34+
The check must be performed on each node involved ; so if you want to write a document in a fresh keyspace, you need to
35+
check every node with the `kv` service before writing the document.
36+
For indexes, unless the index is configured to be on specific nodes, you will need to check every node with the `query` service.
37+
38+
To make sure a keyspace is visible by the query service, you need to execute the following statement and verify our keyspace is
39+
included in the result set :
40+
41+
```sql
42+
SELECT RAW name FROM system:keyspaces
43+
```
44+
45+
## Query fresh documents
46+
47+
Since the indexes are built asynchronously, if we execute a query just after writing a document, the query result
48+
is not guaranteed to include the latest changes.
49+
If we want to have the latest updates, we can execute our query with the `REQUEST_PLUS` consistency.
50+
But it is likely that our actual business query does not require this level of consistency, so the trick here will be to execute a _consistency query_
51+
prior to our actual query.
52+
53+
Let's say we are testing a query that retrieves the blog posts authored by a user :
54+
55+
Given we have the following index: `CREATE INDEX posts_listing ON posts (authorId, categoryId, createdAt)`, we will proceed as follow :
56+
57+
1. We insert our test blog posts
58+
2. We execute `SELECT META().id FROM posts WHERE authorId IS NOT MISSING` with `REQUEST_PLUS` consistency - that's our _consistency query_
59+
3. We call `getUserPosts(userId)` which executes `SELECT META().id, title FROM posts WHERE authorId = $1 ORDER BY createdAt DESC`
60+
61+
The `WHERE` clause in our _consistency query_ is very important because it targets the index we are using in our business query.
62+
We only need to specify the first field of the index for it to be used by the query,
63+
so one _consistency query_ can be reused for every query that use the same index.
64+
65+
So now we will create a helper function `waitForBlogPosts` that executes the query.
66+
If we create a new index on the `posts` collection, we can add another _consistency query_ with an **index hint** in our helper function, so
67+
we only have to know about this helper.

‎packages/http-client/src/services/search/getSearchIndexDocumentsByIds.ts‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
/*
2+
* Copyright (c) 2023-Present Jonathan MASSUCHETTI <jonathan.massuchetti@dappit.fr>.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import { CouchbaseHttpApiConfig } from '../../types.js';
17+
import { ApiSearchResult } from '../../types/Api/search/ApiSearchResult.js';
18+
import { createHttpError } from '../../utils/createHttpError.js';
19+
import { requestGetScopedSearchIndexDocumentsByIds } from './requests/requestGetScopedSearchIndexDocumentsByIds.js';
20+
import { requestGetSearchIndexDocumentsByIds } from './requests/requestGetSearchIndexDocumentsByIds.js';
21+
22+
export async function getSearchIndexDocumentsByIds(
23+
params: CouchbaseHttpApiConfig,
24+
index:
25+
| string
26+
| {
27+
bucket: string;
28+
scope: string;
29+
index: string;
30+
},
31+
docIds: string[]
32+
) {
33+
const response =
34+
typeof index === 'string'
35+
? await requestGetSearchIndexDocumentsByIds(params, index, docIds)
36+
: await requestGetScopedSearchIndexDocumentsByIds(params, index, docIds);
37+
38+
if (response.status !== 200) {
39+
throw await createHttpError('POST', response);
40+
}
41+
42+
return (await response.json()) as ApiSearchResult;
43+
}

‎packages/http-client/src/services/search/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,4 @@
1717
export * from './getSearchIndex.js';
1818
export * from './getSearchIndexes.js';
1919
export * from './getQuerySearchIndexes.js';
20+
export * from './getSearchIndexDocumentsByIds.js';

‎packages/http-client/src/services/search/requests/requestGetScopedSearchIndexDocumentsByIds.ts‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/*
2+
* Copyright (c) 2023-Present Jonathan MASSUCHETTI <jonathan.massuchetti@dappit.fr>.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import 'node-fetch';
17+
18+
import { CouchbaseHttpApiConfig } from '../../../types.js';
19+
import { apiPOST } from '../../../utils/apiPOST.js';
20+
21+
export async function requestGetScopedSearchIndexDocumentsByIds(
22+
params: CouchbaseHttpApiConfig,
23+
index: {
24+
bucket: string;
25+
scope: string;
26+
index: string;
27+
},
28+
docIds: string[]
29+
) {
30+
return apiPOST(
31+
{ ...params },
32+
`/api/bucket/${index.bucket}/scope/${index.scope}/index/${index.index}/query`,
33+
JSON.stringify({
34+
query: {
35+
ids: docIds,
36+
},
37+
}),
38+
'search'
39+
);
40+
}

‎packages/http-client/src/services/search/requests/requestGetSearchIndexDocumentsByIds.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
/*
2+
* Copyright (c) 2023-Present Jonathan MASSUCHETTI <jonathan.massuchetti@dappit.fr>.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import 'node-fetch';
17+
18+
import { CouchbaseHttpApiConfig } from '../../../types.js';
19+
import { apiPOST } from '../../../utils/apiPOST.js';
20+
21+
export async function requestGetSearchIndexDocumentsByIds(
22+
params: CouchbaseHttpApiConfig,
23+
indexName: string,
24+
docIds: string[]
25+
) {
26+
return apiPOST(
27+
{ ...params },
28+
`/api/index/${indexName}/query`,
29+
JSON.stringify({
30+
query: {
31+
ids: docIds,
32+
},
33+
}),
34+
'search'
35+
);
36+
}

‎packages/http-client/src/types/Api/search/ApiSearchResult.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
/*
2+
* Copyright (c) 2023-Present Jonathan MASSUCHETTI <jonathan.massuchetti@dappit.fr>.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
export type ApiSearchResult = {
18+
status: {
19+
total: number;
20+
failed: number;
21+
successful: number;
22+
};
23+
hits: Array<{
24+
index: string;
25+
id: string;
26+
score: number;
27+
sort: string[];
28+
}>;
29+
total_hits: number;
30+
cost: number;
31+
max_score: number;
32+
took: number;
33+
34+
/**
35+
* Please submit a PR for this 🙏
36+
*/
37+
facets: unknown;
38+
};

‎packages/http-client/src/types/Api/search/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,5 @@ export * from './ApiSearchGetIndex.js';
1919
export * from './ApiSearchIndexAnalyzeDocument.js';
2020
export * from './ApiSearchIndexCountDocuments.js';
2121
export * from './ApiSearchQuery.js';
22+
export * from './ApiSearchResult.js';
2223
export * from './types.js';

‎packages/http-client/src/waitFor/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,4 @@ export * from './waitForViewDesignDocument.js';
2626
export * from './waitForQueryIndexer.js';
2727
export * from './waitForQueryIndex.js';
2828
export * from './waitForAnalyticsCluster.js';
29+
export * from './waitForDocumentsInSearchIndex.js';

‎packages/http-client/src/waitFor/waitForDocumentsInSearchIndex.ts‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
/*
2+
* Copyright (c) 2023-Present Jonathan MASSUCHETTI <jonathan.massuchetti@dappit.fr>.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
import { retry } from 'ts-retry-promise';
17+
18+
import { getSearchIndexDocumentsByIds } from '../services/index.js';
19+
import { CouchbaseHttpApiConfig } from '../types.js';
20+
import { waitOptionsModerate } from './options.js';
21+
import { WaitForOptions } from './types.js';
22+
23+
export type WaitForDocumentsInSearchIndexOptions = WaitForOptions;
24+
25+
export async function waitForDocumentsInSearchIndex(
26+
apiConfig: CouchbaseHttpApiConfig,
27+
index:
28+
| string
29+
| {
30+
bucket: string;
31+
scope: string;
32+
index: string;
33+
},
34+
docIds: string[],
35+
options?: WaitForDocumentsInSearchIndexOptions
36+
): Promise<void> {
37+
const resolvedOptions = {
38+
...waitOptionsModerate,
39+
...options,
40+
};
41+
42+
const { expectMissing } = resolvedOptions;
43+
44+
return await retry(async () => {
45+
const result = await getSearchIndexDocumentsByIds(apiConfig, index, docIds);
46+
const allDocsFound = result.total_hits !== docIds.length;
47+
48+
if (!allDocsFound && !expectMissing)
49+
throw new Error('Some documents are not in the search index yet');
50+
if (allDocsFound && expectMissing)
51+
throw new Error('Some documents are still in the search index');
52+
}, resolvedOptions);
53+
}

‎packages/vitest/src/extendedTests/createCouchbaseTest.ts‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,10 @@ import {
3535
} from '../fixtures/couchbase/kv/index.js';
3636
import { IndexFixture, PrimaryIndexFixture } from '../fixtures/couchbase/query/index.js';
3737
import { UserFixture, UserGroupFixture } from '../fixtures/couchbase/rbac/index.js';
38-
import { SearchIndexFixture } from '../fixtures/couchbase/search/index.js';
38+
import {
39+
ScopedSearchIndexFixture,
40+
SearchIndexFixture,
41+
} from '../fixtures/couchbase/search/index.js';
3942
import { ViewDocumentKeyFixture } from '../fixtures/couchbase/views/index.js';
4043
import { LoggerFixture } from '../fixtures/misc/LoggerFixture.js';
4144
import { ServerTestContext } from '../ServerTestContext.js';
@@ -58,6 +61,7 @@ const couchbaseTestFixtures = {
5861
usePrimaryIndex: PrimaryIndexFixture,
5962
useIndex: IndexFixture,
6063
useSearchIndex: SearchIndexFixture,
64+
useScopedSearchIndex: ScopedSearchIndexFixture,
6165
} as const;
6266

6367
export type CouchbaseTestContext = {

0 commit comments

Comments
 (0)