Create replacement milestones

Author: mdbenito

README.md

@@ -34,7 +34,7 @@ After doing this, the autolinking of issues, commits, and branches will work. Se
 
 ## Usage
 
-The user must be a member of the project you want to copy or else you won't see it in the first step.
+The user must be a member of the project you want to copy. This user must be the one
 
 1. `cp sample_settings.ts settings.ts`
 1. edit settings.ts
@@ -114,11 +114,15 @@ As default it is set to false. Doesn't fire the requests to github api and only
 
 #### usePlaceholderIssuesForMissingIssues
 
-If this is set to true (default) then the migration process will automatically create empty dummy issues for every 'missing' GitLab issue (if you deleted an GitLab issue for example). Those issues will be closed on Github and they ensure, that the issue ids stay the same on both, GitLab and Github.
+If this is set to true (default) then the migration process will automatically create empty dummy issues for every 'missing' GitLab issue (if you deleted a GitLab issue for example). Those issues will be closed on Github and they ensure that the issue ids stay the same on both GitLab and Github.
+
+#### usePlaceholderMilestonesForMissingMilestones
+
+If this is set to true (default) then the migration process will automatically create empty dummy milestones for every 'missing' GitLab milestone (if you deleted a GitLab milestone for example). Those milestones will be closed on Github and they ensure that the milestone ids stay the same on both GitLab and Github.
 
 #### useReplacementIssuesForCreationFails
 
-If this is set to true (default) then the migration process will automatically create so called "replacement-issues" for every issue where the migration fails. This replacement issue will be exactly the same, but the original description will be lost. In the future, the description of the replacement issue will also contain a link to the original issue on GitLab. This way users, who still have access to the GitLab repository can still view its content. However, this is still an open task. (TODO)
+If this is set to true (default) then the migration process will automatically create so called "replacement-issues" for every issue where the migration fails. This replacement issue will be exactly the same, but the original description will be lost. In the future, the description of the replacement issue will also contain a link to the original issue on GitLab. This way, users who still have access to the GitLab repository can still view its content. However, this is still an open task. (TODO)
 
 It would of course be better to find the cause for migration fails, so that no replacement issues would be needed. Finding the cause together with a retry-mechanism would be optimal, and will maybe come in the future - currently the replacement-issue-mechanism helps to keep things in order.
 
@@ -150,11 +154,11 @@ Maps the usernames from gitlab to github. If the assinee of the gitlab issue is
 
 ### projectmap
 
-When one renames the project while transfering so that the projects don't loose there links to the mentioned issues.
+This is useful when migrating multiple projects if they are renamed at destination. Provide a map from gitlab names to github names so that any cross-project references (e.g. issues) are not lost.
 
 ## Import limit
 
-Because Github has a limit of 5000 Api requests per hour one has to watch out that one doesn't get over this limit. I transferred one of my project with it ~ 300 issues with ~ 200 notes. This totals to some 500 objects excluding commits which are imported through githubs importer. I never got under 3800 remaining requests (while testing it two times in one hour).
+Because Github has a limit of 5000 Api requests per hour one has to be careful not to go over this limit. I transferred one of my project with it ~ 300 issues with ~ 200 notes. This totals to some 500 objects excluding commits which are imported through githubs importer. I never got under 3800 remaining requests (while testing it two times in one hour).
 
 So the rule of thumb should be that one can import a repo with ~ 2500 issues without a problem.
 
@@ -165,7 +169,9 @@ So the rule of thumb should be that one can import a repo with ~ 2500 issues wit
 See section 'useReplacementIssuesForCreationFails' above for more infos!
 One reason seems to be some error with `Octokit` (error message snippet: https://pastebin.com/3VNUNYLh)
 
-### Milestone refs and issue refs
+### Milestone, MR and issue references
+
+This is WIP
 
 the milestone refs and issue refs do not seem to be rewritten properly at the
 moment. specifically, milestones show up like `%4` in comments
@@ -182,6 +188,6 @@ A throttling mechanism could maybe help to avoid api rate limit errors.
 In some scenarios the ability to migrate is probably more important than the total
 duration of the migration process. Some users may even be willing to accept a very long duration (> 1 day if necessary?), if they can get the migration done at all, in return.
 
-### Make request run in parallel
+### Make requests run in parallel
 
-Some requests could be run in parallel, to shorten the total duration. Currently all GitLab- and Github-Api-Requests are being run sequentially.
+Some requests could be run in parallel, to shorten the total duration. Currently all GitLab- and Github-Api-Requests are run sequentially.

sample_settings.ts

@@ -36,6 +36,7 @@ export default {
     mergeRequests: true,
   },
   debug: false,
+  usePlaceholderMilestonesForMissingMilestones: true,
   usePlaceholderIssuesForMissingIssues: true,
   useReplacementIssuesForCreationFails: true,
   useIssuesForAllMergeRequests: false,

src/githubHelper.ts

@@ -1,13 +1,21 @@
 import settings from '../settings';
 import { GithubSettings } from './settings';
 import * as utils from './utils';
-import {Octokit as GitHubApi, RestEndpointMethodTypes} from '@octokit/rest';
-import { IssuesListForRepoResponseData, PullsListResponseData } from "@octokit/types";
-import GitlabHelper from './gitlabHelper';
+import { Octokit as GitHubApi, RestEndpointMethodTypes } from '@octokit/rest';
+import {
+  IssuesListForRepoResponseData,
+  PullsListResponseData,
+} from '@octokit/types';
+import { GitlabHelper } from './gitlabHelper';
 
 const gitHubLocation = 'https://github.com';
 
-export default class GithubHelper {
+interface Milestone {
+  number: number;
+  title: string;
+}
+
+export class GithubHelper {
   githubApi: GitHubApi;
   githubUrl: string;
   githubOwner: string;
@@ -20,10 +28,12 @@ export default class GithubHelper {
   delayInMs: number;
   useIssuesForAllMergeRequests: boolean;
 
-  constructor(githubApi: GitHubApi,
-              githubSettings: GithubSettings,
-              gitlabHelper: GitlabHelper,
-              useIssuesForAllMergeRequests: boolean) {
+  constructor(
+    githubApi: GitHubApi,
+    githubSettings: GithubSettings,
+    gitlabHelper: GitlabHelper,
+    useIssuesForAllMergeRequests: boolean
+  ) {
     this.githubApi = githubApi;
     this.githubUrl = githubSettings.baseUrl
       ? githubSettings.baseUrl
@@ -51,10 +61,9 @@ export default class GithubHelper {
   async registerRepoId() {
     try {
       await utils.sleep(this.delayInMs);
-      // get an array of GitHub milestones for the new repo
       let result = await this.githubApi.repos.get({
         owner: this.githubOwner,
-        repo: this.githubRepo
+        repo: this.githubRepo,
       });
 
       this.repoId = result.data.id;
@@ -70,7 +79,7 @@ export default class GithubHelper {
   /**
    * Get a list of all GitHub milestones currently in new repo
    */
-  async getAllGithubMilestones() {
+  async getAllGithubMilestones(): Promise<Milestone[]> {
     try {
       await utils.sleep(this.delayInMs);
       // get an array of GitHub milestones for the new repo
@@ -81,12 +90,10 @@ export default class GithubHelper {
       });
 
       // extract the milestone number and title and put into a new array
-      const milestones = result.data.map(x => ({
+      return result.data.map(x => ({
         number: x.number,
         title: x.title,
       }));
-
-      return milestones;
     } catch (err) {
       console.error('Could not access all GitHub milestones');
       console.error(err);
@@ -209,9 +216,12 @@ export default class GithubHelper {
    * TODO description
    */
   async createIssue(milestones, issue) {
-    let bodyConverted = await this.convertIssuesAndComments(issue.description, issue);
+    let bodyConverted = await this.convertIssuesAndComments(
+      issue.description,
+      issue
+    );
 
-    let props : RestEndpointMethodTypes["issues"]["create"]["parameters"] = {
+    let props: RestEndpointMethodTypes['issues']['create']['parameters'] = {
       owner: this.githubOwner,
       repo: this.githubRepo,
       title: issue.title.trim(),
@@ -316,8 +326,9 @@ export default class GithubHelper {
     }
 
     console.log(
-      `\t...Done creating issue comments (migrated ${nrOfMigratedNotes} comments, skipped ${notes.length -
-        nrOfMigratedNotes} comments)`
+      `\t...Done creating issue comments (migrated ${nrOfMigratedNotes} comments, skipped ${
+        notes.length - nrOfMigratedNotes
+      } comments)`
     );
   }
 
@@ -395,7 +406,7 @@ export default class GithubHelper {
     // default state is open so we don't have to update if the issue is closed.
     if (issue.state !== 'closed' || githubIssue.state === 'closed') return;
 
-    let props: RestEndpointMethodTypes["issues"]["update"]["parameters"] = {
+    let props: RestEndpointMethodTypes['issues']['update']['parameters'] = {
       owner: this.githubOwner,
       repo: this.githubRepo,
       issue_number: githubIssue.number,
@@ -404,27 +415,28 @@ export default class GithubHelper {
 
     await utils.sleep(this.delayInMs);
 
-    if (settings.debug) {
-      return Promise.resolve();
-    }
-    // make the state update
+    if (settings.debug) return Promise.resolve();
+
     return await this.githubApi.issues.update(props);
   }
 
   // ----------------------------------------------------------------------------
 
   /**
    * Create a GitHub milestone from a GitLab milestone
+   * @param milestone GitLab milestone data
+   * @return Created milestone data (or void if debugging => nothing created)
    */
-  async createMilestone(milestone) {
+  async createMilestone(milestone): Promise<Milestone | void> {
     // convert from GitLab to GitHub
-    let githubMilestone : RestEndpointMethodTypes["issues"]["createMilestone"]["parameters"] = {
-      owner: this.githubOwner,
-      repo: this.githubRepo,
-      title: milestone.title,
-      description: milestone.description,
-      state: milestone.state === 'active' ? 'open' : 'closed',
-    };
+    let githubMilestone: RestEndpointMethodTypes['issues']['createMilestone']['parameters'] =
+      {
+        owner: this.githubOwner,
+        repo: this.githubRepo,
+        title: milestone.title,
+        description: milestone.description,
+        state: milestone.state === 'active' ? 'open' : 'closed',
+      };
 
     if (milestone.due_date) {
       githubMilestone.due_on = milestone.due_date + 'T00:00:00Z';
@@ -433,8 +445,12 @@ export default class GithubHelper {
     await utils.sleep(this.delayInMs);
 
     if (settings.debug) return Promise.resolve();
-    // create the GitHub milestone
-    return await this.githubApi.issues.createMilestone(githubMilestone);
+
+    const created = await this.githubApi.issues.createMilestone(
+      githubMilestone
+    );
+
+    return { number: created.data.number, title: created.data.title };
   }
 
   // ----------------------------------------------------------------------------
@@ -574,13 +590,13 @@ export default class GithubHelper {
         await this.githubApi.pulls.create(props);
         return Promise.resolve({ data: null }); // need to return null promise for parent to wait on
       } catch (err) {
-        if(err.status === 422) {
+        if (err.status === 422) {
           console.error(
             `Pull request #${pullRequest.iid} - attempt to create has failed, assume '${pullRequest.source_branch}' has already been merged => cannot migrate pull request, creating an issue instead.`
           );
           // fall through to next section
         } else {
-          throw (err);
+          throw err;
         }
       }
     }
@@ -651,8 +667,9 @@ export default class GithubHelper {
     }
 
     console.log(
-      `\t...Done creating pull request comments (migrated ${nrOfMigratedNotes} pull request comments, skipped ${notes.length -
-        nrOfMigratedNotes} pull request comments)`
+      `\t...Done creating pull request comments (migrated ${nrOfMigratedNotes} pull request comments, skipped ${
+        notes.length - nrOfMigratedNotes
+      } pull request comments)`
     );
   }
 
@@ -667,7 +684,7 @@ export default class GithubHelper {
    * @returns {Promise<Github.Response<Github.IssuesUpdateResponse>>}
    */
   async updatePullRequestData(githubPullRequest, pullRequest, milestones) {
-    let props: RestEndpointMethodTypes["issues"]["update"]["parameters"] = {
+    let props: RestEndpointMethodTypes['issues']['update']['parameters'] = {
       owner: this.githubOwner,
       repo: this.githubRepo,
       issue_number: githubPullRequest.number || githubPullRequest.iid,
@@ -746,7 +763,7 @@ export default class GithubHelper {
     if (pullRequest.state !== 'closed' || githubPullRequest.state === 'closed')
       return;
 
-    let props : RestEndpointMethodTypes["issues"]["update"]["parameters"] = {
+    let props: RestEndpointMethodTypes['issues']['update']['parameters'] = {
       owner: this.githubOwner,
       repo: this.githubRepo,
       issue_number: githubPullRequest.number,
@@ -825,7 +842,12 @@ export default class GithubHelper {
         }
       );
 
-      strWithMigLine = await utils.migrateAttachments(strWithMigLine, this.repoId, settings.s3, this.gitlabHelper);
+      strWithMigLine = await utils.migrateAttachments(
+        strWithMigLine,
+        this.repoId,
+        settings.s3,
+        this.gitlabHelper
+      );
 
       return strWithMigLine;
     }
@@ -899,10 +921,7 @@ export default class GithubHelper {
         line = position.old_line;
       }
       const crypto = require('crypto');
-      const hash = crypto
-        .createHash('md5')
-        .update(path)
-        .digest('hex');
+      const hash = crypto.createHash('md5').update(path).digest('hex');
       slug = `#diff-${hash}${side}${line}`;
     }
     // Mention the file and line number. If we can't get this for some reason then use the commit id instead.

src/gitlabHelper.ts

@@ -1,9 +1,18 @@
 import { Gitlab } from '@gitbeaker/node';
 import { GitlabSettings } from './settings';
 import axios from 'axios';
+// import { MilestoneSchema } from '@gitbeaker/core/dist/types/types';
+// export type Milestone = Partial<MilestoneSchema>;
 
+export interface Milestone {
+  id?: number; // internal gitlab identifier
+  iid: number; // milestone number, equivalent to github number
+  title: string;
+  description: string;
+  state: string;
+}
 
-export default class GitlabHelper {
+export class GitlabHelper {
   // Wait for this issue to be resolved
   // https://github.com/jdalrymple/gitbeaker/issues/793
   gitlabApi: InstanceType<typeof Gitlab>;
@@ -94,14 +103,17 @@ export default class GitlabHelper {
     try {
       const host = this.host.endsWith('/') ? this.host : this.host + '/';
       const attachmentUrl = host + this.projectPath + relurl;
-      const data = (await axios.get(attachmentUrl, {
+      const data = (
+        await axios.get(attachmentUrl, {
           responseType: 'arraybuffer',
           headers: {
             // HACK: work around GitLab's API lack of GET for attachments
             // See https://gitlab.com/gitlab-org/gitlab/-/issues/24155
-            Cookie: `_gitlab_session=${this.sessionCookie}`
-          }})).data;
-      return Buffer.from(data, 'binary')
+            Cookie: `_gitlab_session=${this.sessionCookie}`,
+          },
+        })
+      ).data;
+      return Buffer.from(data, 'binary');
     } catch (err) {
       console.error(`Could not download attachment #${relurl}.`);
       return null;
@@ -120,11 +132,11 @@ export default class GitlabHelper {
    */
   async getAllMergeRequestNotes(pullRequestIid: number) {
     try {
-      return (this.gitlabApi.MergeRequestNotes.all(
+      return this.gitlabApi.MergeRequestNotes.all(
         this.gitlabProjectId,
         pullRequestIid,
         {}
-      ) as any) as any[];
+      ) as any as any[];
     } catch (err) {
       console.error(
         `Could not fetch notes for GitLab merge request #${pullRequestIid}.`