Improve data model and the loading service

[vbakke]

Data model

The current data model and data flow, reads the generated.yaml from the DSOMM web server (either the public dsomm.owasp.org. or a private one).

The file contains a list of activity info (all dimensions, subdimensions, activity description, , as well as each team's progress of implementations and evidence.

The generated.yaml if either copied from DSOMM-data or generated from an internal process in the organisation.

A user can update the team's progress in the Circular Heatmap. Then (more or less) a copy of the generated.yaml is stored in the browser's localStorage.

Drawbacks

The current model is storing activity description data in localStorage. And mixes "slowly changing information" with "live data" (i.e. activity + team status).

The current load process is duplicated in the matrix, activity description, circular heatmap, dependency graph, and mapping component. This leads to unnecessary load time, and complex maintenance.

Suggestion

1) Create a central load service

I suggest we create a new service that loads the yaml files, and merging this with progress data stored in localStorage.

This service will handle the race conditions, and let the other components just consume the already loaded data.

Benefit: Cleaner architecture, easier debugging, speed improvements and increased robustness when navigating in DSOMM.

2) Split the Activity info and Team's progress

I suggest we split the current generated.yaml file into:

• activity info
• team progress

(I'm completely open to name suggestions.)

• The Activity Info contains the (fairly) static information contained in today's activity object.

• The Team Progress contains team name, teamsImplemented and teamsEvidence. The Team Progress have a yaml file on the server, where any centralized progress may be updated.

• The Team Progress will be stored in localStorage if any progress is changed in the browser. It will be merged with any updates from the server file when loading.

• The Team Progress may be exported, as today, to update the server file.

This approach will also allow a user to change teams' names even for a public site such as dsomm.owasp.org.

Mermaid diagram

Here is a draft of the suggestion.

graph LR


DsommAct(activities.yaml)

DataAct --> DsommAct
subgraph "GitHub DSOMM-data"
    DataMeta(meta.yaml)

    Build.yaml --> DataAct
    Deploy.yaml --> DataAct
    PathManagement.yaml --> DataAct
    Other("... .yaml") --> DataAct
end

DataMeta --> DsommMeta
DataAct(activities.yaml)
DsommCustAct(custom-activities.yml)

subgraph DSOMM Web server
    DsommMeta(meta.yaml)
    DsommAct
    DsommCustAct
end

DsommMeta --> BrowserTeams("Teams")
DsommAct --> BrowserAct("Activities")
DsommCustAct --> BrowserCustAct(custom-activities.yml)
subgraph Browser 
    BrowserTeams -.-> BrowserProgress
    BrowserAct -.-> BrowserProgress
    BrowserCustAct-.-> BrowserProgress
    BrowserProgress("TeamProgress")
end
subgraph Browser localStorage
    BrowserTeams <--> LocalTeams("Teams")
    BrowserProgress <--> LocalProgress("TeamProgress")
end

Loading

(Mermaid is great. But has some limits. The following draft sketch should have a TeamProgress in the DSOMM Webserver, but then the drawing went bonkers, so I left it out.)

Comments?

@wurstbrot and @0x41head: Any comments on this?

I see that @yashchauhan4579 created a PR #379 just now, to upgrade Angular, aiming for Angular 19, down the line. (👍 🙌 🏆 )

They published that while I was writing this. I think it would be unwise to run these two initiatives in parallel. Even if they have different focus, there will be clashes. Nevertheless, I think it is still worthwhile discussing this idea, and planning the future.

[yashchauhan4579]

Thanks, @vbakke! Appreciate your support! 😊 Looking forward to the next steps.

[vbakke]

@wurstbrot: In assets\YAML\ there is currently a teams.yaml as well as the teams being defined in meta.yaml?

What was the reason to move the team definition into the meta file? Is that reason still valid?

To me, it also makes sense to have the teams defined in a separate teams.yaml

[wurstbrot]

In GSoC with @ptechofficial , we first had teams in teams.yaml and afterwards moved teams to meta.yaml.
I think it makes sense to go back to teams.yaml.

[vbakke]

My first draft for structuring the yaml files as follows:

─ assets
  └─ YAML
     ├─ meta.yaml 
     ├─ teams.yaml
     ├─ teamprogress.yaml
     ├─ default
     |    ├─ activities.yaml
     └─ custom
           └─ custom-activities.yaml

The teamprogress.yaml will start empty, but can be replaced with an exported file from the UI. Will contain data for both teamsImplemeneted and teamsEvidence from today's setup.

The meta.yaml will contain "links" to the activity files. Files in the default folder comes from DSOMM-data, and can be updated from DSOMM-data, without causing any harm to local customizations, nor any team progress.

The customs folder if for local customization, that can override the default DSOMM files. (It should contain a "dummy file" with examples.) A custom activity file should allow an ignore attribute, to "remove" an unwanted default activity.

meta.yaml

as today, but adds:

    checkForDsommUpdates: true

    $teamsRef: teams.yaml
    $progressRef: teamprogress.yaml
    $activityRef:
        - default/activities.yaml 
        - custom/custom-activities.yaml  # For customizing your own activities

Not sure about the naming convention $teamRef. It sort of plays on the yaml $ref feature. But does not replace the content, as the yaml spec says.

The order of $activityRef dictates the priority. Last entry takes priority over earlier files.

Updates from DSOMM

I suggest that the DSOMM client checks github for any new releases of the activities file (currently named generated.yaml).
This must of course be possible to turn off. I propose checkForDsommUpdates in the meta.yaml for controlling the behaviour.

The https://api.github.com/repos/devsecopsmaturitymodel/DevSecOps-MaturityModel-data/releases/latest contains the latest release. The public version can be verified against local version.
The user gets choices in the browser, if a new version is released:

• Remind me again next time
• Don't remind again for this update
• Don't remind me again for any update

Comments

@wurstbrot or @0x41head: I'd like to hear any comments, suggestions or resistance to this.

[vbakke]

After some tinkering with the meta.yaml I have implmented the yaml $ref feature, and suggest:

checkForDsommUpdates: true
teams: 
  $ref: 'default/teams.yaml#/teams'
teamGroups:
  $ref: 'default/teams.yaml#/teamGroups'

teamProgressFile: 'teamprogress.yaml' 
activityFiles:
    - default/activities.yaml 
    - custom/custom-activities.yaml  # For customizing your own activities

Customized activity files may include an ignore: true attribute to "remove" an activity (or dimension) from the standard DSOMM list.

And also, I do not see the purpose of the strings property in the DSOM meta.yaml. Are these remnants from thr DSOMM-Data project, @wurstbrot ?

[wurstbrot]

That is correct, they would be better placed in the data project.

[vbakke]

@0x41head: Do you have any comments on: https://github.com/vbakke/DevSecOps-MaturityModel/tree/feat/central-loader

(Before I refactor how the matrix, circular heatmap, etc load and display its data : )

[vbakke]

Terminology
My gut feeling tells me that DSOMM is focusing more on the 'Sub-Dimension' level, than the 'Dimension' level.

• What do you feel about renaming the two concepts to 'Dimension' and 'Category'?

Thus:

• 'Dimension' -> 'Category'
• 'Sub-Dimension' -> 'Dimension'

(At least my brain always starts sidetracking, following associations as what a subdimension is it sub to? The term 'Category' is more clearly a group of other items. And I find it easier to talk about the 'Design dimension', rather than the 'Design subdimension'.)

Any thoughts from anyone?
@wurstbrot?

[vbakke]

@wurstbrot, is renaming okay for you?

[vbakke]

@wurstbrot or @0x41head: I'd love to hear comments about the refactoring.

I have published a fork of the the latest update in feat/central-loader-matrix on: https://dsomm.pages.dev/ as "experimental edition".

Current status:

• new data-loader-service and yaml-loader
• matrix page is using new format (and IMHO is simplified)
• improved feedback if yaml files don't pass valiudation

Next up is the activity details page, and then circular heatmap.

Filters
My intention was not to modify the UI. But I ended up with a modification of how the filters work for the Matrix.

I prefer the standard BI way of selecting filters:

• no filters => show everything
• some filters selected: show these filters only

Please let me know if you object.

[wurstbrot]

Hi @vbakke,
dimension and sub-dimension are scientific terms used in maturity models for categories and sub-categories.
I am not convinced that a renaming is needed. If a renaming would happen, there shouldn't be a shift from dimension to sub-dimensions being called "dimensions" (which has been used beforehand).

I didn't go through the hole code, but the new architecture looks good in general. We can discuss it in a PR.

Flipping filters is great.

Why did you change the background color from grey to yellowish?

[vbakke]

Thank you @wurstbrot. The yellowish tint is just in the experimental edition, so I don't get confused as to which I am looking at.

Thank you for having a look at the overall architecture. PR will come. But all the rendering gets improved as a result unifying the data structure. So it becomes a quite large PR. And one cannot split it, due to the inherent dependencies.

I don't know any maturity models using the term 'sub-dimensions'. The ones I looked up, uses different terminology on the various levels of the hierarchical break-down. (Such as 'Business function', 'Security practices', and 'Streams' in SAMM.)

The term 'dimensions', is fine used in a DSOMM context. I just don't see the term "Build and Deployment" as one dimension, but rather a group of dimensions. (The 'and' sort of implies that : ) While 'Build' I find more natural being one dimension.

I can revert back to using 'dimensions' and 'sub-dimensions' if you'd like. But I would applaud having terms that indicated the level in the hierarchy, without having to use the word 'sub'. 'Sub' is understandable, but not optimal.