Dynamic content blocks?

[dstengle]

There is an excellent (looking) plugin for Obsidian https://github.com/blacksmithgu/obsidian-dataview that allows you to embed rich and dynamic content in your knowledge base.

Having such dynamic content would be a huge thing for me as it would close the gap for automating some lists of content in certain places.

Is this something that might at least be possible with the API (when it is released)? Would it require another extension on top of foam to take over the viewing and editing of markdown files?

I found the discussion of GTD, but that is getting a little stale.

[riccardoferretti]

That plugin is quite amazing, it would be totally possible with the API.
The API is basically in a stable enough place that I am comfortable with opening that up now (I would still flag it as alpha/beta, as things might come out as the first plugins are built on it).

For the rendering in md preview it would just plug into the regular VS Code markdown-it hook.
Foam would surface all the info about the notes necessary for the main query operators, and if any others are needed, exposing those could be done.

If someone is interested in building something like that for Foam I would be happy to provide support on my side.

[dstengle]

I just found this other project, too, which might help in being a bit more general purpose, although the obsidian plugin is modular enough that it could probably be ported:

https://github.com/datopian/markdowndb

Given how much they were inspired by the dataview plugin and obsidian, I'm not totally sure why they didn't try to use it more directly.

I am interested although I have very little experience with vscode extensions and javascript/typescript. I've coded in other languages for decades, though and with the help of gpt4 and copilot I should be able to do it.

Maybe it would be good to start with some simple stuff with the API and then some pseudo dynamic content before figuring out how to work with real queries.

[dstengle]

In terms of contributing and getting up to speed, how approachable is this vs #1310?

[riccardoferretti]

With regards to Foam, I believe this one requires less knowledge of the system, basically the only thing you'd need to know is the properties exposed in Resource as that would be the basis for the queries; at the same time the work to implement everything else feels non-trivial. (but I am not expert in creating syntaxes and maybe there is a simple approach that could work here.

#1310 instead requires a little more knowledge of the inner workings of Foam and VS Code, plus a design approach around defining the options for the quick pick, but in terms of software project it seems smaller in scope.

In terms of utility for the project, dynamic content blocks have a lot of interesting potential.

Finally, as you pointed out another project along the same lines, a couple that might be interesting are https://github.com/mechatroner/RBQL and https://github.com/obsidian-tasks-group/obsidian-tasks

[stereobooster]

and one more possibility https://astro-digital-garden.stereobooster.com/recipes/obsidian-dataview/

[dstengle]

This looks cool. I'll have to dig in deeper and see how useful this is. Dataview makes a lot of assumptions around the runtime context, so I didn't go that route.

[dstengle]

So I've been learning about vscode extensions and researching how to do this. The biggest blocker at the moment is trying to use markdowndb sqlite inside a vscode extension. Simply put, native node modules are a really bad idea in vscode extensions.

One alternative would be to use sqlite.js. This is a javascript-only implementation of sqlite. It is limited in that it must read the entire database into memory. This might not be a problem for smaller repos but I don't know how it would scale. I'm also not quite sure how indexing would work.

I've looked more at how obsidian-dataview works and it seems like obsidian is a browser environment so they are using a library called localForage which assumes a browser environment. Over the years they've considered nodejs implementation but they've been pretty resistant, so getting help could be challenging.

The most straightforward implementation would be to run a separate server, preferably in a docker instance to simplify local dependencies. This could take care of the querying and the indexing.

As for the query language, I'm not sure it is worth doing something other than sql. The query language in obsidian-dataview gets pretty complex when you start to need joins. I would consider simplifying the sql somewhat to insulate from the underlying implementation for storing arbitrary metadata in a json text field. I can also look at what it would take to use the dataview query language. In general I'm not too interested in writing yet another query language.

I think for my initial attempt to figure this out I'll use an external process for the queries and write an extension to connect to it and hook into the markdown preview rendering and go from there.

Writing this all out got me wondering if there is a markdown extension that will render shell output. Yes there is: https://shd101wyy.github.io/markdown-preview-enhanced/#/code-chunk

I'm going to give that a quick try first, since sqlite3 has markdown output.

[dstengle]

Easier to just show the code: https://github.com/dstengle/foam-queries/blob/main/src/extension.ts

In this very hacky repo, there isn't even currently a preview script registered. The work is getting done in the function I registered as a markdown highlighter plugin. I got this from looking at the graphviz extension, which is pretty simple by comparison to the mermaid extension they point to.

In this way, you could add dynamic content in the plugin side of the extension which has access to all of node and then just put styling scripts into the preview script which runs in a simple web view.

Edit - this code is terrible and I've been getting a pretty steep introduction to typescript, async, etc. The async nature of some of the database init vs the sync nature of the markdown handling. I went down an async rabbit hole with lots of strange errors until I remembered seeing this: https://github.com/markdown-it/markdown-it/blob/master/docs/development.md#faq

As it is, this simple implementation sometimes requires a reload of the preview if things are not lined up.

So, this simple POC works but a more complete use-case might not.

[dstengle]

I've made a couple of updates and now I'm running the SQL supplied in the block and printing out an html table of results. At this point, things seem to be working and the only basic issue is the async initialization of the DB and needing to re-open the preview. I can at least handle that more gracefully than crashing the whole markdown process.

[dstengle]

I moved the repo here and made some notes in the readme. https://github.com/dstengle/markdown-queries

[riccardoferretti]

That looks pretty interesting. I can't quite figure out how the markdownPreview.ts code is hooked up, what am I missing?

[dstengle]

You aren't missing anything, it's not actually hooked up! Just a messy repository. I started out with the preview script but it isn't actually necessary (or possible) to do this.

[dstengle]

I've been making progress, however slowly. I've been able to update markdowndb with support for a separate tasks table so that a query like this is now working:

```markdownquery
select description, due from tasks where 'linkedin' in (select json_each.value from json_each(tasks.metadata, '$.tags')) and 'followup' in (select json_each.value from json_each(tasks.metadata, '$.tags')) and due > date();

There is plenty of opportunity for simple macro templates and I took standard date fields from obsidian dataview and made them columns (due, completion, created, start, scheduled).

[dstengle]

No, not even close! Raw queries work but I have to generate the markdown.db from a local fork of that project to add the tasks data. To get this to some kind of alpha state, I need to get the indexing to run inside the extension and I'd also like to add a sql query parser to make shorthand expressions for json values.

This would allow a query like:

select description, due from tasks where person = 'David Stenglein' and due < date() and checked = false;

due and checked are columns while person is an inline field.

This could be taken further with note types a forematter:

select title from books where author = 'Neal Stephenson';

This would see that "from books" is not one of the tables and replace that with "from notes where (ugly json_extract.type = book) and (another ugly json_extract='Neal Stephenson')"

[riccardoferretti]

Got it! Looking forward to seeing where this takes you :)

[charudatta10]

@riccardoferretti @dstengle this repo https://github.com/sn3akiwhizper-vscode/dataview-vscode tried a similar thing.

[dstengle]

I looked at either using obsidian-dataview or even parts of the code (it is beautifully written) but it is too tied to the obsidian app and the browser environment for someone like me to try to work with.

[dstengle]

My current status is working in a python marimo notebook with sqlglot to try to get transformation of the queries to work and it is pretty promising. I'm not sure there is a good equivalent library in node so I may need to figure out packaging. The transform methods make some of this really simple.

[stereobooster]

I recently created this remark plugin remark-dataview. If you have any markdown-to-db solution (like BrainDB, MarkdownDB) you can easily implement this.

Having real database makes implementation so much easier, no need to write your own SQL parser and evaluator.

[dstengle]

This is really cool! Standard markdown preview in vscode uses markdown-it, so I'm not sure how this would integrate into an extension. There is also the issue of where code runs for for a markdown preview extension, since view code doesn't actually have access to the workspace, so you need to separate your code into parts.

[stereobooster]

markdown-it isn't a problem

import markdownit from 'markdown-it'

// Actual default values
const md = markdownit({
  highlight: function (str, lang) {
    if (lang && lanf === "dataview") {
       return executeQuery(str) 
    }
    return ''; // use external default escaping
  }
});

since view code doesn't actually have access to the workspace

I will take your word for it. I have never created VSCode extensions

[dstengle]

That could work. It's been a while since I looked at it but the way I had to do it was to separate the query and styling logic. At some point I will get back to this and I'll give your library a try, unless someone else makes a vscode extension first.

[Dominic-DallOsto]

Just pinging to check whether there are any updates on this?

[balupton]

I've been on the hunt for an open-source local-first Coda.io alternative. Coda.io is a WYSIWYG document platform, but its WYSIWYG editor supports blocks that can be variables, tables that are actually databases that can be queried, and formulas/macros/algorithms/computations (which can also be saved as variables) that can query the variables, the database tables, or even run arbitrary code and render the result. The databases behind the tables are are viewed through portals/views which can be filtered subtables, kanbans, calendars, cards, forms, etc. Form fields can be added such as lookups, dropdowns, text fields, number fields, buttons, toggles/checkboxes, etc. These dynamic abilities extend to user installable and makeable packs/plugins, enable the coding engine to query anything from anywhere, beyond the builtin document and cross-document references. These features enable a platform where a simple document, extends to one with tables and databases inside, to full on applications within documents, and documents providing functionality to other documents. This has allowed me to create incredible flows within Coda, however the problem is Coda.io is incredibly slow because all that processing is happening via server-side. An open-source implementation wouldn't have that limit and be able to be local and fast.

Here's a video of Coda's dynamic blocks:
https://www.youtube.com/watch?v=vtrZsIKu2qE&list=PLcQRHmSuAAir_WKHUq11eLZDxav8G28l_&index=13
https://help.coda.io/hc/en-us/articles/39555858394637-Basics-of-Coda-formulas

I've been thinking that an open-source alternative would essentially store everything locally as say markdown documents, which preview would allow embedded and evaluated code blocks, where say a table is actually an interactive rendering of a llocal JSON schema object or a sqlite table or so on.

[riccardoferretti]

I should mention that now foam has something similar to data view in the form of "foam-query", more info in the documentation