Great job at making this work - it makes experimenting with DuckDB so much easier and faster while looking decent!
I've been wanting to make use of it for charting some local parquet files via duckdb which did have some rough edges and which I think may be partly a duckdb issue duckdb/duckdb-wasm#1812 which led be down a rabbit hole of digging into the code a bit.
I have made some external connection improvements which now allows me to read my local parquet files properly and far faster as expected (using several GB files in some cases).
(Ignore the moved connection dropdown + top bar, just me playing around)
In a similar vein I did also start tackling file importing via OPFS and local system access, giving the main duckdb-wasm db connection more optimisation options when it comes to larger queries by offloading data chunks to disk. I've just seen you've done a bit of improvements around that which wipes that though - i'll have to reinvestigate the changes but looks good! (dragging a file/folder directly onto the new files section would be a huge +1 for usability).
If you'd accept some PRs a few questions that would ease things:
- Database connections/attached/etc. Is your general idea that we can only be connected to a single database connection at any time and all queries are routed to that connection? Or is there a plan to have per-connection query tabs?
- Importing data loaded the files completely into memory via arrayBuffer which was then passed to duckdb. I did revamp this code area to use FileSystem instances and passing that to duckdb which prevented loading all the data into memory. It seems that your new design now only accepts folders although there is now 2 places for imports. Are there plans to consolidating these?
- When importing data it creates (or auto overwrites which is dangerous..) tables via
create or replace... select * from read_* to load the data. Can you see any issue if that was replaced by creating a view into the select * from read_* instead? This prevents loading everything into memory and can keep the remote data fresh each time it is queried. Possibly the option of "importing" and "linking to" for both options?
- There are cases where importing filename conventions are conflicting. OPFS names whitelist
.db while importing duckdb files checks for .duckdb. Are you open to relaxing this and allowing any naming? I do see a varied mix in the wild.
- The heavy use of AI has made a lot of the code very difficult to work with - any objections for PRs cleaning a few areas up making things easier? Eg. splitting up the different data type logic on imports, that store file is.. a lot, etc.
- Personally I prefer using DuckPlot for my charting but everyone has a preference for their data output. I'd like to integrate a pluggable area on the "Table"/"Charts" bar that allows adding new options there. It can then support opening any type of component in the bottom view such as DuckPlot or a notebook or anything anyone can imagine.
- Objections to getting prettier in there so I'm not conflicting with your styling? Similar vein, the AI stuff has thrown in a lot of errors that turns my IDE bright red with all the issues it finds, mostly linting with some react bugs. Any pipelines / github actions to check for breaking eslinting rules?
- There's a lot of UI areas that are confusing or awkwardly placed eg. connection switcher on the homepage then a read-only connection dropdown on each query tab, query history not available on a query tab, etc. Are you open to UI improvements that rearrange the layout at all?
- I was very confused about the "External" connection type at first as it talks about a quackpy service but then links to the DuckDB HTTP server docs. What are your plans for this? To make this easier I had readjusted the connection screen with a more solid connection step (opened when clicking Add Connection) that gives some default values and will have some info text to the right containing the duckdb connection text to copy/paste and links to further docs. It did need some external query adjustments to work with the http server extension but it now works with out of the box duckdb. If you're happy with this direction I can put this into a PR. It does also add a "Connect a local DuckDB" box on the home tab so it's a 1 or 2 click setup to query your own DB instance.
A bit of a question dump but I'd rather ask these now before making any deep changing PRs. Thanks!
Great job at making this work - it makes experimenting with DuckDB so much easier and faster while looking decent!
I've been wanting to make use of it for charting some local parquet files via duckdb which did have some rough edges and which I think may be partly a duckdb issue duckdb/duckdb-wasm#1812 which led be down a rabbit hole of digging into the code a bit.
I have made some external connection improvements which now allows me to read my local parquet files properly and far faster as expected (using several GB files in some cases).
(Ignore the moved connection dropdown + top bar, just me playing around)
In a similar vein I did also start tackling file importing via OPFS and local system access, giving the main duckdb-wasm db connection more optimisation options when it comes to larger queries by offloading data chunks to disk. I've just seen you've done a bit of improvements around that which wipes that though - i'll have to reinvestigate the changes but looks good! (dragging a file/folder directly onto the new files section would be a huge +1 for usability).
If you'd accept some PRs a few questions that would ease things:
create or replace... select * from read_*to load the data. Can you see any issue if that was replaced by creating a view into theselect * from read_*instead? This prevents loading everything into memory and can keep the remote data fresh each time it is queried. Possibly the option of "importing" and "linking to" for both options?.dbwhile importing duckdb files checks for.duckdb. Are you open to relaxing this and allowing any naming? I do see a varied mix in the wild.A bit of a question dump but I'd rather ask these now before making any deep changing PRs. Thanks!