Docs

Author: CodeHariK

Readme.md

@@ -1,10 +1,15 @@
 #
 
+* [#08 - Tree Indexes: B+Trees (CMU Intro to Database Systems)](https://www.youtube.com/watch?v=scUtG_6M_lU)
+* [SQLite: How it works](https://www.youtube.com/watch?v=ZSKLA81tBis)
 * [Write a database from scratch](https://www.youtube.com/playlist?list=PLWRwj01AnyEtjaw-ZnnAQWnVYPZF5WayV)
 
-##
+* [Understanding B-Trees: The Data Structure Behind Modern Databases](https://www.youtube.com/watch?v=K1a2Bk8NrYQ)
+
+* [Build a NoSQL Database From Scratch in 1000 Lines of Code](https://medium.com/better-programming/build-a-nosql-database-from-the-scratch-in-1000-lines-of-code-8ed1c15ed924)
+* [Writing a SQL database from scratch in Go: 1. SELECT, INSERT, CREATE and a REPL](https://notes.eatonphil.com/database-basics.html)
 
-* Build Your Own Database From Scratch
+##
 
 ```
 1. Persistence. How not to lose or corrupt your data. Recovering from a crash.

secretary/Todo.md

@@ -1,2 +1,16 @@
+* ```go
+allChildren := make([]int32, numNodes*childrenSize) // One big allocation
+
+for i := range nodes {
+    nodes[i].children = allChildren[i*childrenSize : (i+1)*childrenSize] // No new allocation
+}
+```
 * Delete key, if deletes node, keep deleted node in array for removal from disk
-* 
+* Images, binary data visual
+* Kademlia
+* Persist to storage, with compression
+* Bufferpool, Timebaseminheap
+* Inverted tree, index, ngram, bm25
+* hyperloglog, bloomfilter
+* Wal
+* Transaction concurrency

secretary/bustub.md

@@ -0,0 +1 @@
+https://github.com/cmu-db/bustub

secretary/compress.md

@@ -0,0 +1,102 @@
+🚀 Zstd (Zstandard) in Golang
+
+Zstandard (Zstd) is a fast compression algorithm that provides better compression ratios than gzip while being faster. It’s great for high-performance databases, logs, network transmission, and storage.
+
+📌 Why Use Zstd in Golang?
+	•	🔥 High compression ratio (better than gzip)
+	•	⚡ Fast decompression (ideal for real-time applications)
+	•	📉 Adjustable compression levels (trade-off between speed & compression)
+	•	🔄 Streaming support (works well for large files)
+
+📦 Install Zstd for Golang
+
+Use the optimized Zstd package by Klaus Post:
+
+go get github.com/klauspost/compress/zstd
+
+
+
+⸻
+
+✍️ Example: Compress & Decompress Using Zstd
+
+package main
+
+import (
+	"bytes"
+	"fmt"
+	"io"
+	"log"
+
+	"github.com/klauspost/compress/zstd"
+)
+
+// Compresses input data using Zstd
+func compress(data []byte) []byte {
+	var buf bytes.Buffer
+	writer, _ := zstd.NewWriter(&buf)
+	writer.Write(data)
+	writer.Close()
+	return buf.Bytes()
+}
+
+// Decompresses Zstd compressed data
+func decompress(compressed []byte) []byte {
+	reader, _ := zstd.NewReader(bytes.NewReader(compressed))
+	decompressed, _ := io.ReadAll(reader)
+	return decompressed
+}
+
+func main() {
+	data := []byte("Hello, this is a repeated text. Hello, this is a repeated text. Hello, this is a repeated text.")
+
+	// Compress data
+	compressed := compress(data)
+	fmt.Println("Original Size:", len(data))
+	fmt.Println("Compressed Size:", len(compressed))
+
+	// Decompress data
+	decompressed := decompress(compressed)
+	fmt.Println("Decompressed:", string(decompressed))
+}
+
+
+
+⸻
+
+📊 How Much Can Zstd Compress?
+
+Zstd compression depends on the data type. Here’s a rough estimate:
+
+Data Type	Compression Ratio
+Text (logs, JSON)	70-85% smaller
+Binary (executables, data)	15-40% smaller
+Already Compressed Data (JPEG, MP3, MP4)	~0%
+
+	•	Level 1 (fastest) → Lower compression, high speed.
+	•	Level 22 (slowest) → Best compression, slow speed.
+	•	Default level is 3, which is a good balance.
+
+🚀 Streaming Compression (for Large Data)
+
+If you’re dealing with large files, use streaming compression to avoid excessive memory usage:
+
+func streamCompress(reader io.Reader, writer io.Writer) error {
+	zstdWriter, _ := zstd.NewWriter(writer)
+	_, err := io.Copy(zstdWriter, reader)
+	zstdWriter.Close()
+	return err
+}
+
+This is useful for compressing database logs, network data, and large files without loading everything into memory.
+
+⸻
+
+✅ Where Can Zstd Be Useful in Your Projects?
+	•	Bitcask: Compress log segments to reduce disk usage.
+	•	B+ Tree: Compress index nodes/pages for better storage efficiency.
+	•	Kademlia (DHT): Compress large values to save bandwidth.
+	•	Gemini Server: Compress responses for faster transmission.
+
+Would you like a benchmark for your specific data in Go? 🚀
+

secretary/kademilia.md

@@ -20,6 +20,7 @@
 * https://github.com/libp2p/specs/blob/master/kad-dht/README.md
 * https://github.com/libp2p/go-libp2p/blob/master/examples/libp2p-host/host.go
 * https://github.com/libp2p/go-libp2p-kad-dht
+* https://github.com/nictuku/dht
 * https://github.com/pdelong/Kademlia
 * https://github.com/prettymuchbryce/kademlia
 

secretary/lock.md

@@ -170,3 +170,109 @@ Tradeoffs
 	•	Harder to debug than traditional locking mechanisms.
 
 Would you like me to refine specific parts, like concurrent range queries or node splits? 🚀
+
+
+
+
+
+Database Locks: Types and Usage
+
+Database locks are mechanisms used to ensure consistency, integrity, and concurrency control in multi-user environments. They prevent race conditions, dirty reads, and data corruption when multiple transactions access the same data.
+
+⸻
+
+Types of Database Locks
+
+1. Pessimistic Locking
+	•	Blocks access to a resource until a transaction is complete.
+	•	Ensures no other transaction modifies the data while a lock is held.
+	•	Typically used in high-contention scenarios (e.g., banking systems).
+
+Example (MySQL FOR UPDATE)
+
+START TRANSACTION;
+SELECT * FROM accounts WHERE id = 1 FOR UPDATE; -- Locks row until COMMIT/ROLLBACK
+UPDATE accounts SET balance = balance - 100 WHERE id = 1;
+COMMIT;
+
+Use Cases:
+✅ Ensures strong consistency.
+❌ Can cause performance issues due to waiting/blocking.
+
+⸻
+
+2. Optimistic Locking
+	•	Allows concurrent access but detects conflicts before committing.
+	•	Uses version numbers or timestamps to check if data was modified.
+	•	If a conflict is detected, the transaction is retried.
+
+Example (Using Version Number)
+
+SELECT id, balance, version FROM accounts WHERE id = 1;
+UPDATE accounts SET balance = balance - 100, version = version + 1 
+WHERE id = 1 AND version = 1; -- Fails if version changed
+
+Use Cases:
+✅ Best for low-contention scenarios.
+❌ Requires extra logic for retrying transactions.
+
+⸻
+
+3. Table Locks
+	•	Locks the entire table, preventing other transactions from reading or writing.
+	•	Used when bulk updates need consistency.
+
+Example (MySQL Table Lock)
+
+LOCK TABLES accounts WRITE;
+UPDATE accounts SET balance = balance - 100 WHERE id = 1;
+UNLOCK TABLES;
+
+Use Cases:
+✅ Guarantees full consistency.
+❌ Not scalable for multi-user applications.
+
+⸻
+
+4. Row-Level Locks
+	•	Locks only specific rows affected by a transaction.
+	•	Allows higher concurrency than table locks.
+
+Example (PostgreSQL SELECT FOR UPDATE)
+
+BEGIN;
+SELECT * FROM orders WHERE id = 123 FOR UPDATE; -- Locks row
+UPDATE orders SET status = 'shipped' WHERE id = 123;
+COMMIT;
+
+Use Cases:
+✅ Efficient for concurrent updates on different rows.
+❌ Can cause deadlocks if transactions lock rows in different orders.
+
+⸻
+
+5. Deadlocks and Handling
+
+A deadlock occurs when two transactions hold locks and wait for each other to release them.
+
+Example Deadlock (Two Transactions)
+
+Transaction A: LOCK row 1 → WAIT for row 2  
+Transaction B: LOCK row 2 → WAIT for row 1
+
+Preventing Deadlocks
+	•	Access resources in a consistent order.
+	•	Use shorter transactions to minimize lock time.
+	•	Set timeouts on locks (e.g., SELECT FOR UPDATE NOWAIT).
+
+⸻
+
+Which Locking Strategy to Use?
+
+Scenario	Best Locking Strategy
+High contention on updates	Pessimistic Locking (FOR UPDATE)
+Low contention, high concurrency	Optimistic Locking (versioning)
+Bulk operations	Table Locks (LOCK TABLES)
+Multiple transactions updating different rows	Row-Level Locks
+
+Would you like a deep dive into deadlocks, isolation levels, or specific databases (PostgreSQL, MySQL, etc.)? 🚀

secretary/lsmtree.md

@@ -1,3 +1,13 @@
+[#04 - Database Storage: Log-Structured Merge Trees & Tuples (CMU Intro to Database Systems)](https://www.youtube.com/watch?v=IHtVWGhG0Xg&t=1372s)
+
+https://github.com/facebook/rocksdb/wiki
+
+https://github.com/krasun/lsmtree
+https://github.com/skyzh/mini-lsm
+
+
+
+
 Object stores typically do not use B-trees like databases. Instead, they use hash-based indexing or LSM-trees (Log-Structured Merge Trees) depending on the use case. Here’s why:
 
 1. Hash-Based Indexing (Common for Object Stores)

secretary/olap.md

@@ -0,0 +1 @@
+https://github.com/risinglightdb/risinglight

secretary/raft.md

@@ -1 +1,5 @@
-https://notes.eatonphil.com/2023-05-25-raft.html
+* https://notes.eatonphil.com/2023-05-25-raft.html
+
+* https://github.com/otoolep/hraftd
+* https://github.com/hashicorp/raft-boltdb
+* https://github.com/Jille/raft-grpc-example

secretary/vector.md

@@ -0,0 +1 @@
+https://github.com/skyzh/write-you-a-vector-db