KEMBAR78
Couchdb: No SQL? No driver? No problem | KEY
Uploaded bydelagoya
KEY, PDF3,254 views

Couchdb: No SQL? No driver? No problem

The document discusses CouchDB, a document-oriented NoSQL database that utilizes JSON for storage and an HTTP RESTful API for interactions. It covers key concepts such as the CAP theorem, document management, ACID guarantees on a per-document basis, and the use of JavaScript for application logic through design documents. Additionally, it outlines practical operations like creating, reading, updating, and deleting documents, as well as backup and replication strategies.

Downloaded 184 times
* CouchDB No SQL? No Driver? No problem. Angel Pizarro angel@upenn.edu * www.bauwel-movement.co.uk/ sculpture.php
About Me Me: CBIL alumni! Work in mass spec proteomics Lots of data in lots of formats in bioinformatics Ruby for programming and Ruby on Rails for Web apps But that doesn’t matter for CouchDB! Interested in CouchDB for AWS deployment
Overview Talk about Key-Value stores Introduce some general theory and concepts CouchDB specifics Example problem More CouchDB specifics Questions?
Key-Value Databases Datastore of values indexed by keys (duh!) Hash or B-Tree index for keys Cassandra Hash is FAST, but only allows single-value lookups B-Tree is slower, but allows range queries Horizontally scalable - via key partitioning
The CAP theory : applies when business logic is separate from storage Consistency vs. Availability vs. Partition tolerance RDBMS = enforced consistency PAXOS = quorum consistency CouchDB (and others) = eventual consistency and horizontally scalable http://www.julianbrowne.com/article/viewer/brewers-cap-theorem
CouchDB
CouchDB Document Oriented Database JSON documents
CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca * Hovercraft: http://github.com/jchris/hovercraft/
CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis * Hovercraft: http://github.com/jchris/hovercraft/
CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views * Hovercraft: http://github.com/jchris/hovercraft/
CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views Back-ups and replication are easy-peasy * Hovercraft: http://github.com/jchris/hovercraft/
Javascript Object Notation
Javascript Object Notation * * http://www.json.org
Example JSON { “name”: “J. Doe”, “friends”: 0, “traits”: [“nice”, “outgoing”] }
REST
REST Representational State Transfer
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP)
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs
REST Representational State Transfer Clients-Server separation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs GET, PUT, POST, DELETE, HEAD
CouchDB REST/CRUD GET read PUT create or update DELETE delete something POST bulk operations
CouchDB passes the ACID test Each document is completely self-sufficient Each document has a version number An update operation writes a complete new copy of the the record and is assigned the new version number Append-only file structure allows the write to occur while still serving read requests
MVCC RDBMS CouchDB Multi-Version Concurrency Control RDBMS enforces consistency using read/write locks Instead of locks, CouchDB just serve up old data Multi-document (mutli-row) transactional semantics must be handled by the application
Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
Database API Create a DB: Protocol $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
Database API Create a DB: CouchDB server $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
Database API Create a DB: DB name $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"}
Database API Create a DB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"} Not recoverable! Delete a DB: $ curl -X DELETE http://localhost:5984/friendbook {"ok":true}
Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]}
Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}
Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json Read a JSON file {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}
Full JSON document Before { "name": "J. Doe", "friends": 0 } After { "_id": "j_doe", "_rev": "1-062af1c4ac73287b7e07396c86243432", "name": "J. Doe", "friends": 0 }
Updating a document $ curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conflict","reason":"Document update conflict."}
Updating a document $ curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conflict","reason":"Document update conflict."} Must give _rev (revision number) for updates! revised.json { "_rev":"1-062af1c4ac73287b7e07396c86243432", "name":"J. Doe", "friends": 1 } $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @revised.json {"ok":true,"id":"j_doe","rev":"2-0629239b53a8d146a3a3c4c63e 2dbfd0"}
Deleting a document $ curl -X DELETE http://localhost:5984/friendbook/j_doe {"error":"conflict","reason":"Document update conflict."} Must give revision number for deletes! $ curl -X DELETE http://localhost:5984/friendbook/j_doe? rev=2-0629239b53a8d146a3a3c4c63e2dbfd0 {"ok":true,"id":"j_doe", "rev":"3-57673a4b7b662bb916cc374a92318c6b"} Returns a revision number for the delete $ curl -X GET http://localhost:5984/friendbook/j_doe {"error":"not_found","reason":"deleted"}
Bulk operation POST /database/_bulk_docs with a JSON document containing all of the new or updated documents. // documents to bulk upload { "docs": [ {"_id": "0", "integer": 0, "string": "0"}, {"_id": "1", "integer": 1, "string": "1"}, {"_id": "2", "integer": 2, "string": "2"} ] // reply from CouchDB } [ {"id":"0","rev":"1-62657917"}, {"id":"1","rev":"1-2089673485"}, {"id":"2","rev":"1-2063452834"} ]
GOTCHA’s! Version storage is not guaranteed! Do not use this as a VCS! POST to /db/_compact deletes all older vesions To “roll back a transaction” you must: Retrieve all related records, cache these Insert any updates to records. On failure, use the returned revision numbers to re-insert the older record as a new one
Our Example Problem
Our Example Problem Hello world? Blog? Twitter clone?
Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead
Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 DBSOURCE REFSEQ: accession NC_012920.1 KEYWORDS . SOURCE mitochondrion Homo sapiens (human) ORGANISM Homo sapiens Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo.
Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 FEATURES DBSOURCE REFSEQ: accession NC_012920.1 Location/Qualifiers KEYWORDS . source 1..227 SOURCE /organism="Homo sapiens" mitochondrion Homo sapiens (human) ORGANISM Homo sapiens /organelle="mitochondrion" /isolation_source="caucasian" Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; /db_xref="taxon:9606" Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo./tissue_type="placenta" /country="United Kingdom: Great Britain" /note="this is the rCRS" Protein 1..227 /product="cytochrome c oxidase subunit II" /calculated_mol_wt=25434 http://www.ncbi.nlm.nih.gov/
Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 FEATURES DBSOURCE REFSEQ: accession NC_012920.1 Location/Qualifiers KEYWORDS . source 1..227 SOURCE /organism="Homo sapiens" mitochondrion Homo sapiens (human) ORGANISM Homo sapiens /organelle="mitochondrion" /isolation_source="caucasian" Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; /db_xref="taxon:9606" Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo./tissue_type="placenta" /country="United Kingdom: Great Britain" /note="this is the rCRS" Protein 1..227 /product="cytochrome c oxidase subunit II" /calculated_mol_wt=25434 http://www.ncbi.nlm.nih.gov/
Futon : A Couchapp
Futon : A Couchapp
Futon : A Couchapp
Futon : A Couchapp This one is going to be a bit tougher
Design Documents
Design Documents The key to using CouchDB as more than a key-value store
Design Documents The key to using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB
Design Documents The key to using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB Contain sections for map-reduce views, data validation, alternate formatting, ... Also library code & data structures specific to the design document
Soy Map!
Soy Map! Views use a Map-Reduce model for indexing and defining “virtual” documents Fits well with assumptions of self-sufficient documents and eventual consistency
Soy Map! Views use a Map-Reduce model for indexing and defining “virtual” documents Fits well with assumptions of self-sufficient documents and eventual consistency Map function is applied to all documents in the database Emits (parts of) documents that pass mustard Indexing is incremental after an initial definition You can choose to defer an index update for insert speed
Map function example
Map function example
Complex Map
Complex Map
View Result
View Result
GET by the indexed key
GET by the indexed key GET /refseq_human/_design/gb/_view/dbXref?key="GeneID:10" {"total_rows":7,"offset":2,"rows":[ {"id":"NP_000006", "key":"GeneID:10", "value":"NP_000006"} ]}
Reduce functions Optional and used in concert with a specific map function Great for summarizing or collating numerical data points E.g. counts, number of over time X, average load, probability of conversion Not really applicable to our example, so we’ll not cover it today
Show me the ... HTML? JSON is great, but what about, ya know, something useful? You can make a separate app to reformat the JSON OR you can use the “shows” section of a _design document. Rich formating possible with functions, templates, and special include macros
FASTA format “shows” : { “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ...
FASTA format “shows” : { “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ... GET /refseq_human/_design/gb/_show/fasta/NP_000006 >NP_000006 MDIEAYFERIGYKNSRNKLDLETLTDILEHQIRAVPFENLNMHCGQAMELGLEAIFDHIVRR NRGGWCLQVNQLLYWALTTIGFQTTMLGGYFYIPPVNKYSTGMVHLLLQVTIDGRNYIV DAGSGSSSQMWQPLELISGKDQPQVPCIFCLTEERGIWYLDQIRREQYITNKEFLNSHLLPK KKHQKIYLFTLEPRTIEDFESMNTYLQTSPTSSFITTSFCSLQTPEGVYCLVGFILTYRKFNYKD NTDLVEFKTLTEEEVEEVLRNIFKISLGRNLVPKPGDGSLTI
Backups & Replication
Backups & Replication Backup: simply copy the database file
Backups & Replication Backup: simply copy the database file Replicate: send a POST request with a source and target database
Backups & Replication Backup: simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL)
Backups & Replication Backup: simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notification API
Backups & Replication Backup: simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notification API $ curl -X POST http://localhost:5984/_replicate -d '{"source":"db", "target":"db-replica", "continuous":true}'
Data normalization? Schema? Foreign Keys? Column Constraints?
Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die”
Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts”
Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available
Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available Validates a record on update with a JS function
Required Fields function(newDoc, oldDoc, userCtx) { function require(field, message) { message = message || "Document must have a " + field; if (!newDoc[field]) throw({forbidden : message}); }; if (newDoc.type == "blogPost") { require("title"); require("created_at"); require("body"); Convention alert! require("author"); } ... }
Thank You! Learn http://couchdb.apache.org/ http://books.couchdb.org/relax http://wiki.apache.org/couchdb/ Awesome posts by community http://planet.couchdb.org Development Libraries http://github.com/jchris/couchrest http://github.com/couchapp/couchapp

More Related Content

PDF
Apache CouchDB talk at Ontario GNU Linux Fest
byMyles Braithwaite
 
KEY
An introduction to CouchDB
byDavid Coallier
 
PDF
CouchDB
bycodebits
 
PDF
Dealing with Azure Cosmos DB
byMihail Mateev
 
PDF
MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data
byMongoDB
 
PDF
OrientDB
byaemadrid
 
PDF
CouchDB Mobile - From Couch to 5K in 1 Hour
byPeter Friese
 
ODP
Introducing CouchDB
byHatem Ben Yacoub
 
Apache CouchDB talk at Ontario GNU Linux Fest
byMyles Braithwaite
 
An introduction to CouchDB
byDavid Coallier
 
CouchDB
bycodebits
 
Dealing with Azure Cosmos DB
byMihail Mateev
 
MongoDB .local Toronto 2019: Using Change Streams to Keep Up with Your Data
byMongoDB
 
OrientDB
byaemadrid
 
CouchDB Mobile - From Couch to 5K in 1 Hour
byPeter Friese
 
Introducing CouchDB
byHatem Ben Yacoub
 

What's hot

PDF
wtf is in Java/JDK/wtf7?
byScott Leberknight
 
PDF
Elasticsearch for SQL Users
byAll Things Open
 
PDF
Elasticsearch und die Java-Welt
byFlorian Hopf
 
PDF
아파트 정보를 이용한 ELK stack 활용 - 오근문
byNAVER D2
 
PDF
Mito, a successor of Integral
byfukamachi
 
PPTX
Getting Started with MongoDB and NodeJS
byMongoDB
 
PPTX
Elasticsearch 설치 및 기본 활용
by종민 김
 
PPTX
Unqlite
byPaul Myeongchan Kim
 
PDF
NoSQL and JavaScript: a Love Story
byAlexandre Morgaut
 
PDF
10 Key MongoDB Performance Indicators
byiammutex
 
PDF
Map/Confused? A practical approach to Map/Reduce with MongoDB
byUwe Printz
 
PDF
10gen Presents Schema Design and Data Modeling
byDATAVERSITY
 
ODP
Polyglot persistence with Spring Data
byCorneil du Plessis
 
PDF
[NodeConf.eu 2014] Scaling AB Testing on Netflix.com with Node.js
byAlex Liu
 
PDF
Beyond Cookies, Persistent Storage For Web Applications Web Directions North ...
byBradNeuberg
 
PDF
What do you mean, Backwards Compatibility?
byTrisha Gee
 
PDF
Replacing Oracle with MongoDB for a templating application at the Bavarian go...
byComsysto Reply GmbH
 
PPTX
Building .NET Apps using Couchbase Lite
bygramana
 
PPTX
Solr vs. Elasticsearch - Case by Case
byAlexandre Rafalovitch
 
PDF
EWD 3 Training Course Part 21: Persistent JavaScript Objects
byRob Tweed
 
wtf is in Java/JDK/wtf7?
byScott Leberknight
 
Elasticsearch for SQL Users
byAll Things Open
 
Elasticsearch und die Java-Welt
byFlorian Hopf
 
아파트 정보를 이용한 ELK stack 활용 - 오근문
byNAVER D2
 
Mito, a successor of Integral
byfukamachi
 
Getting Started with MongoDB and NodeJS
byMongoDB
 
Elasticsearch 설치 및 기본 활용
by종민 김
 
Unqlite
byPaul Myeongchan Kim
 
NoSQL and JavaScript: a Love Story
byAlexandre Morgaut
 
10 Key MongoDB Performance Indicators
byiammutex
 
Map/Confused? A practical approach to Map/Reduce with MongoDB
byUwe Printz
 
10gen Presents Schema Design and Data Modeling
byDATAVERSITY
 
Polyglot persistence with Spring Data
byCorneil du Plessis
 
[NodeConf.eu 2014] Scaling AB Testing on Netflix.com with Node.js
byAlex Liu
 
Beyond Cookies, Persistent Storage For Web Applications Web Directions North ...
byBradNeuberg
 
What do you mean, Backwards Compatibility?
byTrisha Gee
 
Replacing Oracle with MongoDB for a templating application at the Bavarian go...
byComsysto Reply GmbH
 
Building .NET Apps using Couchbase Lite
bygramana
 
Solr vs. Elasticsearch - Case by Case
byAlexandre Rafalovitch
 
EWD 3 Training Course Part 21: Persistent JavaScript Objects
byRob Tweed
 

Similar to Couchdb: No SQL? No driver? No problem

KEY
CouchDB : More Couch
bydelagoya
 
PDF
Python-CouchDB Training at PyCon PL 2012
byStefan Kögl
 
ODP
Intro to CouchDB
bysbisbee
 
PPT
No sql Database
bymymail2ashok
 
PPTX
Cluster of unreliable commodity hardware (couchdb) (2)
byNamitha Acharya
 
PPTX
Couch DB
byNamitha Acharya
 
PDF
CouchDB Talk JChris NYC
byChris Anderson
 
PDF
CouchDB Open Source Bridge
byChris Anderson
 
PDF
Introduction to CouchDB
byBogdan Sabău
 
PDF
Couchdb Nosql
byelliando dias
 
KEY
OSCON 2011 Learning CouchDB
byBradley Holt
 
PPTX
NOSQL, TYPES, Couch DB, MongoDB , Couch DB VS MongoDB
bymaaithilisaravanan
 
KEY
NoSQL: Why, When, and How
byBigBlueHat
 
PDF
CouchDB
byRashmi Agale
 
PDF
Couch db
byarunamore
 
PDF
CouchDB Google
bySteve Souders
 
PDF
CouchDB - Local Web Platform
byChris Anderson
 
PPT
Couch db
byRashmi Agale
 
PDF
Introduction to CouchDB - LA Hacker News
byMichael Parker
 
PPTX
Couch db
byamini gazar
 
CouchDB : More Couch
bydelagoya
 
Python-CouchDB Training at PyCon PL 2012
byStefan Kögl
 
Intro to CouchDB
bysbisbee
 
No sql Database
bymymail2ashok
 
Cluster of unreliable commodity hardware (couchdb) (2)
byNamitha Acharya
 
Couch DB
byNamitha Acharya
 
CouchDB Talk JChris NYC
byChris Anderson
 
CouchDB Open Source Bridge
byChris Anderson
 
Introduction to CouchDB
byBogdan Sabău
 
Couchdb Nosql
byelliando dias
 
OSCON 2011 Learning CouchDB
byBradley Holt
 
NOSQL, TYPES, Couch DB, MongoDB , Couch DB VS MongoDB
bymaaithilisaravanan
 
NoSQL: Why, When, and How
byBigBlueHat
 
CouchDB
byRashmi Agale
 
Couch db
byarunamore
 
CouchDB Google
bySteve Souders
 
CouchDB - Local Web Platform
byChris Anderson
 
Couch db
byRashmi Agale
 
Introduction to CouchDB - LA Hacker News
byMichael Parker
 
Couch db
byamini gazar
 

More from delagoya

PDF
Nyc big datagenomics-pizarroa-sept2017
bydelagoya
 
PDF
Machine Learning on the Cloud with Apache MXNet
bydelagoya
 
KEY
Ruby FFI
bydelagoya
 
KEY
padrino_and_sequel
bydelagoya
 
PPTX
Itmat pcbi-r-course-1
bydelagoya
 
PPTX
Everything comes in 3's
bydelagoya
 
Nyc big datagenomics-pizarroa-sept2017
bydelagoya
 
Machine Learning on the Cloud with Apache MXNet
bydelagoya
 
Ruby FFI
bydelagoya
 
padrino_and_sequel
bydelagoya
 
Itmat pcbi-r-course-1
bydelagoya
 
Everything comes in 3's
bydelagoya
 

Recently uploaded

PDF
Meta and Apple close to settling EU cases.pdf
byLUMINATIVE MEDIA/PROJECT COUNSEL MEDIA GROUP
 
PDF
Using Copilot with Microsoft Office Apps
byDeb Walther
 
PDF
Beyond Generative AI: Creating Demand for Compute in the 2030s
byReidar Wasenius
 
PDF
The invisible key: Securing the new attack vector of OAuth tokens
byGianluca Varisco
 
PDF
UnityNet Digital Sovereignty Checklist 2025-10-13
byLHelferty
 
PPTX
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
PDF
NSSHOEU-J 4x35mm² Cable Specification.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PDF
Unlocking Full-Stack Observability for AIOps: A Precisely Ironstream for Big ...
byPrecisely
 
PDF
Data Structure - 12 Graph
byAndiNurkholis1
 
PPTX
UiPath Automation Suite Installation (Hands-On) [2/3]
bysuhanisingh58689
 
PDF
AGV PROTOCOL BRAND KIT COMPLETE VIEW.pdf
byfagbolade
 
PDF
Rivian's Push Notification Sub Stream with Mega Filter by Marcus Kim & Saahil...
byScyllaDB
 
PDF
KV Caching Strategies for Latency-Critical LLM Applications by John Thomson
byScyllaDB
 
PDF
Application Monitoring and Observability: Elastic Stack Solution for Producti...
byPattabhi Amperayani
 
PDF
Ethical Initiatives in AI and accountability.pdf
byShiwani Gupta
 
PDF
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
PDF
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 
PDF
Expert Python App Development Company for Modern Enterprises
bySynapseIndia
 
PPTX
WUG-DMV: Workfront Wizards: Tips & Tricks Exchange (Sept 2025)
byWUG-DC MARYLAND VIRGINIA
 
PPTX
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 
Meta and Apple close to settling EU cases.pdf
byLUMINATIVE MEDIA/PROJECT COUNSEL MEDIA GROUP
 
Using Copilot with Microsoft Office Apps
byDeb Walther
 
Beyond Generative AI: Creating Demand for Compute in the 2030s
byReidar Wasenius
 
The invisible key: Securing the new attack vector of OAuth tokens
byGianluca Varisco
 
UnityNet Digital Sovereignty Checklist 2025-10-13
byLHelferty
 
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
NSSHOEU-J 4x35mm² Cable Specification.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
Unlocking Full-Stack Observability for AIOps: A Precisely Ironstream for Big ...
byPrecisely
 
Data Structure - 12 Graph
byAndiNurkholis1
 
UiPath Automation Suite Installation (Hands-On) [2/3]
bysuhanisingh58689
 
AGV PROTOCOL BRAND KIT COMPLETE VIEW.pdf
byfagbolade
 
Rivian's Push Notification Sub Stream with Mega Filter by Marcus Kim & Saahil...
byScyllaDB
 
KV Caching Strategies for Latency-Critical LLM Applications by John Thomson
byScyllaDB
 
Application Monitoring and Observability: Elastic Stack Solution for Producti...
byPattabhi Amperayani
 
Ethical Initiatives in AI and accountability.pdf
byShiwani Gupta
 
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 
Expert Python App Development Company for Modern Enterprises
bySynapseIndia
 
WUG-DMV: Workfront Wizards: Tips & Tricks Exchange (Sept 2025)
byWUG-DC MARYLAND VIRGINIA
 
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 

Couchdb: No SQL? No driver? No problem

  • 1.
    * CouchDB No SQL? No Driver? No problem. Angel Pizarro angel@upenn.edu * www.bauwel-movement.co.uk/ sculpture.php
  • 2.
    About Me Me: CBILalumni! Work in mass spec proteomics Lots of data in lots of formats in bioinformatics Ruby for programming and Ruby on Rails for Web apps But that doesn’t matter for CouchDB! Interested in CouchDB for AWS deployment
  • 3.
    Overview Talk about Key-Valuestores Introduce some general theory and concepts CouchDB specifics Example problem More CouchDB specifics Questions?
  • 4.
    Key-Value Databases Datastore ofvalues indexed by keys (duh!) Hash or B-Tree index for keys Cassandra Hash is FAST, but only allows single-value lookups B-Tree is slower, but allows range queries Horizontally scalable - via key partitioning
  • 5.
    The CAP theory: applies when business logic is separate from storage Consistency vs. Availability vs. Partition tolerance RDBMS = enforced consistency PAXOS = quorum consistency CouchDB (and others) = eventual consistency and horizontally scalable http://www.julianbrowne.com/article/viewer/brewers-cap-theorem
  • 6.
  • 7.
  • 8.
    CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca * Hovercraft: http://github.com/jchris/hovercraft/
  • 9.
    CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis * Hovercraft: http://github.com/jchris/hovercraft/
  • 10.
    CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views * Hovercraft: http://github.com/jchris/hovercraft/
  • 11.
    CouchDB Document Oriented Database JSON documents HTTP protocol using REST operations No direct native language drivers * Javascript is the lingua franca ACID & MVCC guarantees on a per- document basis Map-Reduce indexing and views Back-ups and replication are easy-peasy * Hovercraft: http://github.com/jchris/hovercraft/
  • 12.
  • 13.
    Javascript Object Notation * * http://www.json.org
  • 14.
    Example JSON { “name”: “J. Doe”, “friends”: 0, “traits”: [“nice”, “outgoing”] }
  • 15.
  • 16.
  • 17.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP)
  • 18.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies
  • 19.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request
  • 20.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such
  • 21.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs
  • 22.
    REST Representational State Transfer Clients-Serverseparation with uniform interface (HTTP) Load-balancing, caching, authorization & authentication, proxies Stateless - client is responsible for creating a self- sufficient request Resources are cacheable - servers must mark non-cacheable resources as such Only 5 HTTP verbs GET, PUT, POST, DELETE, HEAD
  • 23.
    CouchDB REST/CRUD GET read PUT create or update DELETE delete something POST bulk operations
  • 24.
    CouchDB passes the ACID test Each document is completely self-sufficient Each document has a version number An update operation writes a complete new copy of the the record and is assigned the new version number Append-only file structure allows the write to occur while still serving read requests
  • 25.
    MVCC RDBMS CouchDB Multi-Version Concurrency Control RDBMS enforces consistency using read/write locks Instead of locks, CouchDB just serve up old data Multi-document (mutli-row) transactional semantics must be handled by the application
  • 26.
    Database API Create aDB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
  • 27.
    Database API Create aDB: Protocol $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
  • 28.
    Database API Create aDB: CouchDB server $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
  • 29.
    Database API Create aDB: DB name $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true}
  • 30.
    Database API Create aDB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"}
  • 31.
    Database API Create aDB: $ curl -X PUT http://127.0.0.1:5984/friendbook {"ok":true} Try it Again: {"error":"db_exists"} Not recoverable! Delete a DB: $ curl -X DELETE http://localhost:5984/friendbook {"ok":true}
  • 32.
    Inserting a document Allinsert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]}
  • 33.
    Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}
  • 34.
    Inserting a document All insert require that you give a unique ID. You can request one from CouchDB: $ curl -X GET http://localhost:5984/_uuids {"uuids":["d1dde0996a4db7c1ebc78fb89c01b9e6"]} We’ll just give one: $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @j_doe.json Read a JSON file {"ok":true, "id":"j_doe", "rev":"1-062af1c4ac73287b7e07396c86243432"}
  • 35.
    Full JSON document Before {"name": "J. Doe", "friends": 0 } After { "_id": "j_doe", "_rev": "1-062af1c4ac73287b7e07396c86243432", "name": "J. Doe", "friends": 0 }
  • 36.
    Updating a document $curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conflict","reason":"Document update conflict."}
  • 37.
    Updating a document $curl -X PUT http://localhost:5984/friendbook/j_doe -d '{"name": "J. Doe", "friends": 1 }' {"error":"conflict","reason":"Document update conflict."} Must give _rev (revision number) for updates! revised.json { "_rev":"1-062af1c4ac73287b7e07396c86243432", "name":"J. Doe", "friends": 1 } $ curl -X PUT http://localhost:5984/friendbook/j_doe -d @revised.json {"ok":true,"id":"j_doe","rev":"2-0629239b53a8d146a3a3c4c63e 2dbfd0"}
  • 38.
    Deleting a document $curl -X DELETE http://localhost:5984/friendbook/j_doe {"error":"conflict","reason":"Document update conflict."} Must give revision number for deletes! $ curl -X DELETE http://localhost:5984/friendbook/j_doe? rev=2-0629239b53a8d146a3a3c4c63e2dbfd0 {"ok":true,"id":"j_doe", "rev":"3-57673a4b7b662bb916cc374a92318c6b"} Returns a revision number for the delete $ curl -X GET http://localhost:5984/friendbook/j_doe {"error":"not_found","reason":"deleted"}
  • 39.
    Bulk operation POST /database/_bulk_docs with a JSON document containing all of the new or updated documents. // documents to bulk upload { "docs": [ {"_id": "0", "integer": 0, "string": "0"}, {"_id": "1", "integer": 1, "string": "1"}, {"_id": "2", "integer": 2, "string": "2"} ] // reply from CouchDB } [ {"id":"0","rev":"1-62657917"}, {"id":"1","rev":"1-2089673485"}, {"id":"2","rev":"1-2063452834"} ]
  • 40.
    GOTCHA’s! Version storage isnot guaranteed! Do not use this as a VCS! POST to /db/_compact deletes all older vesions To “roll back a transaction” you must: Retrieve all related records, cache these Insert any updates to records. On failure, use the returned revision numbers to re-insert the older record as a new one
  • 41.
  • 42.
    Our Example Problem Hello world? Blog? Twitter clone?
  • 43.
    Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead
  • 44.
    Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 DBSOURCE REFSEQ: accession NC_012920.1 KEYWORDS . SOURCE mitochondrion Homo sapiens (human) ORGANISM Homo sapiens Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo.
  • 45.
    Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 FEATURES DBSOURCE REFSEQ: accession NC_012920.1 Location/Qualifiers KEYWORDS . source 1..227 SOURCE /organism="Homo sapiens" mitochondrion Homo sapiens (human) ORGANISM Homo sapiens /organelle="mitochondrion" /isolation_source="caucasian" Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; /db_xref="taxon:9606" Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo./tissue_type="placenta" /country="United Kingdom: Great Britain" /note="this is the rCRS" Protein 1..227 /product="cytochrome c oxidase subunit II" /calculated_mol_wt=25434 http://www.ncbi.nlm.nih.gov/
  • 46.
    Our Example Problem Hello world? Blog? Twitter clone? Let’s store all human proteins instead LOCUS YP_003024029 227 aa linear PRI 09-JUL-2009 DEFINITION cytochrome c oxidase subunit II [Homo sapiens]. ACCESSION YP_003024029 VERSION YP_003024029.1 GI:251831110 DBLINK Project:30353 FEATURES DBSOURCE REFSEQ: accession NC_012920.1 Location/Qualifiers KEYWORDS . source 1..227 SOURCE /organism="Homo sapiens" mitochondrion Homo sapiens (human) ORGANISM Homo sapiens /organelle="mitochondrion" /isolation_source="caucasian" Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; /db_xref="taxon:9606" Mammalia; Eutheria; Euarchontoglires; Primates; Haplorrhini; Catarrhini; Hominidae; Homo./tissue_type="placenta" /country="United Kingdom: Great Britain" /note="this is the rCRS" Protein 1..227 /product="cytochrome c oxidase subunit II" /calculated_mol_wt=25434 http://www.ncbi.nlm.nih.gov/
  • 47.
    Futon : ACouchapp
  • 48.
    Futon : ACouchapp
  • 49.
    Futon : ACouchapp
  • 50.
    Futon : ACouchapp This one is going to be a bit tougher
  • 51.
  • 52.
    Design Documents The keyto using CouchDB as more than a key-value store
  • 53.
    Design Documents The keyto using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB
  • 54.
    Design Documents The keyto using CouchDB as more than a key-value store Just another JSON document, but contain javascript functions that CouchDB treats as application code Functions are executed within CouchDB Contain sections for map-reduce views, data validation, alternate formatting, ... Also library code & data structures specific to the design document
  • 55.
  • 56.
    Soy Map! Views usea Map-Reduce model for indexing and defining “virtual” documents Fits well with assumptions of self-sufficient documents and eventual consistency
  • 57.
    Soy Map! Views usea Map-Reduce model for indexing and defining “virtual” documents Fits well with assumptions of self-sufficient documents and eventual consistency Map function is applied to all documents in the database Emits (parts of) documents that pass mustard Indexing is incremental after an initial definition You can choose to defer an index update for insert speed
  • 58.
  • 59.
  • 60.
  • 61.
  • 62.
  • 63.
  • 64.
    GET by theindexed key
  • 65.
    GET by theindexed key GET /refseq_human/_design/gb/_view/dbXref?key="GeneID:10" {"total_rows":7,"offset":2,"rows":[ {"id":"NP_000006", "key":"GeneID:10", "value":"NP_000006"} ]}
  • 66.
    Reduce functions Optional andused in concert with a specific map function Great for summarizing or collating numerical data points E.g. counts, number of over time X, average load, probability of conversion Not really applicable to our example, so we’ll not cover it today
  • 67.
    Show me the... HTML? JSON is great, but what about, ya know, something useful? You can make a separate app to reformat the JSON OR you can use the “shows” section of a _design document. Rich formating possible with functions, templates, and special include macros
  • 68.
    FASTA format “shows” :{ “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ...
  • 69.
    FASTA format “shows” :{ “fasta” : “function(doc, req) { return ‘>’ + doc._id + ‘n’ + doc.seq; }”, ... GET /refseq_human/_design/gb/_show/fasta/NP_000006 >NP_000006 MDIEAYFERIGYKNSRNKLDLETLTDILEHQIRAVPFENLNMHCGQAMELGLEAIFDHIVRR NRGGWCLQVNQLLYWALTTIGFQTTMLGGYFYIPPVNKYSTGMVHLLLQVTIDGRNYIV DAGSGSSSQMWQPLELISGKDQPQVPCIFCLTEERGIWYLDQIRREQYITNKEFLNSHLLPK KKHQKIYLFTLEPRTIEDFESMNTYLQTSPTSSFITTSFCSLQTPEGVYCLVGFILTYRKFNYKD NTDLVEFKTLTEEEVEEVLRNIFKISLGRNLVPKPGDGSLTI
  • 70.
  • 71.
    Backups & Replication Backup:simply copy the database file
  • 72.
    Backups & Replication Backup:simply copy the database file Replicate: send a POST request with a source and target database
  • 73.
    Backups & Replication Backup:simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL)
  • 74.
    Backups & Replication Backup:simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notification API
  • 75.
    Backups & Replication Backup: simply copy the database file Replicate: send a POST request with a source and target database Source and target DB’s can either be local (just the db name) or remote (full URL) “continous”: true option will register the target to the source’s _changes notification API $ curl -X POST http://localhost:5984/_replicate -d '{"source":"db", "target":"db-replica", "continuous":true}'
  • 76.
    Data normalization? Schema? Foreign Keys? Column Constraints?
  • 77.
    Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die”
  • 78.
    Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts”
  • 79.
    Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available
  • 80.
    Data normalization? Schema? Foreign Keys? Column Constraints? forgetaboutit Italian for “forget about it” … “or die” Denormalize “until it hurts” But there are validations are available Validates a record on update with a JS function
  • 81.
    Required Fields function(newDoc, oldDoc,userCtx) { function require(field, message) { message = message || "Document must have a " + field; if (!newDoc[field]) throw({forbidden : message}); }; if (newDoc.type == "blogPost") { require("title"); require("created_at"); require("body"); Convention alert! require("author"); } ... }
  • 82.
    Thank You! Learn http://couchdb.apache.org/ http://books.couchdb.org/relax http://wiki.apache.org/couchdb/ Awesome postsby community http://planet.couchdb.org Development Libraries http://github.com/jchris/couchrest http://github.com/couchapp/couchapp

Editor's Notes

  • #5 - If the key is a DateTime, then B-tree is a much better choice
  • #6 Brewer’s CAP Theorem http://www.julianbrowne.com/article/viewer/brewers-cap-theorem Partition tolerance encompasses both business logic and data partitioning. PAXOS will override more recent updates to a disconnected resource if it did not vote on a previous transaction.
  • #7 Highlighted words covered later in order that they appear
  • #8 Highlighted words covered later in order that they appear
  • #9 Highlighted words covered later in order that they appear
  • #10 Highlighted words covered later in order that they appear
  • #11 Highlighted words covered later in order that they appear
  • #12 Highlighted words covered later in order that they appear
  • #16 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #17 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #18 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #19 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #20 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #21 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #22 Other stuff, but this is the most relevant for the discussion Older browsers only support green verbs
  • #23 CRUD = Create Read Update Delete
  • #25 Next is the API discussions
  • #35 You can give a “count” parameter to UUID function: $ curl -X GET http://localhost:5984/_uuids?count=10
  • #36 You can give a “count” parameter to UUID function: $ curl -X GET http://localhost:5984/_uuids?count=10
  • #40 Can give it as an URL parameter or in the E-Tag HTTP header. You cannot delete a specific revision! The revision number is only there so that the server can definitively say you are talking about the most recent record. You need delete rev for replication of delete operations on other servers that are being synced to this one.
  • #42 Might also be able to delete a particualr version. Will have to check that.
  • #49 Note: I could’ve made GI a number, but did not in this case Zipcodes would be a bad thing to turn into numbers, b/c of possible leading zeros
  • #50 Note: I could’ve made GI a number, but did not in this case Zipcodes would be a bad thing to turn into numbers, b/c of possible leading zeros
  • #51 Note: I could’ve made GI a number, but did not in this case Zipcodes would be a bad thing to turn into numbers, b/c of possible leading zeros
  • #52 Best practice = One design document per application or set of requirements Next: Map-Reduce Views
  • #53 Best practice = One design document per application or set of requirements Next: Map-Reduce Views
  • #54 Best practice = One design document per application or set of requirements Next: Map-Reduce Views
  • #64 We are just going to take a look at a simple plain text example of FASTA file
  • #66 Append-only file structure ensures that your DB is always valid, even during mid-write server failures.
  • #67 Append-only file structure ensures that your DB is always valid, even during mid-write server failures.
  • #68 Append-only file structure ensures that your DB is always valid, even during mid-write server failures.
  • #69 Append-only file structure ensures that your DB is always valid, even during mid-write server failures.
  • #70 Append-only file structure ensures that your DB is always valid, even during mid-write server failures.