Downloaded 269 times
More Related Content
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Cassandra Codebase 2011
- 1. Codebase 2011Getting toknow the codebaseGary Dusbabek@gdusbabek
- 2.
- 3.
- 4.
- 5. How to Contributehttp://wiki.apache.org/cassandra/HowToContributeJIRA:“lhf” label (Low hanging fruit)Scratch your itch
- 6. How to ContributeRunthe testsant testnosetests test/system/test_thrift_server.py
- 7. How to Contributehttp://wiki.apache.org/cassandra/CodeStyleAvoid:Reformattingwhite spaceRenaming things everywhereUnrelated changes
- 8. How to ContributeUsegitAttach patchesgit format-patch as jira attachments.Group them sensibly
- 9. How to ContributeSomeonewill review your codeUsually a committerPersistence helpsDon’t get your feelings hurtIt usually takes a few rounds
- 10. How to ContributeParticipate!#cassandra-devon freenodedev@cassandra.apache.org
- 11.
- 12. ServicesRing Operations (StorageService)StorageOperations (StorageProxy)
- 13. Startup Sequencebin/cassandraFinds cassandra.in.sh$CLASSPATH(mandatory)$CASSANDRA_HOME$CASSANDRA_CONF (mandatory)Executes $CASSANDRA_CONF/cassandra-env.shSets heap sizes (gc tuning goes here!)
- 14.
- 15. AbstractCassandraDaemonACD.setup():Reads configuration: DatabaseDescriptorLoadsschema: DD.loadSchemas()Scrub directoriesInitialize storage (keyspaces + CFs)Commit log recovery: CL.recover()StorageService.initServer() -> StorageService.joinTokenRing()
- 16. Attn Tinkerers!Abstracted initializationof transport.Handy if you’re experimenting with transports/RPCJust extend AbstractCassandraDaemon and make sure that class is started up via bin/cassandra.
- 17. o.a.c.thrift.CassandraServerImplements thrift interfacemethods (the API).Start here when trying to understand the read/write path and RPC.
- 18. ConfigurationDatabaseDescriptorSide-effect of ACD.setup()Readsconfig settings from yamlDefines system tablesChanges regularlyI hate this code. Please fix it.
- 19.
- 20. Did you justsay ‘Singletons?’
- 21.
- 22. JMX MBeansTooling suppliedby MbeansAnything that does measureable/configurable work is tooledThread poolsCompactionHinted handoffStreamingStorageCommit log
- 23. StorageServiceinitServer() -> joinTokenRing()StartsgossipStarts MessagingServiceNegotiates bootstrapMany ring operations live here.Repository of ring topologyTokenMetadata (quasi-singleton via SS.tokenMetadata_)Partitioner instance is also here
- 24. MessagingServiceVerb handlers livehere (initialized from SS).Main event handlers, haven’t changed much.Socket listener2 threads per ring nodeMessage gatewayemitted from MessageProducerimplsMS.sendRR()MS.sendOneWay()MS.receive()Messages are versioned now (0.8)IncomingTCPConnection
- 25. StorageProxyTop level ofall read/write operationsCalled from o.a.c.thrift.CassandraServerWrite path changed because of countersNotion of WritePerformerEventually to Table and ColumnFamilyStoreFurther, to SSTable and related classes.
- 26. StageManagerFancy java ThreadPoolExecutorSEDA: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdfconsumes callables from a queue.Manages concurrency.Hasn’t changed much.
- 27. Adding API MethodsDefinemethod+structures in IDLinterface/cassandra.thriftRegenerate filesant gen-thrift-java gen-thrift-pyImplement stubs:o.a.c.thrift.CassandraServerCreate a system testtests/system/test_thrift_server.py
- 28. ReadingSocket->CassandraServerPermissionsRequest validationMarshallingReadCommands createdin CS.multigetSliceInternal, passed to StorageProxy1 per key
- 29. ReadingStorageProxy.read(), fetchRows()For eachReadCommandDetermine endpointsLocal & remote branches
- 30. ReadingStorageProxy localREAD stageexecutes a LocalReadRunnableTrue read vs digestTable, ColumnFamilyStoreCFS.getTopLevelColumnsMake QueryFilterQuery MemtablesQuery SSTablesCoalesce in iterators
- 31. ReadingStorageProxy remoteread commandResponsehandlerSend to remote nodesRead repair happens in SP.fetchRows().
- 32. WritingCS.doInsert()Marshalling, creates RMsStorageProxylocal/remotebranchSP.sendToHintedEndpoints()RowMutationone Key per (several CFs)ColumnFamilyCollection of column modifications
- 33. WritingRM.apply->Table.applyWrite to CLIterateover RM CFsCFS.apply()Overwrites results on pre-existing column families
- 34. WritingRM is serializedinto a Message and sent to other nodesWaits for ACKs depending on CL
- 35.
- 36. ChallengesTo have anin-depth understanding of everything.Hard for hobbyist/part-timersOutside of Datastax, little support for full-timersStill changing fastKeeping up
- 37. Challenge: Lines ofCode0.4 (Sep 2009)52 kloc0.5 (Jan 2010)59 kloc0.6 (Apr 2010)73 kloc0.7 (Jan 2011)122 kloc0.8 (Jun 2011)146 klocTrunk (yesterday)149 klocAverage:4,500 lines per month
- 38. ChallengesCodewise Growing painsSoftwarematurityDecisions made early on