KEMBAR78
OpenSolaris Introduction | ODP
OPEN SOURCE CONFERENCE  2009 MSC Malaysia Satyajit Tripathi ISV-Engineering, Sun Microsystems
Audience Prospective Customer Independent Software Vendor (ISV) Chief Executive Officer (CXO, CTO) System Administrator Software Developer Research Scientist Students Rest Of The World NOTE :- The presentation is customized for the mixed audience at MSC OpenSource Conference 2009 Malaysia
OpenSolaris Operating System (OS) Code, based on industry leading Enterprise Solaris by Sun Microsystems Freely Shareable and Redistributable Thriving Community Extension of OpenSolaris technology Innovation and Collaboration Open Source project CDDL v1.0, OpenSolaris Binary lic. Existing license applicable to the respective community projects
Historic Events Solaris 2.0 : June 1992 Development started in 1988. Derived from Unix System V4 Solaris 2.6 : July 1997 Solaris 7 : November 1998 Solaris 8 : February 2000 Solaris 9 : May 2002 (SPARC), January 2003 (x86) Solaris 10 : January 2005 Solaris open sourced :  June 14 th  2005 SxDE First Release : February 2007 1 st  Release of OpenSolaris Distro : May 5 th  2008 2 nd  Release of OpenSolaris Distro : November 6 th  2008 3 rd  Release of OpenSolaris Distro : June 1 st  2009
Solaris Release Model
One Solaris Platform Enterprise class Unmatched level of support Longer release cycle (3-5 years) Web economy Latest Innovation and extensions Shorter release cycle (every 6 months)
OpenSolaris Inclusions Software Included Firefox Browser Thunderbird Email client Event Manager GIMP (Image Editor) Pidgin (Chat IM) Java technology Explore for more... Software Easy Access OpenOffice Apache Web Server GlassFish ES MySQL OpenDS NetBeans PHP, Ruby, Perl Explore for more...
OpenSolaris Distro Solaris Express Community Edition BeleniX Jaris MartUX mBE MilaX NexentaOS SchilliX Power Of OpenSolaris with usability of Linux Small size Live CD distribution
OpenSolaris Communities A Social Group to have Open Conversation Governance Process Roles, Responsibilities Endorse Technical Projects Start and Own Projects Propose New Community Groups Participate and Network
OpenSolaris Project Enhance and Improve OpenSolaris Build Community of Committed Members Multiple Opportunities to Contribution Contributing Packages Contributing Code Documentation Testing Participate and Build
Participation Is Key Everyone Has Something to Contribute Code Of Conduct Be Inclusive Be Respectful and Honest Be Considerate of Diversity
What It Means To Prospective Customer Low Cost Entrant Highly Reliable, Secure and  Robust Unmatched Support Widely Compatible Scalable and Highly Available Easily Maintainable
What It Means To ISV CXO/ CTO High Quality and Industry Standard Out-Of-Box Performance Platform Neutrality Access to Expert Advise Skill Availability Wide Compatibility Clear Roadmap
What It Means To System Administrator Easy Installation, Administration, Monitoring Cool Features & Tools ZFS and Auto backup Virtualization with Container FMA and SMF Observability with DTrace Live Upgrade with Minimum Downtime
What It Means To Software Developer Latest Innovations and Cutting Edge technology Ready Environment for Software Development High Productivity with Open Source Software Free Access to Knowledge base and Documentation Fruitful Career and Growth
What It Means To Research Scientist and Student Diverse Community to discuss and share Ideas Access to Open Source technologies & Knowledge Applicability to the Industry High Visibility & Outreach
What It Means To You ! Easy To Get Easy To Install Easy To Use Easy To Maintain Easy To Share
OpenSolaris Unleashing Potential
Useful Information OpenSolaris.Com Download  http://www.opensolaris.com/get Support Subscription  http://www.sun.com/service/opensolaris OpenSolaris.Org Community  http://www.opensolaris.org Packaging Repository  http://pkg.opensolaris.org Documentation  http://www.opensolaris.org/os/documentation Sun Partner Advantage Program  http://www.sun.com/isv Contact ISV-Engineering   < [email_address] >
Congratulations for Participating
Next Session OpenSolaris and Advanced Technologies
OpenSolaris Advanced Technologies ZFS (Zettabyte File System) SMF (Service Management Facility) Zones and Containers DTrace And more ...
Zettabyte File System (Zeta = 10 21 ) Better, Safer way to Manage Data First 128-bit File System Capacity 1 billion TB. Virtually unlimited. Fast and High Performance No Silent Data Corruption End-to-end integrity Aggressive 256-bit checksums Instantaneous Snapshots Almost Zero Overhead and Easy Rollback Application Compatibility Easy Administration. No Volume Manager Efficient Mirroring and Remote Replication
ZFS Virtually Unlimited Traditional Volumes Abstraction : Virtual disk Partition for each FS Grow or shrink manually Storage is fragmented, stranded ZFS Pooled Storage Abstraction : malloc, free No partition to manage Grow or shrink automatic Storage in the Pool is shared Volume FS Volume FS Volume FS ZFS Storage Pool ZFS ZFS
Transaction Group Batch I/O Schedule, Aggregate, and   Issue I/O at will Loss of Power. No Resync Runs at platter speed Object based Transaction Make these N# changes to   these M# objects Atomic (all-or-nothing) Transaction Group Commit Atomic for entire group Always consistent on disk No Journal. Not needed ZFS I/O Stack ZPL ZFS POSIX Layer SPA Storage Pool Allocator DMU Data Management Unit Block Device Interface “ Write this block,   then that block, ...” Loss of Power. Loss of    on-disk consistency Workaround : Journaling,   which is slow and complex Block Device Interface Write each block to each   disk immediately to keep   mirrors in sync Loss of Power. Resync. Synchronous and slow FS Volume
ZFS Dynamic Stripping BEFORE Writes : stripped across all 3 mirrors Reads : wherever data was written Block Allocation Policy Considers Capacity Performance (latency, BW) Health (degraded mirrors) AFTER Writes : stripped across all 4 mirrors Reads : wherever data was written No need to migrate existing data Old data stripped across 1-3 New data stripped across 1-4 Copy-On-Write (COW) gently reallocates old data ZFS Storage Pool ZFS ZFS 1 2 3 1 2 3 4 ZFS Storage Pool ZFS ZFS
ZFS Additional Capabilities Intelligent Prefetch Multiple Independent Pre-fetch Streams Very useful for Streaming Service Provider Automatic Length and Stride Detection Very useful for HPC Applications Variable Block Size Built-in Compression. Built-in Encryption Per-user and Per-group Quota Support Common Internet FS (CIFS) Support Integrated with Zones and FMA And more ...
Service Management Facility (SMF) Simplified Service Management Services are Objects easily managed with few simple commands Automated Restart of Failed Services SMF Monitors services and  Pro actively  Restart on Failure Persistent Service Configuration Service Definition and Configuration persist across reboot Explicit Dependencies Easier Debugging Faster Boot/Shutdown Processes Delegated Service Administration Administrative Role Delegation to non Root users
SMF Service Components SMF Service Manifest Defines Default Service Properties in the Repository System Manifest files under /var/svc/manifest. Imported at boot Service Start Method(s) Defines the Interaction between Service Restarter and the Service Service Executable(s) Executable called by Method(s) to Implement the Service Service Log Records the output of the Service. Useful in Debugging Failure /var/svc/log/milestone-multi-user-server:default.log Service Fault Management Resource Identifier (FMRI) Identifies specific Service Instance  Example : svc:/milestone/multi-user-server:default
Zones and Container Virtualization instrumented within OS Kernel Resource Management Controlled and Dynamic Resource Allocation Prioritize Applications Name space, Security, and Fault Isolation Light Weight and No Overhead Theoretical limit is 8192 Easy to Clone and Migrate Rapidly create and multiply OS environment Increase System Utilization Scales with OS Single Point Administration at Global Zone
Zone States Configured Configuration completed and committed Installed Packages installed successfully Ready Virtual Platform established Running Zone booted successfully and running Shutting Down Temporary state in the process of shutting down Down Temporary state completed shutting down, to go to Installed state
Zones Usefulness for Developers Virtual OS Environment Partitioning One large system can be utilized by multiple users non-intrusively Rapidly proliferate similar Development Environment Clone within the same System Migrate to the new System Simulate Distributed Application on multiple zones Apply latest OS patch Once on the Global Zone Available to all Zones Preserve the Pre-configured Development or Deployment Environment with simple commands
Branded Zone (BrandZ) BrandZ extends Solaris Zones Infrastructure Brand Attribute set at Zone Create time Branded Installation Routine for Arbitrary Collection of Software Pre/Post-boot scripts for boot-time setup and configuration Simple commands zoneadm and zonecfg Brand lx to run Linux binary application unmodified within a Zone with Solaris Kernel Brand lx runs on x86/x64 booted with 32/64-bit Kernel
Dynamic Tracing (D Trace) DTrace, A Comprehensive Framework for Solaris TM  Operating Environment Implement New DTrace Providers Implement fully Configurable DTrace Probes Implement New DTrace Consumers and Data Display Observability using DTrace Aggregate Arbitrary Behavior of the OS and User Programs Dynamically Enable and Manage Probes Dynamically Associate Predicates and Actions with Probes Dynamically Manage Trace Buffers and Probe Overhead Examine Live Production System or a Crash Dump
DTrace World Probes
D Language 'D' Language, like 'C' and constructs similar to awk Complete Access to Kernel C types Complete Access to Statics and Globals Rich built-in Variable set Complete Support for ANSI-C Operators Example : Trace the pid of every process named date with syscall open(2) #!/usr/sbin/dtrace -s  syscall::open:entry /execname == “date”/ { trace(pid); } Probe Name   o Entry point of open Predicate   o Process is named “date” Action   o Print the process ID
DTrace Probe Point of Instrumentation within OS Kernel Has a  Name Identifies the  Module  and  Function  it Instruments Accessible through  Provider 4 Attributes, Name, Module, Function & Provider, defines a tuple to uniquely identify a Probe   probe description (provider:module:function:name) / predicate / { action statements } Each Probe is Assigned an Integer Identifier
DTrace Provider Methodology for Instrumenting Probes Provides Probe Access to DTrace Framework Informed by DTrace to Enable a Probe Transfers Control of the Probe to DTrace when Enabled
DTrace Consumer Process that interacts with DTrace Concurrent Consumers unlimited. DTrace handles Multiplexing dtrace(1M) is DTrace Consumer. Generic front-end to the DTrace Facility
Write an Application Probe Define Provider provider myserv { probe query_receive(string, string); probe query_respond(); } Include sdt.h Add Probe code to the Application query = wait_for_new_query(); DTRACE_PROBE2(myserv, query_receive, query->clientname, query->msg); process_query(query) Compile the Application dtrace -G -32 -s myserv.d src1.o src2.o ... cc -o myserv myserv.o src1.o src2.o ...
Performance Tools Process Stats cputrack : per-processor hardware counter pargs : process arguments pflags : process flags pcred : process credentials pldd : process library dependency psig : process signal disposition pstack : process stack dump pmap : process memory map pfiles : open files and names prstat : process statistics ptree : process tree ptime : process micro-state times pwdx : process working directory Process Control pgrep  :  grep for processes pkill  :  kill processes list pstop  :  stop processes prun  :  start processes prctl  :  view/set process resources pwait  :  wait for process preap  :  reap a zombie process Process Tracing & Debugging abitrace : trace ABI interfaces dtrace : trace the world mdb : debug/control processes truss : trace functions,system calls Kernel Tracing & Debugging dtrace : trace and monitor kernel  lockstat : monitor locking statistics lockstat -k : profile kernel mdb : debug live and kernel cores System Stats acctcom : process accounting busstat : Bus hardware counters cpustat : CPU hardware counters iostat : IO & NFS statistics kstat : display kernel statistics mpstat : processor statistics netstat : network statistics nfsstat : nfs server stats sar : kitchen sink utility vmstat : virtual memory stats
Thank You! for Participating
Track back URL http://blogs.sun.com/stripathi

OpenSolaris Introduction

  • 1.
    OPEN SOURCE CONFERENCE 2009 MSC Malaysia Satyajit Tripathi ISV-Engineering, Sun Microsystems
  • 2.
    Audience Prospective CustomerIndependent Software Vendor (ISV) Chief Executive Officer (CXO, CTO) System Administrator Software Developer Research Scientist Students Rest Of The World NOTE :- The presentation is customized for the mixed audience at MSC OpenSource Conference 2009 Malaysia
  • 3.
    OpenSolaris Operating System(OS) Code, based on industry leading Enterprise Solaris by Sun Microsystems Freely Shareable and Redistributable Thriving Community Extension of OpenSolaris technology Innovation and Collaboration Open Source project CDDL v1.0, OpenSolaris Binary lic. Existing license applicable to the respective community projects
  • 4.
    Historic Events Solaris2.0 : June 1992 Development started in 1988. Derived from Unix System V4 Solaris 2.6 : July 1997 Solaris 7 : November 1998 Solaris 8 : February 2000 Solaris 9 : May 2002 (SPARC), January 2003 (x86) Solaris 10 : January 2005 Solaris open sourced : June 14 th 2005 SxDE First Release : February 2007 1 st Release of OpenSolaris Distro : May 5 th 2008 2 nd Release of OpenSolaris Distro : November 6 th 2008 3 rd Release of OpenSolaris Distro : June 1 st 2009
  • 5.
  • 6.
    One Solaris PlatformEnterprise class Unmatched level of support Longer release cycle (3-5 years) Web economy Latest Innovation and extensions Shorter release cycle (every 6 months)
  • 7.
    OpenSolaris Inclusions SoftwareIncluded Firefox Browser Thunderbird Email client Event Manager GIMP (Image Editor) Pidgin (Chat IM) Java technology Explore for more... Software Easy Access OpenOffice Apache Web Server GlassFish ES MySQL OpenDS NetBeans PHP, Ruby, Perl Explore for more...
  • 8.
    OpenSolaris Distro SolarisExpress Community Edition BeleniX Jaris MartUX mBE MilaX NexentaOS SchilliX Power Of OpenSolaris with usability of Linux Small size Live CD distribution
  • 9.
    OpenSolaris Communities ASocial Group to have Open Conversation Governance Process Roles, Responsibilities Endorse Technical Projects Start and Own Projects Propose New Community Groups Participate and Network
  • 10.
    OpenSolaris Project Enhanceand Improve OpenSolaris Build Community of Committed Members Multiple Opportunities to Contribution Contributing Packages Contributing Code Documentation Testing Participate and Build
  • 11.
    Participation Is KeyEveryone Has Something to Contribute Code Of Conduct Be Inclusive Be Respectful and Honest Be Considerate of Diversity
  • 12.
    What It MeansTo Prospective Customer Low Cost Entrant Highly Reliable, Secure and Robust Unmatched Support Widely Compatible Scalable and Highly Available Easily Maintainable
  • 13.
    What It MeansTo ISV CXO/ CTO High Quality and Industry Standard Out-Of-Box Performance Platform Neutrality Access to Expert Advise Skill Availability Wide Compatibility Clear Roadmap
  • 14.
    What It MeansTo System Administrator Easy Installation, Administration, Monitoring Cool Features & Tools ZFS and Auto backup Virtualization with Container FMA and SMF Observability with DTrace Live Upgrade with Minimum Downtime
  • 15.
    What It MeansTo Software Developer Latest Innovations and Cutting Edge technology Ready Environment for Software Development High Productivity with Open Source Software Free Access to Knowledge base and Documentation Fruitful Career and Growth
  • 16.
    What It MeansTo Research Scientist and Student Diverse Community to discuss and share Ideas Access to Open Source technologies & Knowledge Applicability to the Industry High Visibility & Outreach
  • 17.
    What It MeansTo You ! Easy To Get Easy To Install Easy To Use Easy To Maintain Easy To Share
  • 18.
  • 19.
    Useful Information OpenSolaris.ComDownload http://www.opensolaris.com/get Support Subscription http://www.sun.com/service/opensolaris OpenSolaris.Org Community http://www.opensolaris.org Packaging Repository http://pkg.opensolaris.org Documentation http://www.opensolaris.org/os/documentation Sun Partner Advantage Program http://www.sun.com/isv Contact ISV-Engineering < [email_address] >
  • 20.
  • 21.
    Next Session OpenSolarisand Advanced Technologies
  • 22.
    OpenSolaris Advanced TechnologiesZFS (Zettabyte File System) SMF (Service Management Facility) Zones and Containers DTrace And more ...
  • 23.
    Zettabyte File System(Zeta = 10 21 ) Better, Safer way to Manage Data First 128-bit File System Capacity 1 billion TB. Virtually unlimited. Fast and High Performance No Silent Data Corruption End-to-end integrity Aggressive 256-bit checksums Instantaneous Snapshots Almost Zero Overhead and Easy Rollback Application Compatibility Easy Administration. No Volume Manager Efficient Mirroring and Remote Replication
  • 24.
    ZFS Virtually UnlimitedTraditional Volumes Abstraction : Virtual disk Partition for each FS Grow or shrink manually Storage is fragmented, stranded ZFS Pooled Storage Abstraction : malloc, free No partition to manage Grow or shrink automatic Storage in the Pool is shared Volume FS Volume FS Volume FS ZFS Storage Pool ZFS ZFS
  • 25.
    Transaction Group BatchI/O Schedule, Aggregate, and Issue I/O at will Loss of Power. No Resync Runs at platter speed Object based Transaction Make these N# changes to these M# objects Atomic (all-or-nothing) Transaction Group Commit Atomic for entire group Always consistent on disk No Journal. Not needed ZFS I/O Stack ZPL ZFS POSIX Layer SPA Storage Pool Allocator DMU Data Management Unit Block Device Interface “ Write this block, then that block, ...” Loss of Power. Loss of on-disk consistency Workaround : Journaling, which is slow and complex Block Device Interface Write each block to each disk immediately to keep mirrors in sync Loss of Power. Resync. Synchronous and slow FS Volume
  • 26.
    ZFS Dynamic StrippingBEFORE Writes : stripped across all 3 mirrors Reads : wherever data was written Block Allocation Policy Considers Capacity Performance (latency, BW) Health (degraded mirrors) AFTER Writes : stripped across all 4 mirrors Reads : wherever data was written No need to migrate existing data Old data stripped across 1-3 New data stripped across 1-4 Copy-On-Write (COW) gently reallocates old data ZFS Storage Pool ZFS ZFS 1 2 3 1 2 3 4 ZFS Storage Pool ZFS ZFS
  • 27.
    ZFS Additional CapabilitiesIntelligent Prefetch Multiple Independent Pre-fetch Streams Very useful for Streaming Service Provider Automatic Length and Stride Detection Very useful for HPC Applications Variable Block Size Built-in Compression. Built-in Encryption Per-user and Per-group Quota Support Common Internet FS (CIFS) Support Integrated with Zones and FMA And more ...
  • 28.
    Service Management Facility(SMF) Simplified Service Management Services are Objects easily managed with few simple commands Automated Restart of Failed Services SMF Monitors services and Pro actively Restart on Failure Persistent Service Configuration Service Definition and Configuration persist across reboot Explicit Dependencies Easier Debugging Faster Boot/Shutdown Processes Delegated Service Administration Administrative Role Delegation to non Root users
  • 29.
    SMF Service ComponentsSMF Service Manifest Defines Default Service Properties in the Repository System Manifest files under /var/svc/manifest. Imported at boot Service Start Method(s) Defines the Interaction between Service Restarter and the Service Service Executable(s) Executable called by Method(s) to Implement the Service Service Log Records the output of the Service. Useful in Debugging Failure /var/svc/log/milestone-multi-user-server:default.log Service Fault Management Resource Identifier (FMRI) Identifies specific Service Instance Example : svc:/milestone/multi-user-server:default
  • 30.
    Zones and ContainerVirtualization instrumented within OS Kernel Resource Management Controlled and Dynamic Resource Allocation Prioritize Applications Name space, Security, and Fault Isolation Light Weight and No Overhead Theoretical limit is 8192 Easy to Clone and Migrate Rapidly create and multiply OS environment Increase System Utilization Scales with OS Single Point Administration at Global Zone
  • 31.
    Zone States ConfiguredConfiguration completed and committed Installed Packages installed successfully Ready Virtual Platform established Running Zone booted successfully and running Shutting Down Temporary state in the process of shutting down Down Temporary state completed shutting down, to go to Installed state
  • 32.
    Zones Usefulness forDevelopers Virtual OS Environment Partitioning One large system can be utilized by multiple users non-intrusively Rapidly proliferate similar Development Environment Clone within the same System Migrate to the new System Simulate Distributed Application on multiple zones Apply latest OS patch Once on the Global Zone Available to all Zones Preserve the Pre-configured Development or Deployment Environment with simple commands
  • 33.
    Branded Zone (BrandZ)BrandZ extends Solaris Zones Infrastructure Brand Attribute set at Zone Create time Branded Installation Routine for Arbitrary Collection of Software Pre/Post-boot scripts for boot-time setup and configuration Simple commands zoneadm and zonecfg Brand lx to run Linux binary application unmodified within a Zone with Solaris Kernel Brand lx runs on x86/x64 booted with 32/64-bit Kernel
  • 34.
    Dynamic Tracing (DTrace) DTrace, A Comprehensive Framework for Solaris TM Operating Environment Implement New DTrace Providers Implement fully Configurable DTrace Probes Implement New DTrace Consumers and Data Display Observability using DTrace Aggregate Arbitrary Behavior of the OS and User Programs Dynamically Enable and Manage Probes Dynamically Associate Predicates and Actions with Probes Dynamically Manage Trace Buffers and Probe Overhead Examine Live Production System or a Crash Dump
  • 35.
  • 36.
    D Language 'D'Language, like 'C' and constructs similar to awk Complete Access to Kernel C types Complete Access to Statics and Globals Rich built-in Variable set Complete Support for ANSI-C Operators Example : Trace the pid of every process named date with syscall open(2) #!/usr/sbin/dtrace -s syscall::open:entry /execname == “date”/ { trace(pid); } Probe Name o Entry point of open Predicate o Process is named “date” Action o Print the process ID
  • 37.
    DTrace Probe Pointof Instrumentation within OS Kernel Has a Name Identifies the Module and Function it Instruments Accessible through Provider 4 Attributes, Name, Module, Function & Provider, defines a tuple to uniquely identify a Probe probe description (provider:module:function:name) / predicate / { action statements } Each Probe is Assigned an Integer Identifier
  • 38.
    DTrace Provider Methodologyfor Instrumenting Probes Provides Probe Access to DTrace Framework Informed by DTrace to Enable a Probe Transfers Control of the Probe to DTrace when Enabled
  • 39.
    DTrace Consumer Processthat interacts with DTrace Concurrent Consumers unlimited. DTrace handles Multiplexing dtrace(1M) is DTrace Consumer. Generic front-end to the DTrace Facility
  • 40.
    Write an ApplicationProbe Define Provider provider myserv { probe query_receive(string, string); probe query_respond(); } Include sdt.h Add Probe code to the Application query = wait_for_new_query(); DTRACE_PROBE2(myserv, query_receive, query->clientname, query->msg); process_query(query) Compile the Application dtrace -G -32 -s myserv.d src1.o src2.o ... cc -o myserv myserv.o src1.o src2.o ...
  • 41.
    Performance Tools ProcessStats cputrack : per-processor hardware counter pargs : process arguments pflags : process flags pcred : process credentials pldd : process library dependency psig : process signal disposition pstack : process stack dump pmap : process memory map pfiles : open files and names prstat : process statistics ptree : process tree ptime : process micro-state times pwdx : process working directory Process Control pgrep : grep for processes pkill : kill processes list pstop : stop processes prun : start processes prctl : view/set process resources pwait : wait for process preap : reap a zombie process Process Tracing & Debugging abitrace : trace ABI interfaces dtrace : trace the world mdb : debug/control processes truss : trace functions,system calls Kernel Tracing & Debugging dtrace : trace and monitor kernel lockstat : monitor locking statistics lockstat -k : profile kernel mdb : debug live and kernel cores System Stats acctcom : process accounting busstat : Bus hardware counters cpustat : CPU hardware counters iostat : IO & NFS statistics kstat : display kernel statistics mpstat : processor statistics netstat : network statistics nfsstat : nfs server stats sar : kitchen sink utility vmstat : virtual memory stats
  • 42.
    Thank You! forParticipating
  • 43.
    Track back URLhttp://blogs.sun.com/stripathi