Debugging over tcp and http

Debugging over tcp and http

• 1.
  Developing a Debuggerover TCP/IP and HTTP – Kaniska Mandal The goal is to understand how to create a debugger framework for any domain model. Here we create a debugger for an imaginary XML-based language say SML (Service Markup Language) We won't go into the details of Language Semantics and what domain model it represents. Also we shall assume there is a language interpreter and AST-Generator which will return the list of variables and there processed values upon receiving a line number from breakpoint debug event. Sample code to debug : L1 : <Declare param=”result” /> L2: <CallService serviceId=”AjaxianRSS” operationId=”getResult” output=”result”> L3: <Search name=”BreakingNews” criteria=””/> L4: </CallService> L5: <Show param=”result”/>
• 2.
  Architecture of ServiceMarkup Language Debugger 1.Debug Launcher will start the Debugger and establish a communication (TCP / HTTP) with the (Local / Remote) Debug Target. 2.The Debug View will be rendered and every Debug Event (STEP_INTO, STEP_OVER, RESUME) will be trapped by the DebugEventListner. 3.Event Listener will submit the SML Service Processing request via a DebuggerProxy to the local/server host. 4.Finally, DebugEventProcessor with the help of Language Interpreter, AST Utilities and Service Repo will generate the Debug Response 5. and send it back to the Listening Debugger Client (TCP Client Socket / Debugger Web Client). 6.The Debugger Client in turn will notify the DebugEventListner 7.Event Listener will refresh the Debug View to show variables and values an will also print in the console and do whatever else needed The sample code does not provide the complete implementation of the debugger flow. It gives good idea about how to create a debugger over TCP or Http The following pictures depicts the above mentioned flow
• 3.
  Debugging over Web Submits Request to process a debug event Starts the Remote Debugger DebuggerServlet DebuggerProxy (HTTP request handler) Debug Launcher Shows the stack frame, Send/Receive Debug Event debug thread Language As per the line number, AST Interpreter statement is Debugger UI retrieved and sends executed STEP_INTO, Debugger STEP_OVER, RESUME events View to listner DebugEvent Listner DebugEvent Processor Debug Listner also Model refreshes the Service UI when Repo variables an d values received from server Shows the stack frame, Send/Receive Debug Event debug thread Debug Launcher Debugging over TCP DebuggerHost Starts the TCP Client. (TCPClientSocket) Debug In embedded mode also starts the server, Target otherwise connects to (TCPServerSocket) the remote server
• 4.
  How to implementthe above mentioned Debug-Flow ? A. Launching Debugger : com.pramati.sml.debugger.launcher B. Client-Side Debug Event Handling : com.pramati.sml.debugger.core, com.pramati.sml.debugger.client C. Server-Side Debug Event Hanling : com.pramati.sml.debugger.server A. Launching Debugger : Launch Dialog for spawns O/S Process connecting to TCP server socket Debug Launcher Fwk Launch Dialog for Web connecting to Web Server 1.Define Launch Configuration Type and Launch Delegate - Contribute any <launchConfigurationTypes> - Specify delegate if needed (upgrade resource mappings) - Implement ILaunchConfiugrationDelegate - Contribute with <launchConfigurationType> or use <launchDelegates> for existing types 2.Associate the Launch Delegate with Launch Tabs - Contribute <launchConfigurationTabGroups> - create SMLDebuggerLaunchTabGroup to show SMLDebuggerIntegrationTab - create SMLDebuggerIntegrationTab extends AbstractLaunchConfigurationTab implements IPreferenceChangeListener ** The list of servers actually managed in preference page ** The list is displayed in the combo box of launch configuration page −maintain resource mappings for contextual launch −associate SMLDebuggerIntegrationTab with configurationType 3.SMLDebuggerLaunchConfigurationDelegate will either invoke DebuggerOnTCP or DebuggerOnWeb both of which extends StandardVMDebugger. SMLDebuggerLaunchConfigurationDelegate will create a new VM instance and capture the Launch-Configuration details from the Dialog Page.
• 5.
  >> DebuggerProxy willnotify SMLClientEventHandler once it receives any Server response. DebuggerProxy will send the DebugEvent requests. >> SMLClientEventHandler will accordingly handle the Debug Events and delegate the tasks of refreshing/updating the View to DebugTarget >> SMLClientEventHandler will be invoked if any UI event (resume/step over/suspend) takes place and accordingly it will submit the request to the DebuggerProxy who will in turn post it to the server. ` B. Client-Side Debug-Event Handling SourceCode Debugger DebugEvent Debug Display UI Controller Model (StackFrame, Variables, WatchList) VIEW CONTROLLER MODEL After defining a launch configuration, we should create Debug VIEW : (1) make source code editor adapt to breakpoints and show markers, > Figure out what types of breakpoints are needed and accordingly we have to contribute an <breakpoint> extension. > Define <marker> extension for each breakpoint > Install the breakpoints in the debug target through IbreakpointListener interface. > Implement BreakpointManagerListener in your debug target to support “skip breakpoints”. > Create an implementation of an IToggleBreakpointsTarget adapter using, and register the adapter with your editor > Contribute a RulerToggleBreakpointActionDelegate to your editor using the <editorActions> extension point (2)provide presentation UI (label, image) for breakpoints, variables > Contribute a <debugModelPresentation> extension > Provide corresponding implementation of IDebugModelPresentation , which is an ILabelProvider > Whenever a breakpoint attribute changes the breakpoint label and image are modified (3)and provide sourcePathComputers for sychronizing stack frame-threads with source code editor. > when a stackframe is selected
• 6.
  (core) >> SourcePathComputerand SourceLocator Get the SMLSourceLookupParticipant from SMLStackFrame >> contribute proper SMLLineBreakpoint (ui) SMLDebugModelPresentation provide SMLBreakpointAdapterFactory so that Editor adapts to SMLLineBreakpointAdapter - create breakpoint Debug Model: Implement debug elements: IDebugTarget , IThread , IStackFrame … Implement supported capabilities: IStep , ITerminate … Ensure model fires required DebugEvents debug model contains Debug Target SMLDebugTarget -> IDebugTarget Threads SMLThread -> IThread Stack Frames SMLStackFrame -> IStackFrame Variables SMLVariable -> IVariable Register Groups SMLRegisterGroup -> IRegisterGroup Register SMLRegister Value SMLValue Debug Controller : >> Events are sent over a separate socket >> listens for new debug events >> Either DebugUI will generate the Events or DebuggerProxy will propagate the events from Server-Side to the controller C.Server-Side Debug-Event Handling Language Local / DebugEvent Remote Processor Interpret Target er (TCPServerSocket) Client-Side DebuggerStub - sending requests – and - Service receiving Repo response -
• 7.
  Debugger in Action– whats happening behind the scene Enough of theory ! Lets dive into sample code flow … Debugging the sample script of ServiceMarkupLanguage <Declare param=”result” /> <InvokeService serviceId=”” operationId=”” output=”result”> <Search name=”” criteria=””/> </InvokeService> <Show param=”result”/> Scenario 1 : Debugging over TCP A -- Start the Debugger Client Socket and Server Socket The very first thing to do is – getting our launch dialog ready ! Section-A (Launching Debugger) already explains how to configure the Launcher. SMLDebuggerLaunchTabGroup (is a AbstractLaunchConfigurationTabGroup ) – creates the tabs as per Debugger UI requirements. SMLDebuggerIntegrationTab (is a AbstractLaunchConfigurationTab) shows the list of servers that user can configure from SMLPreferencePage. User selects local server mode so that a SML Script will be Debugged through a new OS process in a separate JVM instance. Hitting – ‘Launch’ will trigger SMLDebuggerLaunchConfigurationDelegate. *** SMLDebuggerLaunchConfigurationDelegate#launchServer(ILaunchConfiguration configuration,ILaunch launch, IProgressMonitor monitor, String mode) LaunchDelegate will collect the LaunchConfiguration data (selected file, debugger port) from the Dialog. Once the user is authenticated, required jars (for server-side event handling and language compiling) will be loaded on classpath and all VM configuration details (classpath, environment variables, program arguments) will be set up required for the VM. SMLDebuggerLaunchConfigurationDelegate will pass on the VM instance to DebuggerOnTCP ( StandardVMDebugger). *** DebuggerOnTCP#run(VMRunnerConfiguration config, ILaunch launch, IProgressMonitor monitor) (1) DebuggerOnTCP will spawn a new o/s process to run DebuggerHost (mainClass set up in the env) in that process. Process p = exec(cmdLine, workingDir, envp); IProcess process= DebugPlugin.newProcess(launch, p, label, attributes); *** DebuggerHost will start the ServerSocket for listning to the incoming events
• 8.
  and sending outgoingevents. ServerSocket serverReq = new ServerSocket(); serverReq.bind(new InetSocketAddress(recvReqPort)); logger.warn("Server listening for connections on ports " + recvReqPort); while (true) { final Socket session = serverReq.accept(); new Thread("Debug Session Handler") { public void run() { new SMLClientEventHandler(session); } }.start(); } *** SMLClientEventHandler adds itself as a listener for the DEBUG_EVENTS generated by DebugEventProcessor. DebugEventProcessor actually invokes the interpreter to process the Script based on the line numbers retrieved from breakpoint markers and then create Value objects holding {variable,value} pairs. It will generate the StackFrame event which will be sent over TCP/IP by the SMLClientEventHandler. *** It reads REQUESTs and posts RESPONSE over the same synchronized stream. (created per debugger client session) (2) DebuggerOnTCP will connect to the target VM instance via com.sun.jdi.connect.ListeningConnector .. (Listens for one or more connections initiated by target VMs). Then it will run the DebuggerProxy which will be able to receive data from ServerSocket. *** "com.sun.jdi.SocketListen"  Bootstrap.virtualMachineManager().listeningConnectors() ConnectorRunnable runnable = new ConnectorRunnable(connector, map); Thread connectThread = new Thread(runnable, "Listening Connector"); //$NON-NLS-1$ connectThread.setDaemon(true); connectThread.start(); while (connectThread.isAlive()) { DebuggerOnTCP will create the SMLDebugTarget (SMLClientEventHandler) which in turn will start the SMLClientEventHandler thread. **** SMLClientEventHandler#IStatus run(IProgressMonitor monitor) { *** initializeDebuggerClient() // SMLClientEventHandler will invoke the DebuggerProxy to start the Client Socket. while (!isTerminated() { debuggerProxy = new DebuggerProxy(); *** requestSocket = SocketUtils.connectToLocalServer(sendReqPort); SocketUtils.addShutDownHook(requestSocket); new Thread() { public void run() { listenToServer(); } }.start(); reqSocketInput = requestSocket.getInputStream();
• 9.
  reqSocketOutput = requestSocket.getOutputStream(); *** } } } So, this is how the – debugger will - Send request in one thread on client side and and Receive response in another thread on TCP server side. Once DebuggerProxy is setup, it sends the first DEBUG_EVENT request LOGIN to the DebuggerServer (DebuggerHost.java) debugger.login >> new ObjectOutputStream(reqSocketOutput).writeObject(request); reqSocketOutput.flush(); SMLClientEventHandler will get the object thru the server socket .. *** listenForRequests() { while (!requestSocket.isClosed()) { // check if we can receive requests. SocketUtils.checkRecvChannel(requestSocket); // schedule the request for running // and sending acknowledgment to client. synchronized (requestSocket) { if (reqSocketInput.available() > 0) { request = (Request) new ObjectInputStream(reqSocketInput).readObject(); if (request != null) { String methodName = request.getDebuggerMethod(); Object[] args = request.getArgs(); if(methodName.equals("login")){ // store token info in session .. debuggerProxy.registerForDebug() >> ServerDebugEventProcessor stores the SML script for processing. B – Show the Debugger UI Event Notification is performed in a separate thread which queues up the events and fire them EventListner listens to the debug events and then accordingly refreshes the UI … i.e. if the Event is IDebugtarget.create then expands the target, resume – select target, update label and refresh the children. Similarly for IThread.suspend – updates thread labels and top frame labels. ClientDebugEventDelegate#started() … 1.After initializing client socket, ClientDebugEventDelegate (SMLDebugElement)shows the Debug UI *** fires creation event : fireEvent(new DebugEvent(this, DebugEvent.CREATE)); 2.Next it reads all the installed breakpoints and notifies the SMLClientEventHandler *** fireBreakPointEvents() { IBreakpoint[] breakpoints = DebugPlugin.getDefault().getBreakpointManag- er().getBreakpoints(SMLDebugConstants.SML_DEBUG_MODEL_ID); for (int i = 0; i < breakpoints.length; i++) {
• 10.
  clientSMLClientEventHandler#breakpointAdded(breakpoints[i]); } } *** breakpointAdded(IBreakpoint breakpoint) { if (supportsBreakpoint(breakpoint)) { boolean isManagerEnabled = DebugPlugin.getDefault().getBreakpointManager().isEnabled(); if (breakpoint.isEnabled() && isManagerEnabled) { if(clientLookupThread == null) return; RemoteDebugger client = clientLookupThread.getClient(); if(client != null){ synchronized (client) { clientLookupThread.setBreakpoint(breakpoint); } } *** ILineBreakpoint lBp = (ILineBreakpoint) breakpoint; IMarker marker = lBp.getMarker(); String scriptId = marker.getResource().getFullPath() .toPortableString(); int lineNumber = lBp.getLineNumber(); markerIdVsBreakPointLineNum.put(marker.getId(),new Integer(lineNumber)); // keep track of the Breakpoints on the Server debugger.setBreakpoint(scriptId, lineNumber, true); *** The same breakpoint information is stored on the server side scriptVsbreak points Map by ServerBreakPointManager….. SMLServerEventHandler#setBreakpoint(String scriptId, int location, boolean active) { Breakpoint breakpoint = new Breakpoint(scriptId, null, location, active); breakpointManager.setBreakpoint(breakpoint); } ServerBreakPointManager#setBreakpoint(Breakpoint breakpoint) { Map scriptBreakpoints = getScriptsBreakpoint(breakpoint.getScriptId()); scriptBreakpoints.put(breakpoint.getLocation(), breakpoint); } *** This is how all the break points get stored on the server side. DebuggerProxy receives a START Event from Server once breakpoints are stored and notifies the client-side debug event listners. SMLClientEventHandler#handleEvent(DebugEvent debugEvent) case DebugEvent.THREAD_START: SMLThread mt = new SMLThread(smlDebugTarget, threadID, debugEvent.getScriptId()); smlDebugTarget.addThread(mt); smlDebugTarget.fireCreationEvent(); **** RESUME all the threads resume() throws DebugException { // resume all threads for (int i = 0; i < fThreads.size(); i++) { IThread indexThread = (IThread) fThreads.get(i);
• 11.
  indexThread.resume(); ****SMLClientEventHandler#resume() – first sends the script Id corresponding to this thread to the server so that any request to process an ASTStatement at a particular line number can be performed properly. *** String scriptId = smlThread.getScriptId(); debugger.setStepMode(scriptId, false); String modelIdentifier = smlThread.getModelIdentifier(); ILaunchConfiguration launchConfiguration = getLaunchConfigura- tion(); HashMap inputParameters = DebugUtils.getInputParameters(launchConfiguration); debugger.resume(scriptId, modelIdentifier, inputParameters); **** ServerDebugEventProcessor – sets the scriptId as the current scripted of the Interpreter Scrip- Execution Context and delegates the execution request to Interpreter. Interpreter extracts the ASTStatement with given line number (provided thru the params) and executes it. The execution logic populates locationVsVariable map. The Command notifies the listeners i.e. SMLClientEventHandler. **** if ( breakpointManager.isStepMode(scriptId) ) context.getEventManager().broadcastListeners(scriptId, DebugEvent.STEP_OVER, currentLineNumber); else if ( breakpointManager.hasBreakpoint(scriptId, currentLineNumber) && breakpointManager.isBreakPointActive(scriptId, currentLi- neNumber) ) context.getEventManager().broadcastListeners(scriptId, DebugEvent.BREAK- POINT, currentLineNumber); } SMLClientEventHandler – retrieves variable name against location and sets proper value to the variable. **** suspends server-side threads int location = debugEvent.getLocation(); if (debugEvent.getEventId() == DebugEvent.BREAKPOINT) { Thread currentThread = Thread.currentThread(); tm.suspendThread(currentThread,location); }else if(debugEvent.getEventId() == DebugEvent.STEP_OVER) { String scriptId = debugEvent.getScriptId(); boolean stepMode = debuggee.getBreakpointManager().isStepMode(scriptId); if(stepMode){ Thread currentThread = Thread.currentThread(); tm.suspendThread(currentThread,location); } } SMLClientEventHandler – posts the DebugEvent Response SMLClientEventHandler#handleEvent(..) .. receives BreakPoint notification : It sends STACK_FRAME event to server to get the values for the variables. At the same time, it notifies SMLClientEventHandler to suspend the SMLThreads
• 12.
  DebugTargetProxy in turnhandles the events and notifies ModelChnageListners to refresh the views (fireModelChanged (IModelDelta delta)…) case DebugEvent.BREAKPOINT: { // if a break point is hit request for all the variables debugger.getStackFrame(threadID); smlDebugTarget.suspend(); Show the StackFrame : client receives STACKFRAME Event >> create SMLStackFrame case DebugEvent.STACKFRAME: Debugdata rd = (Debugdata) data; SMLThread mt = smlDebugTarget.getThread(threadID); String scriptId = mt.getScriptId(); IResource resource = getResource(scriptId); if(resource != null){ SMLStackFrame sf = new SMLStackFrame(mt, resource.getName(), debugEvent.getLocation(), rd); ArrayList stackFrames = getStackFramesForThread(threadID); stackFrames.clear(); stackFrames.add(sf); // viewed stack frame. now show the data .. Debug Flow has to stop .. >> getThread(threadID).suspend(); smlDebugTarget.suspended(threadID,org.eclipse.debug.core.DebugEvent.BREAKPOINT); highlights the current breakpoint in the source code. Scenario-2 : Debugging over the web runner = new RemoteSMLServerDebugger(vm, launch); // debug mode for remote private void submitRequest(Request request) { Response response = transport.publishRequestAndGetResponse(request); if (response.getException() != null) { Throwable ex = response.getException(); if(ex instanceof JEMSException){ throw (JEMSException)ex; }else{ throw new RuntimeException(response.getException()); } } DebugEvent[] events = response.getEvents(); if (events != null) { for (int i = 0; i < events.length; i++) { fireDebugEvent(events[i]); } } }
• 13.
  The Rest issame as debugging over tcp.
• 14.
  Now so farwe have seen how communication is established between debug client an debug target (local / remote) and debug events are processed both at the Client and Server side. Next few obvious questions pop up : 1. How does break point get created upon toggle / double click ? Toggle breakpoint Action asks the active workbench part () for its its IToggleBreakpointsTarget adapter. The action interacts with the adapter class SMLLineBreakpointAdapter#toggleLineBreakpoints(IWorkbenchPart part, ISelection selection) The editor gets a call back for decorating. SMLDebugModelPresentation provides labels and images for breakpoint. BreakPoint Dialog allows user to modify properties. IBreakpoint stores the information required to install a breakpoint in a specific type of debugger : For example, In case of a language debugger, a line breakpoint stores a fully qualified type name and line number. In case of a graphical debugger, a figure breakpoint will store corresponding diagram model element uri and its location. IDebugTarget Listens for breakpoints being added/removed/changed during its lifecycle, and updates them in the underlying runtime Slide 87 of Debug_Platform_The_Basics 2. How to locate the code for a selected frame ? Slides 91-105 of Debug_Platform_The_Basics 3. How to enable source highlighting for non-textual editor ? Graphical Editors should implement IDebugEditorPresentation. Debugger will send add/remove annotation events based on which editor will highlight the diagram elements. 4. How to display custom variable view an watch expression ? slide 127 of Debug_Platform_The_Basics Eclipse Debug UI Utilities interact with the Standard Debug Model. Debug Model Elements – represent the program being debugged which has the capabilities to step, resume and terminate. DebugPlugin.fireDebugEventSet(DebugEvent[] events) DebugPlugin.addDebugEventListener(IDebugEventSetListener listener) EventListner listens to the debug events and then accordingly refreshes the UI.
• 15.
  Reference : http://www.eclipsecon.org/2008/sub/attachments/Debug_Platform_The_Basics.ppt