.. _Debugging Tips: Debugging Tips ============== This section describes various method for debugging VisIt_ when you run into problems. Whether it is a crash with a new filter you've designed, or badly formed data from a new database reader, one of these options should help. .. _DebugLogs: Debug logs ---------- The first method for debugging in VisIt_ is by using VisIt_'s debug logs. When you run ``visit`` on the command line, you can optionally add the ``-debug 5`` arguments to make VisIt_ write out debugging logs. The number of debugging logs can be 1, 2, 3, 4, or 5, with debugging log 5 being the most detailed. When VisIt_'s components are told to run with debugging logs turned on, each component writes a set of debugging logs. For example, the database server component will write A.mdserver.1.vlog, A.mdserver.2.vlog,...,A.mdserver.5.vlog if you pass ``-debug 5`` on the VisIt_ command line. Subsequent runs of VisIt_ will rename existing logs with the initial letter advanced to the next letter in the alphabet. For example A.mdserver.5.vlog will be renamed to B.mdserver.5.vlog. At most five sets of debugging logs will be kept. The logs from the most current run will always begin with *A*. If you don't want that behavior, you may add ``-clobber_vlogs`` to VisIt_'s command line arguments. The A.mdserver*.vlog and A.engine*.vlog files are useful when debugging a database reader plug-in. The A.viewer*.vlog and A.engine*.vlog files are usefule when debugging a plot plugin. The debugging logs will contain information written to them by the debugging statements in VisIt_'s source code. If you want to add debugging statements to your AVT code then you can use the *debug1*, *debug2*, *debug3*, *debug4*, or *debug5* streams as shown in the following code listing. Keep in mind that level 1 debug log files are less populated than level 5, and may be most useful when making temporary code modifications and debugging specific problems and not just providing general information. .. container:: collapsible .. container:: header Example for using debug streams .. code-block:: c // NOTE - This code is incomplete and is for example purposes only. // Include this header for debug streams. #include vtkDataSet * avtXXXXFileFormat::GetMesh(const char *meshname) { // Write messages to different levels of the debug logs. debug1 << "Hi from avtXXXXFileFormat::GetMesh" << endl; debug4 << "Many database plug-ins prefer debug4" << endl; debug5 << "Lots of detail from avtXXXXFileFormat::GetMesh" << endl; return 0; } Other forms of the debug argument ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To create debug logs for only specific components, use ``-debug_ ``. For example ``-debug_mdserver 4`` will run the mdserver with level 4 debugging. Multiple ``-debug_ `` args are allowed. For parallel engine logs, ``-debug_engine_rank `` can be used to restrict debug output to the specified rank. You can even have only every Nth processor output debug logs by using ``-debug-process-stride N``. .. _DumpingPipelineInfo: Dumping VTK objects and pipeline information to disk ---------------------------------------------------- In addition to the ``-debug`` argument, VisIt_ also supports a ``-dump`` argument. The ``-dump`` argument tells VisIt_'s compute engine to write intermediate results from AVT filters, scalable rendered images, and html pages. The ``-dump`` option takes an optional argument that specifies the directory for -dump output files. The intermediate results are VTK files containing the data for every stage of the pipeline execution so you can view the changes to the data made by each AVT filter. Each VTK filename begins with a number indicating the order of the filter in the pipeline that saved the data, followed by an indication of whether it is an ``input`` or ``output`` for the filter, and finally the filter name. For example, the input to the project filter could be ``0006.input.avtProjectFilter.vtk``. The html files contain information about input to and output from each filter, including spatial and data extents, pipeline flags, and number of data files input and output. While the VTK files dumped by this option are more useful when debugging plots and operators, they can still be useful for debugging database plugins, as data sent from the plugin can be examined. The files generated at this first stage have ``gdb`` as the beginning of the filename, such as ``gdb.0003.output.GetOutput.dom0000.vtk`` When you run VisIt_ with the ``-dump`` argument, many files will be created since the data is saved at every stage in the execution of VisIt_'s data processing pipeline. It is a good idea to keep this in mind and to remove those files from time to time. Dumping only pipeline html files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To get only the html pages, and no VTK objects, use ``-info-dump`` instead. It also takes an optional argument specifying the directory for output files. Comparing --dump outputs ~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes the output from ``-dump`` looks correct and doesn't immediately reveal why things are broken. In those instances it may be helpful to compare a ``-dump`` run from a version of VisIt_ that isn't broken to the ``-dump`` run from the version that is broken. Doing such a comparison might more quickly reveal which filter in the pipeline has changed it's output VTK object or even pipeline information. Attaching a debugger -------------------- VisIt_ has various options for attaching debuggers on Linux machines, including ``gdb``, ``totalview``, and ``valgrind`` to name a few. The full list is available in the :ref:`Startup options` section under ``Debugging options``. Typically, when starting up VisIt_ and attaching a debugger, the launcher logic starts the debugger and loads the VisIt_ component executable and sets the command-line arguments but does not actually start the executable. Instead, it opens with the debugger's interactive prompt where the user must enter ``run`` to actually start the component. This allows the user to set breakpoints even before starting the component. Alternatively, the startup option, ``-break`` can be used to inject a breakpoint into the debugger before it starts running the VisIt_ component. For example, the following command will start the ``mdserver`` component under ``lldb`` (debugger used on macOS) in its own X-terminal and set a breakpoint in the function ``MDServerMain`` which is one of the first functions executed when the ``mdserver`` is starting up... :: ./bin/visit -xterm -lldb mdserver -break MDServerMain WaitUntilFile function ---------------------- VisIt_ has a utility function called `WaitUntilFile` that will halt process execution until the file passed into the function has been created. It takes one argument, a full-path filename referencing a file that does not yet exist. The function will enter a loop, alternating between short sleeps and checking if the given filename exists. Once it determines the filename exists, the function will exit and normal program flow will continue. This allows time for you to attach a debugger to the running process and set breakpoints before creating the filename that signals the function to exit. While this function can be used anywhere in VisIt_'s pipeline, it is especially useful for debugging problems with a component's startup process, where it may be harder to attach a debugger in time. `WaitUntilFile` is declared in VisIt_'s `Utility.h` header. To use `WaitUntilFile` to debug a component's startup process, simply modify the `main` program of the component, adding a call to the `WaitUntilFile` at the very beginning of the method. Then rebuild and run VisIt_. Once the desired component is in the `wait` state, attach the debugger, and set a breakpoint. Then create the file that was passed as the argument to `WaitUntilFile`. Don't forget the wait file will need to be deleted in between subsequent debugging sessions. See the table below for components, the files containing their `main` method, and the name of `main` method. ========= ======================== ================ component file containing main main method name ========= ======================== ================ gui src/gui/main.C GUIMain viewer src/viewer/main/viewer.C ViewerMain engine src/engine/main/main.C EngineMain cli src/visitpy/cli/cli.C main ========= ======================== ================ .. container:: collapsible .. container:: header An example of modifying GUIMain with WaitUntilFile .. code-block:: c // Example only, the code block is incomplete. #include int GUIMain(int argc, char **argv) { WaitUntilFile("~/guiwait.txt"); int retval = 0; TRY { // Initialize error logging. VisItInit::SetComponentName("gui"); raise(SIGSTOP) -------------- Sometimes, its easiest to get a debugger to break exactly where you want by modifing the source code using the `raise() `__ function. The following example shows how to use ``raise()`` to have the program break in ``factorial()`` .. container:: collapsible .. container:: header An example of using raise(SIGSTOP) .. code-block:: c #include int factorial(int a) { int retval = 1; raise(SIGSTOP); for (int i = 1; i <= a; i++) retval *= i; return retval; } int main(int argc, char **argv) { return factorial(argc); } When the program runs, it will stop in the function ``raise(SIGSTOP)`` and will not continue until it has been sent a signal to continue, ``SIGCONT``. This allows the user to attach a debugger and then simply type ``cont`` once attached to continue. Or, the ``kill`` shell command can be used to send a signal to the stopped program as in :: kill -SIGCONT Where ```` is the program's process id. The command ``kill -l`` will list all the signals that can be sent to the program. PrintCallStack() ---------------- The VisIt_ sources also includes a helpful utility function in ```` that will print the call stack in effect at the moment the function is reached. For example, to debug MOAB's database read options ``PrintCallStack()`` can be added to the ``avtMOABOptions.C`` .. container:: collapsible .. container:: header An example of using PrintCallStack .. code-block:: c #include . . . DBOptionsAttributes * GetMOABReadOptions(void) { printf("In GetMOABReadOptions()\n"); PrintCallStack(std::cout, __FILE__, __LINE__); DBOptionsAttributes *rv = new DBOptionsAttributes; rv->SetBool("Parallel format", true); . . . When the code gets executed, it will produce output something like .. container:: collapsible .. container:: header An output from using PrintCallStack .. code-block:: c In GetMOABReadOptions() Call stack from /Users/miller86/visit/visit/34rc/src/databases/MOAB/avtMOABOptions.C 39 1: 1 libMMOABDatabase.dylib 0x00000001194cd671 _Z18GetMOABReadOptionsv + 49 2: 2 libMMOABDatabase.dylib 0x00000001194b2541 _ZNK20MOABCommonPluginInfo14GetReadOptionsEv + 17 3: 3 mdserver 0x000000010d7aeb06 _ZN18MDServerConnection15GetDBPluginInfoEv + 502 4: 4 mdserver 0x000000010d7a0705 _ZN26GetDBPluginInfoRPCExecutor6UpdateEP7Subject + 165 5: 5 libvisitcommon.dylib 0x00000001136148d8 _ZN7Subject6NotifyEv + 168 6: 6 libvisitcommon.dylib 0x00000001133fa39d _ZN16AttributeSubject6NotifyEv + 29 7: 7 libvisitcommon.dylib 0x00000001136763d6 _ZN4Xfer7ProcessEv + 470 8: 8 mdserver 0x000000010d7abf81 _ZN18MDServerConnection12ProcessInputEv + 65 9: 9 mdserver 0x000000010d7a5b57 _ZN19MDServerApplication7ExecuteEv + 471 10: 10 mdserver 0x000000010d7c3189 _Z12MDServerMainiPPc + 281 11: 11 mdserver 0x000000010d7c37e2 main + 34 12: 12 dyld 0x00007ff81344c418 start + 1896 Debugging a regression failure outside of the test suite -------------------------------------------------------- Sometimes the testing harness infrastructure gets in the way of debugging a failing regression test, and you just want to run the testing script or a portion of the script directly with VisIt_'s cli. Here's a quick way to do just that. First, you need a script that mimics some of the testing harness functions, so you don't need to modify the actual testing script as much. Here's an example of what is needed: .. container:: collapsible .. container:: header TestingStuff.py .. code-block:: python # script to aid in debugging regression tests outside of the testing harness # it mimics some of the testing methods so that actual test scripts don't # need to be modified so much # use this script by adding 'Source("TestingStuff.py")' to the top of a # regression test. Use full path if the regression test doesn't live at # the same location as this script. # mimic testing 'data_path' by specifying a location where the testdata # can be found. It is best if this points to an actual build/testdata dir # so that you are using the same data as the regression tests def data_path(fname): return "/my/path/to/VisIts/testdata/%s"%fname def silo_data_path(fname): return data_path("silo_hdf5_test_data/%s"%fname) def TurnOnAllAnnotations(givenAtts=0): """ Turns on all annotations. Either from the default instance of AnnotationAttributes, or using 'givenAtts'. """ if (givenAtts == 0): a = AnnotationAttributes() else: a = givenAtts a.axes2D.visible = 1 a.axes3D.visible = 1 a.axes3D.triadFlag = 1 a.axes3D.bboxFlag = 1 a.userInfoFlag = 0 a.databaseInfoFlag = 1 a.legendInfoFlag = 1 SetAnnotationAttributes(a) def TurnOffAllAnnotations(givenAtts=0): """ Turns off all annotations. Either from the default instance of AnnotationAttributes, or using 'givenAtts'. """ if (givenAtts == 0): a = AnnotationAttributes() else: a = givenAtts a.axes2D.visible = 0 a.axes3D.visible = 0 a.axes3D.triadFlag = 0 a.axes3D.bboxFlag = 0 a.userInfoFlag = 0 a.databaseInfoFlag = 0 a.legendInfoFlag = 0 SetAnnotationAttributes(a) def Test(fname): swa = SaveWindowAttributes() swa.family = 0 swa.fileName = fname swa.screenCapture = 0 SetSaveWindowAttributes(swa) SaveWindow() def Test(fname, swa = 0, alreadySaved=0): if (swa != 0): sa = swa else: sa = SaveWindowAttributes() sa.screenCapture = 1 sa.family = 0 sa.fileName = fname SetSaveWindowAttributes(sa) SaveWindow() def TestText(name, results): print("%s: %s"%(name, results)) def TestSection(stuff): print(stuff) def Exit(): exit() Now, you can copy a regression test to the same directory as this script, add ``Source("TestingStuff.py")`` to the top of the regression test, and run ``visit -cli -s testname.py``, along with any debugging options you desire.