Sphinx Day One

This is a weird combo tour/tutorial. This is a dive — the goal isn’t full comprehension of Sphinx when you’re done, but to absorb most of the major bits you need to learn about it, culminating with its most customized part, which is the handling of Itasca’s commands and FISH.

Running

When building, you’re running “sphinx-build.exe” (a wad of python scripts bound into an executable) with your root document, “contents.rst” as an argument (more on this below). All Sphinx-based documentation is in your code source, so if you have that, you’re ready to build. At this point it might be useful to open a Windows Explorer and go to [source]\[codename]\docproject; we’ll be looking at stuff there.

You run “sphinx-build.exe” on the command line. In ICG documentation, the command you run is housed in the “make.bat” file that is found in the code’s [source]\[codename]\docproject folder. When you run it, you specify the target, like so:

make html

The argument html indicates you want the html (only) output for the doc set. Using make htmlhelp will output all needed html and then also construct the html help (CHM) file. make clean will clean out all previous output and the next build will be a complete build. You can try one of these now if you like. It is necessary to run make.bat from the docproject folder (so cd [source]\[codename]\docproject before using).

Complete Builds Are Slow

Bad news: doing a full build is slow. For the bigger document sets it can take > 20 minutes. Good news: once a full build is done, the whole output is “pickled”, which is like a zip file snapshot. On subsequent builds, only matter that has changed will be processed and re-built. The rest will be supplied from the pickle and it is likelier to take < one minute to complete. So — avoid full builds. Bad news: when an exception is thrown in the build process, there is no recovery and a full rebuild is required on the next attempt to make. Warnings and errors shouldn’t trigger this, but a full crash on a build will.

Output

Output is stored in [source]\[codename]\docproject\build\html (for make html) and [source]\[codename]\docproject\build\htmlhelp (for make htmlhelp). To view output: either open the constructed CHM file in the htmlhelp folder or navigate to the html output page in the html folder and open it (in a browser, presumably). For edit–>view cycles, it is faster to work on html output for two reasons: the build does not include the step of compiling the CHM, which adds 10-45 seconds to the process. Also, html pages can stay up and loaded in a browser on successive make html builds. By contrast, the CHM file must be closed before running make htmlhelp, which means on completion it must be reopened and you must drill back to the page that you’re working on each time you do a build.

Note

If your CHM file is open on make htmlhelp, this will not crash your build but your new CHM will not be built. So, always close the CHM before make htmlhelp.

The Two Main Files and Paths

The docproject folder also holds contents.rst, which is the root document for the entire document set, and conf.py, which holds all configuration settings for the Sphinx system. They must be at the “top” of the folder tree for the doc set. The [source]\[codename]\docproject folder is not, in fact, at the top of your document set. The top of the doc set is two folders higher (that is, [source]). Observe that at the same level as [codename] folder you’ll find \common. The doc set will need material in \common, so the top of the folder tree must be above that level. For this reason, the first thing that happens on make html or make htmlhelp, before the command string that starts Sphinx even, is that the files contents.rst and conf.py files are copied up from \docproject to [source], so that they can be processed by Sphinx from the proper “elevation.” You need to know this because when specifying the path to something, you have two choices: relative paths or absolute paths.

To get an absolute path, put a slash (/) at the front of the path string you’re supplying. This will put the doc set root in the place of the first slash. So this:

/common/kernel/doc/manual/history_manual/history.rst

Is equivalent to this:

[source]\common\kernel\doc\manual\history_manual\history.rst

A relative path omits the first slash, and uses the current file’s folder as the current location. So:

image.png              'find image.png in same
                       'folder as the current file folder
images/flac/image.png  'find image.png in a folder named "flac"
                       'that is two levels below the current file folder
../../other/image.png  'go up two folders from the current folder, then
                       'into a folder named "other" and find image.png

Usage note: the last line is valid but discouraged. We are trying to follow a general rule where relative paths are ok as long as no path is named because the resource is in the current file’s folder, or the supplied path moves in a “downward” direction. If you need to access something “above” the current file folder, use an absolute path.

Doc Set Structure and contents.rst

Let’s start looking at reStructuredText by perusing “contents.rst”. The top of the file should have a comment, which is indicated by two periods (..), followed by the matter of the comment. Additional lines after the first line are also comments if they maintain the same indentation level (super important: this is how Sphinx works — it is very white space aware). Note you can’t do an inline comment, nor can you “add” a comment at the end of a line (like you can with “;” in a code command).

Below that you should see an anchor that looks like this:

.. _docsetoutline:

In reStructuredText, two periods .. indicate the start of some kind of markup. So a comment, as seen above, is “empty” markup in reStructuredText parlance. Here, with the anchor, the structure is .. followed by one space exactly followed by _ followed by a valid string (docsetoutline here) followed by :. Elsewhere in the documentation, a link to this spot can now be constructed by adding :ref:`link label text <docsetoutline>`. It will appear like so in rendered output: link label text. Note the opening underscore and trailing colon are not part of the anchor’s identifying text, they are markup.

An anchor such as this one consumes no space in the output (it does not appear). It must be linked to something that does appear in the output so that hyperlinks that use it have somewhere to “jump” to. In reStructuredText an anchor as above will always be linked to whatever element appears next. In the case of “contents.rst”, that is the document title, which appears like this:

Itasca's |udec| Documentation
###############################

This is a section heading, which is indicated by the under-decoration (###). The bit |udec| is a substitution that is used to handle the fact that the formatting of UDEC should be (as you see three words previous) in a different font, and italicized.

Beneath the title, look at .. toctree:: (the toctree directive). Observe that the content of the directive is a set of lines that list a bunch of files (without extensions — these are assumed to be “.rst”). This commences the articulation of the document tree for the doc set. This document is the starting point, and the list in the toctree is the first level of branches. Further articulation (2nd, 3rd levels, etc.) of the tree is done simply by adding a toctree as/where needed in succeeding documents. You can get an idea by opening one of the documents listed in the “contents.rst” toctree. In all likelihood, the document you open will contain a toctree; if it doesn’t, that means you picked a page that exists at the first level and that contains no further topics (an index page, for instance). When you run build-sphinx from make you’re basically just asking it to parse “contents.rst” — but parsing that file sets off a chain reaction of further document processing that is delineated by toctrees in the source RST files.

Note

When Sphinx builds a doc set, it will search for and process any RST file in the root folder and all subfolders, whether or not the document is on a toctree somewhere. The output from the build will issue a warning for all files that are processed but not found on any toctree.

Command Indexing

If you haven’t already, do a build htmlhelp for your doc set, then open the resulting CHM file. Now navigate to the “Data” commands. This will vary by code.

UDEC = Data\Commands

FLAC3D = FLAC3D Elements\Data\Commands

PFC = General Components\Data

Ok, now open the corresponding RST source file. Because this file comes from \common, there is only one source file for it.

[source]\common\kernel\doc\manual\data\commands\commands.rst

Looking at the help file, you see that this page “Commands” is sitting in a folder named “Data”. If you were to open the source RST file for that document, you’d see that it has a toctree with two lines in it — the first lists this page, and the second lists the “FISH Functions” document that is below this one.

The first few elements should look familiar, from previous discussion: the comment at top, the anchor beneath it, and the title beneath that. Following that, you’ll see a rubric directive (.. rubric:: Data Commands — and see where this appears on the CHM page). A rubric is a way of adding a heading-type element to a page that will not be also added to the navigation tree in the help file (any heading built with under-decoration is automatically added to the tree and this behavior has no override — hence rubric). If you look at “Command Indexing” above, it was built with this:

.. rubric:: Command Indexing
   :class: h2

Using this rubric as an example, here is the anatomy of a reStructuredText directive. It begins, as mentioned before, with two periods (..). The name of the directive follows after precisely one space, rubric in this case. Exactly two colons follow the name. Matter following the colons is termed an argument. According to its rules a directive may take zero, one, or n arguments (zero or one are the most common). Options are switches that come in successive single lines (no blank lines allowed!) and they must match the directive’s indentation such that the colon of each option aligns with the first letter of the directive name. Values for the option appear after the terminating colon (h2 in the above case).

The rubric as defined above cause the words “Command Indexing” to be rendered in the style of a second level heading. However, had it been built like this:

Sphinx Day One
##############

... body text here after title of page ...

Command Indexing
----------------

If you haven't already, do a ...

then “Command Indexing” would be styled as a second level heading (which is wanted) and also appear as a node under “Commands” on the help file’s navigation tree (which is not wanted).

The next bit in the file .. cmdindex:: is a customization built by Itasca. As you can surmise by seeing the length mismatch between the markup and the table that it creates on the page, something special is going on here. More on this in a bit.

Look at the toctree which is the last part of the document. The presence of this incorporates the actual commands data label create, data label delete, etc. to the help file and its navigation tree. Note the :glob: argument of the toctree directive. This allows specifying all files in the current folder that start with “cmd_” (using a wildcard, cmd_*) to be added (alphabetically) to the toctree. Now, in Windows Explorer look in the folder containing the current document. You’ll see a document for each command that starts with “cmd_” (e.g., cmd_data.label.create.rst). These are the files the toctree will glob. You will also note there is a corresponding “kernel” file (e.g., _data.label.create.rst). All command and FISH documentation is built this way — with a “shell” (the cmd_ file; or fish_ for FISH files) and a “kernel” (a file that starts with an underscore and is otherwise identically named as the shell). More on this in a minute when we look at an actual command file (which you now realize is two files). Let’s go back to .. cmdindex::.[1]

Basically, .. cmdindex:: is a bit of automagic brought to you by Itasca that tells Sphinx to build the table you see on this page. There is a lot that goes into that, not all of which will be explained here. Ignore the :class: not modified option for now. The main thing to note here is that the contents of the table to be created are commands identified by the option :module: data. The module name is connected in a specific way to the command files, which we’ll look at now.

Command File Anatomy

For illustration we will work with the command data label create. Navigate to that command in the CHM file. Open the two files that constitute that command:

[source]\common\kernel\doc\manual\data\commands\cmd_data.label.create.rst

[source]\common\kernel\doc\manual\data\commands\__data.label.create.rst

Before getting into the contents of the files, something about these names is worth noting here. As you know, the periods shown in FISH function names are used in the codes (e.g., type data.label.create exactly to use that function), but periods that you see in the command file names are not (e.g., type data label create [spaces, no periods!] exactly to use that command). So why the periods in the command files? More automagic. You can build a link to either a function or a command anywhere in the documentation like so:

:cmd:`data.label.create`
:func:`data.label.create`
:cmd:`geometry.edge.create.by-arc.segments` ;illustration: you can
                                            ;tack keywords on with periods
                                            ;link will go right to keyword

The last line illustrates the ability to link not just to the command page, but to keywords within the command, simply by tacking them on with periods (in a syntactically valid order, of course). Click me: geometry edge create by-arc segments. So the longer answer to “why the periods in command file names?”: they help simplify and make consistent the construction of the automagic, which is, I hope you’ll agree, kind of supercool.

A Command shell

Now let’s walk through the file cmd_data.label.create.rst. It has a title, easily identified by the under-decoration (####). Note the use of :lcmd:`data label create`. This is very close to the usage :cmd:`data.label.create`. The addition of “l” to the role indicates that this is literal, so something different happens here: a link does not need to be built (which would just be a link to this very page), but otherwise the formatting of this bit should be identical to one of the command links. We don’t include the periods in the command either. Because this is literal, we must omit them so they will not appear in the rendered text — which would be an inaccurate presentation of the command. Literal roles, which are also available for data types (:int:`i` vs. :lint:`3`, :str:`s` vs. :lstr:`fred`, etc.) are also available; all are Itasca customizations.

The ..rubric:: that follows should be familiar. The option :class: h2 added to it specifies that this rubric should appear like a second level heading (there is an “h1”, “h2”, and “h3” that can be applied to rubrics; these match the automatically supplied styles provided when using under-decoration for sections).

The ..htmlinclude:: bit does exactly what you suspect: it “sucks in” the kernel file. A relative path is used here, since shell and kernel file live in the same folder.

Below that you see a comment (the “empty” directive ..). All lines below it are at the same indentation level, so they are all commented out.

A Command kernel

The kernels are the heart of the commands and FISH documentation. Again, let’s walk through.

The ..raw:: latex directive isn’t actually used; it is a leftover from when there was the prospect of having a print/PDF output. However, we’ll spend a moment with it to illustrate this point: a directive may or may not have “content” (in Sphinx parlance), and the content of the directive is anything below the lines constituting the directive itself and any supplied arguments, separated by a blank line, that is at an indentation level equal to the name of the directive (raw in this case). So, by that definition, you see that the content of the .. raw:: directive is the line ``.  Lastly, the context of the directive is broken by "breaking" the indentation level, which is done in the line starting with ``.. command::. As mentioned earlier, Sphinx makes heavy use of whitespace. Errors in your markup from wrong use of whitespace are 1) easy to make and 2) difficult to identify, because they won’t necessarily generate errors or warnings that will show up in the build output. You’ll just not like your output and have to figure out where you do or do not have a missing/needed carriage return or space character. This very page and all commands and FISH are good illustrations of whitespace at work in Sphinx.

The next directive is another custom Itasca extension of Sphinx:

.. command:: DAta.Label.Create keyword ...

The directive indicates that this is an Itasca code command. The periods used in the directive are tied to the automatic mechanism that allows you to write :cmd:`data.label.create` elsewhere in the doc set to build a link. This very spot is the exact target of such a link. Capitalization of letters in the command signature specifies that these characters are to be underlined in the output. Underlining is the indicator for the minimum characters necessary for recognition by the code command processor (so the abbreviation-minded can type in da.l.c for this command). keyword indicates that a keyword can follow the command words, and ... indicates that more than one keyword can be supplied.

Remember how :module:`data` appeared as an option of the .. cmdindex:: directive in the file [source]\common\kernel\doc\manual\data\commands\commands.rst (the bit of automagic that built the index table of alphabetized “data” commands)? Well, this is the other shoe dropping. Every .. command:: should be tagged with a :module:. As you can see, this one is tagged data. When the index page uses:

.. cmdindex::
   :module: data

It will gather up all commands like this one that are tagged with data and put them into the index table.

The main description of the command follows, at the same indentation level as the .. command directive.

Now, the .. kwd:: directive is nested within the .. command:: directive. It maintains the indentation level of the content for the .. command:: directive. As you can see by scanning to the bottom of the document, in fact all remaining content is within the .. command:: directive. As with the main command, capitals are used to indicate minimum required characters for the keyword. The keyword itself is followed by whatever data type is required, if any.

Warning

In the Sphinx system, variables should in all possible cases at least start with their initial, and maybe only use their initial. So, you want :bool:`b` to indicate a boolean; to be literal you’d use :lbool:`true` if you really really have to supply “true” and not “b”. You may run into cases of multiples, such as :int:`i1`, :int:`i2`, :int:`i3`, in which case numbering things as shown is fine as long as the initial is there first. You need: “i” for ints, “f” for floats, “s” for strings, “v” for vectors, and so on. If a variable is named wrongly (e.g., you’ve supplied something like :int:`fred`), you’ll get an error message in your output. So please, make it :int:`i` if possible or :int:`ifred`, if you must.

We don’t see this in data label create, but keywords are added below other keywords by further nesting, ad infinitum:

.. kwd:: FIrstlevelkeyword keyword ...

   Description of this guy here.

   .. kwd:: SECondlevel1

      I'm at the second level.

   .. kwd:: SECondlevel2 keyword

      So am I.

      .. kwd:: THIRDlevel

         I'm a keyword of :lkwd:`secondlevel2`.

End of Sphinx Day One

Ok, let’s stop here. Believe it or not, you know most of what you need to get started with Sphinx, and if you’ve gotten this far then you clearly have the system up and running. From here, if a “Sphinx Day Two” was built, it would cover lots of specifics, rules, details, etc. It isn’t built, so your best bet for figuring out how to do things is to keep the CHM handy, drill around in it to find something like what you want, then open the source RST and copy it. Trust me, you already know enough about Sphinx to do that and be able to read which bits of the source RST you need, and which bits are the part you’ll alter for your content.

Footnotes

Footnotes

[1]

Ok, you get a prize if you’re really observant and have said, “Hey, wait a minute. All those “kernel” files are not on a toctree anywhere — the glob didn’t get those. Aren’t they going to cause a lot of warning messages to spew out during a build?! And what a heckuva lot of them we’ll have…” Yes, they would, except in the configuration file (conf.py), there is a mechanism for specifying files to ignore. For our doc sets, we have specified ignoring all files that start with an underscore. So a final bit of advice here is do not give a file of “regular” content a name that starts with an underscore.