Notes for how to do includes in parsely 
23 December 1999
on the plane to Puerto Rico

1) Restrict includes to cover a single node (or range of a list of
nodes).  This is important so that we can don't need to have nodes
that are partially in one file and partially in another.

2) Includes at scan time are pretty easy to scan -- just open the new
file.  But parsing the results properly? Let's brainstorm a few
choices:
	a) add 'fileStart' and 'fileEnd' tokens for the parser.  This
	   is a bit bogus, since you'd really need an
	      x -> fileStart x fileEnd
	   production for every x.

	b) Mark the first and last node.  Postprocess the tree to see
	   if, for any node, its first token-descendant is a filestart
	   and its last descendanat is a fileEnd.  If any nodes are left
	   over, then complain horribly and die: we crossed a boundary?
	   
	   Not quite good enough -- this doesn't handle including several
	   members of a list.  Hmm.  Okay; allow:
		   i) There is a node N whose FIRST is a start and whose last
		      is an END, __or__
		   ii) There is a ListNode L which has Li and Lj, i<=j, such
		       that the first of Li is a start and the last of lj is
		       an end.

           We should also keep an include stack handy to know which nodes
	   have includes, which go with which, and so forth.

3) It sure would be handy to have the ability to de-augment and
   re-augment a parser so as to allow different start states. THis
   would let me say: 'Parse an X.  Parse a Y.  Parse a sequence of Z.'
   This would, however, damage reentrancy (which we don't have here
   anyway), and be near impossible to do in C/bison, since then I don't
   get to change the start state on the fly.
   
   So I guess not, in the general case.  But since in C we can always
   call back to the scanner from the parser to handle our includes, it
   makes sense to do this earley-only.

4) When an include occurs, it really corresponds to _two_ subtrees:
   the node doing the including, and the included nodes.  How to store
   these?  How to interface them?
       - In the one-node case, have an 'includeNode' that looks like
         the included tree.  It has 'getInclude' and 'setInclude'
         methods.  All nodes have 'isInclude' methods.
       - In the multi-node case:
          a) have _several_ includeNodes, one tagged ' start' that
             supports a 'setInclude' and 'getInclude' method?
	  b) have a single includeNode for the start of the file, and
	     have the rest of the list be references into that
	     include?
	  
5) Parser includes are tricky in Python.  If we see one on an element
   of a list, we need to allow multiple; otherwise, not.  We can't in
   general tell whether a given symbol is appearing in a list -- maybe
   only solution is to _always_ allow a list, and then backtrack?  Or 
   better: postpone include until after reduction of first tree, and
   as we add include nodes to lists, mark them as 'list-friendly'.

6) Our notion of 'dump' needs to be better.  Ideally, we want to save
   all changed files that need to be changed.  But see below...
 
7) When we include the same file twice, that's trouble.  Why?  Because
   a client might change it under the first include point, but not
   expect the second to change -- but we still have to save a signle
   version of the file back to the disk. Perhaps it's best to defer
   this or treat it as an error?
   
   Eventually, I think, we'd want to allow the client to decide
   between
       a) Having the same file node duplicated in the tree
       b) Keeping two versions of the node, and saving them as
          two files (forking!) if one changes
       c) Something still more subtle.

8) Circular includes are easy to detect -- and even better, the decision
   in 7 above rules them out.
====================================================================
1 January 2000

This include business is too complex.  It seems that this might be another
situation where, even if I manage to implement it, it would be quite
hard to know whether the semantics were right--and possibly for users to
form a good mental model of the semantics.  Solution:
 - It's time for the Wrong Thing.  Includes don't interpolate into lists.
   You must specify the include type.  This is quite wrong, but may work.

