[texinfo-pretest] conditional macros problem

[Stepan Kasal]

Hi,

On Sun, Feb 13, 2005 at 05:59:37PM -0500, Karl Berry wrote:
> @newmacro @mymac {
> line1
> line2
> @quotation
> whatever
> @end quotation
> }
> 
> The problem is that commands like @quotation are delimited by an ^^M
> char, so ^^M will have to be active when the @newmacro definition is
> read.  But ^^M shouldn't always mean a new line, of course -- e.g., the
> one after line1.  [...]

OK, so the main problem is that we too often change parsing rules during
the run of TeX.  In other words, we use \catcode too often.

In order to be able to change parsing rules in the middle of @macro, we
use the \scantokens primitive of e-TeX (or its emulation in TeX).
This is what makes things ugly.

The main problem is the difference between places where ^M is endline
and where it's active.  It would be nice if we were able to find a
definition of ^M which would work in normal text--producing \par on an
empty line and space otherwise (but without making a double space if
the line ended with a space).  But as this is not possible, we have to
switch the \catcode often.  And to get @macro's right, we have to save
them with ^M remembered, ie. the catcode of ^M cannot be end-of-line,
and we have to use \scantokens.

So this hack is not going to be removed from texinfo.tex, unless...

Unless we state that @macro's _must_ be expanded by makeinfo before we
run tex.  That would save us a lot of work.

Let me attack the same problem from another angle:
LaTeX users often wonder why
	\newenvironment{myverb}
		{\begin{verbatim}}
		{\end{verbatim}}
doesn't work.

They ask this question, because their perception is that the LaTeX
_first_ expands _their_ defined macros, and then it processes the result.
This is not true, of course.  I see this as the main reason why LaTeX
cannot succeed with its attempt to build a new, different, consistent
macro language over the (inconsistent) language of TeX.

We have the same misunderstanding here at texinfo, with @macro's.
People imagine that @macro's are expanded first, and then the result
is processed.  But luckily, _we_ can accept that view and do it indeed
that way.  As I said, this would save us lot of trouble.

If we go that way, the whole ugly code for @macro's will go away from
texinfo.tex, replaced by a simple error message pointing to texi2dvi.
(texi2dvi would execute makeinfo -E --iftex, which has to be fixed first,
of course.)

Some instances of \scantokens would still remain in texinfo.tex, eg. for
@copying/@insertcopying.  Now, when we require pre-processing by makeinfo,
we can handle those the same way:  makeinfo -E  would remove @copying and
replace @insertcopying by the actual text.

One more final note:  texinfo.tex changes catcodes on too many places.
I explained that the main problem with the catcode of ^M cannot be
eliminated, but there are many other problems, eg. with changing catcodes
for indices and toc.  The following cleanup would be very helpful:
1) identify the various catcode states ("state" means a vector of 256
catcode values) which could appear when running texinfo.tex
2) identify when can which state be rached and make the changes between
those states clearer
3) identify states which can be eliminated, eg. by changing the format
of auxiliary files, and make the changes to eliminate them.

Changes to the way input is parsed are dangerous and should be used
only if necessary, and in a transparent way.

Well, I'm afraid I will never find time to implement this.  ;-)

Have a nice day,
	Stepan