Stages Built Into
NetRexx Pipelines 4.03
&
CMS Pipelines V7R1
and Their Differences

• delegates to comment.
• Not in CMS Pipelines;
• This is a STAGE, not a programming comment. It must have a SPACE after --.
• It must have either a stageEnd or pipeEnd.
• If used before a driver stage, it must have a pipeEnd.

• Records are parsed via the parse_template_delimited_string.
• Variables are named $n, where n is 1 to 9.
• The by_NetRexx_delimited_string is interpreted. This is 0 or more semicolon separated NetRexx statements, probably using the $n variables, which can have the value altered.
• Other variables may be used, and are persistent while the stage is active, so can be used as accumulators.
• The values of the variables are put into the output_template_delimited_string replacing $n.
• For a literal $n that won't be changed, use $$n.
• A first_NetRexx_delimited_string, if present, is interpreted before reading any record from the primary input steam. This is 0 or more semicolon separated NetRexx statements, probably using the $n variables. Any variables used in the by_NetRexx_delimited string must be defined here.
• A finally_template_delimited_string, if present, is written as a final output record after the primary input stream is finished, using the $n's.
• Any keyword phrases must, in any order, follow any non-keyworded FROM & TO phrases.
• This is NetRexx Pipelines only, not in CMS.

• Uses the Java RegEx classes and its dialect of RegEx. See Java's Pattern class and replaceFirst and replaceAll methods of String for full documentation.
• (1) First, FROM, delimitedString is a Java RegEx expresion for what is to be replaced.
• (2) Second, TO, delimitedString is the replacement string. It may contain elements from the first one.
• Ths is NetRexx Pipelines only, not in CMS.

• Not in CMS Pipelines;
• This is a STAGE, not a programming comment. It must have a SPACE after --.
• It must have either a stageEnd or pipeEnd.
• If ended with a stageEnd, it passes records through on primary input to output streams.
• If ended with a pipeEnd, it does NOT pass records through.
• If used before a driver stage, it must have a pipeEnd.

• (1) -1 = Primary is shorter/less, 0 = equal, 1 = Secondary is shorter/less
• (2) 0 = equal, 1 = not equal
• (3) Primary is LESS/shorter (or MORE/longer) than secondary
• (4) DStrings can use any of the following escapes (or the lowercase) for the unequal situation:
  • \C (count) for the record number,
  • \B (byte) for column number
  • \P (primary) for the primary stream record
  • \S (secondary) for the secondary stream record
  • \L (Least) for the stream number that is shorter, -1 if equal
  • \M (Most) for the stream number that is longer, -1 if equal
• (5) Equal or not, this DString precedes any of the others.
• (6) This is NetRexx Pipelines only, not included in CMS
• (7) In reporting \P & \S, control charactors, except new line, \n, are transliterated to [blob, 219.d2c()]
• (8) Without ECHO, this stops and reports at first non-compare. With ECHO, each primary input is reported; after first non-compare primary input stream records continue to be read and reported, but no testing is done.
• (9) Options work in any order
• Input streams:
  • 0: Data 1
  • 1: Data 2
• Output streams:
  • 0: Result (single record, possibly multiple lines)
  • 1: Last primary record read at first no match, or end of stream
  • 2: Last secondary record read at first no match, or end of stream

• NOTE: CMS is only 64DECODE, and does not have the options; it does MIME.
• BASIC - Output is mapped to a set of characters lying in A-Za-z0-9+/. The encoder does not add any line feed in output, and the decoder rejects any character other than A-Za-z0-9+/.
• URL - Output is mapped to set of characters lying in A-Za-z0-9+_. Output is URL and filename safe.
• MIME - Output is mapped to MIME friendly format. Output is represented in lines of no more than 76 characters each, and uses a carriage return '\r' followed by a linefeed '\n' as the line separator. No line separator is present to the end of the encoded output.
• 3.11: New to NetRexx. Add MIME, BASIC, & URL options.

• As in CMS, equivalent to diskr (Pipes for NetRexx Only) or <.

• As in CMS, equivalent to diskr (Pipes for NetRexx Only) or <.

• DISPLAY works similar to and as a replacement for CONSOLE for output. But instead of going to the terminal window, it goes to a HTML file browser tab. This allows for HTML+CSS tags to control fonts, colors, and layout. This stage can be found in examples-pipes.
• To work, these are required outside Pipelines and NetRexx:
  • A working HTML browser program
  • The operating system to associate the filetype "html" with the browser, so the Pipelines stage "COMMAND PipeDisp.html" does call the browser and display the file.
  • The system have a Temp directory, known to Java.
• The DISPLAY stage overwrites the named file, by default PipeDisp.html, in the system Temp directory, then calls the COMMAND stage to display it. The file is not erased automatically by this stage.
• Each DISPLAY stage invocation opens a new browser tab, which remains open.
• The AS option causes the data to be surrounded by html tags.
  • The default TEXT or PRE puts on <pre> and </pre>. Most browsers use:
    • Fixed width font
    • Display all the white spaces: line feeds and multiple spaces
  • HTML uses <html> and </html>. Most browsers use:
    • Variable width font
    • Consolidate strings of white space into a single space
    • All the HTML tags
  • TABLE uses <table> and </table>
    • Expects the data records to begin with <tr><td> (or <tr><th>)
  • NOTAG uses <pre> & </pre>, but first converts all & characters to the entity &amp; and < characters to &lt; so HTML tags are not processed.
  • NONE uses no extra tags. Most browsers use:
    • HTML display
• OPTIONs QString is included in the opening tag for the AS option. This could be CLASS, STYLE, or other options.
• TITLE QString adds <title>delimitedString</title> to the beginning of the output. This should show as the title in the browser's tab.
  Note: This officially should go into a HEAD section; here it won't be there. Most modern browsers will honor it anyplace in the file. If it is not honored as a tag, QString will be the top line of the display.
• STYLE QString adds <link rel="stylesheet" href="QString"> to the beginning of the output. This should include and use the named stylesheet. The name may have relative path names, or be an absolute file name. If there are spaces, enclose it in quotes.
  Note: This officially should go into a HEAD section; here it won't be there. Most modern browsers will honor it anyplace in the file. If it is not honored as a tag, it will not show -- except in the NOTAG option. The file itself is copied from its stated location into the system Temp directory, overwriting any existing file. This file is not erased automatically by this stage.
  QString: It is optional to enclose the name in quotes, but quotes are required if the name includes spaces.
• FILETYPE may be used to change the default "html". This permits use of other types that MAY be preprocessed if the system, external to Pipelines, is set up to recognize it, for example, "JSP" or "PHP". A "dot" is optional; only one will be used.
  Note: filetypes other than .html may be handled by the system by some program other than the browser.
  QString: It is optional to enclose the type in quotes.
• FILENAME may be used to write and display another file. It may include a path designation, either absolute or relative. A relative path is based on the working directory. If no path is specified in the name, the system Temp directory, as determined by Java, is used.
  QString: It is optional to enclose the name in quotes, but quotes are required if the name includes spaces.
• Records from the primary input stream are also put out on the primary output stream unchanged, if it is connected.

• NOTE: CMS is only 64DECODE, and does not have the options; it does MIME.
• BASIC - Output is mapped to a set of characters lying in A-Za-z0-9+/. The encoder does not add any line feed in output, and the decoder rejects any character other than A-Za-z0-9+/.
• URL - Output is mapped to set of characters lying in A-Za-z0-9+_. Output is URL and filename safe.
• MIME - Output is mapped to MIME friendly format. Output is represented in lines of no more than 76 characters each, and uses a carriage return '\r' followed by a linefeed '\n' as the line separator. No line separator is present to the end of the encoded output.
• 3.11: New to NetRexx. Add MIME, BASIC, & URL options.

• NetRexx Pipelines only.
• Records matching the RegEx are put out on primary output.
• Records not matching are put out on secondary, if connected, or discarded.
• .
• (1) Regex_string is a Java RegEx expresion. Null string passes all records.
• (2) Records are prefaced with records number, 10 characters, right justified.
• (3) Number of records put out after a matching record.
• (4) Number of records put out before and after a matching record.
• (5) Inserted before a group of "before records" or the found record with "after records."
• (6) Send all matching records (no numbers) to tertiary output stream, if connected.
• (7) Only a count of matches is put out on the primary output stream. (Other options probably should not be used with this.)

• HTMLROWs reads rows from its primary input stream and writes them to its primary output stream, altering them to have the proper HTML tags for TABLE ROWS.
• I.e., it converts
    abc,mnop,xyz
  into
    <tr><td>abc</td><td>mnop</td><td>xyz</td></tr>
• The SEPARATOR QStirng, by default the comma character, can be specified.
• There are options to put additional data inside the tags. This could be used for class or style tag options, for example.
  • ROW QString : puts its information into the <tr>-tags
  • DATA QString : puts its information into the <td>-tags
  • HEAD QString : puts its information into the <th>-tags (1)
• QString is a quoted string of characters. The quote marks may be either single or double, but must match. If there are no spaces in the string, the quote marks are optional.
• (1) If there is a HEAD option, the first row read has <th>-tags instead of <td>-tags. It must have a QString of at least "". Succeeding rows have the standard <td>-tags.

• in stream 0 are detail records
• in stream 1 are master records
• in stream 2 adds to masters
• in stream 3 delete from masters
• out stream 0 are matched records
• out stream 1 are unmatched detail records
• out stream 2 are unmatched or counted master records
• out stream 3 deleted masters
• out stream 4 duplicate masters
• out stream 5 unmatched master deletes
• lookup does not consider an unconnected output stream an error. It does proprogate EOFs from output streams.

• Pipes for NetRexx only.

• HOLD keeps the last record from each stream, except primary, and uses it if the stream ends.
• TRANSPARENT means that character can be different from the PAD character.
  If omitted, it is the same as PAD character.
• dstream can be used instead of a non-primary stream.
• (1) NetRexx Pipelines only
• (2) same as highest (+1) stream; implies HOLD

• Records are parsed via the parse_template_delimited_string.
• Variables are named $n, where n is 1 to 9.
• The values of the variables are put into the output_template_delimited_string replacing $n.
• For a literal $n that won't be changed, use $$n.
• NetRexx Pipelines only.
• .
• Example:
    parse / 2 $1 +1/ /The second letter is "$1". $$1 won't be changed./

• (1) Can be before PAD/NOPAD. Depreciated.
• (2) The backslash (\) may need to be escaped, doubled, in some systems shells.

• Records are parsed via the parse_delimited_string.
• Variables are named $n, where n is 1 to 9.
• The values of the variables are put into the logic_delimited_string replacing $n and evaluated. If TRUE, the record is put out on the stream numbered by the dstring's position.
• The stream for a Dstring of ELSE is used if no previous logic Dstring is TRUE.
• If there is no specific ELSE, there is an implied one at the end; if that stream is not connected, the record is discarded.
• If ONE then the record is put out on, at most, one stream: the first one matched.
• If ALL then the record is put out on all streams matched.
• If SINGLE then the records are all put out on the primary output stream.
• The parse_delimited_string and logic_delimited_string(s) follow normal NetRexx rules.
• (1) Implied ELSE after last specified dstring.
• (2) Up to 10 logic_Dstrings may be specified to go to up to 11 ouput streams (including an implied ELSE).
• Not implemented in CMS Pipelines.

Pickparse permits selecting records by a NetRexx logical expression, using parts of the record selected by a Rexx PARSE template.

A simple example has two delimited strings, a Rexx template and a logical expression:

pickparse / . . $3 . 50 $5 +5 / / $3 < $5 /

The parse template selects the 3rd word, and the 5 characters starting in column 50. the variable names are a dollar sign and a digit. Then those variables can be used in the logic expression. When run, and records matching the logic expression are written to the primary output stream, others to the secondary. If either stream is not connected, the corresponding records are discarded.

There can be multiple logic expressions, each in its own delimited string. Parenthetical expressions may be used. Records are matched to each in turn. Any records matching are written to that output stream, if connected.

With the option ONE, the default, each record is written to one output stream: the first one it matches. With the option ALL, the matching goes on and a record could be written to multiple output streams.

There is an implicit or explicit ELSE as the last logic expression. Records that have not matched any of the previous expressions match this and are written or discarded depending on if the stream is connected or not.

The parse template can define up to 9 separate zones, $1 to $9. The variables $_n are also available for the logic expressions; they are the values from the previous record. Initially these are "".

There can be up to 10 output streams defined, and up to 9 logic expressions plus ELSE.

• Blocks its primary input and excutes stage supplied as an argument. The output from this stage are put to the primary output stream. When its compete the primary input is shorted.
• Not implemented in CMS Pipelines.

• NetRexx Pipelines only.
• Records matching the RegEx are put out on primary output.
• Records not matching are put out on secondary, if connected, or discarded.
• .
• (1) Regex_string is a Java RegEx expresion. Null string passes all records.
• (2) Records are prefaced with records number, 10 characters, right justified.
• (3) Number of records put out after a matching record.
• (4) Number of records put out before and after a matching record.
• (5) Inserted before a group of "before records" or the found record with "after records."
• (6) Send all matching records (no numbers) to tertiary output stream, if connected.
• (7) Only a count of matches is put out on the primary output stream. (Other options probably should not be used with this.)

• {class} if class is specified deserialize input to objects of this type otherwise serialize input objects.
• Pipes for NetRexx only.
• For some reason readObject does not like more than one object network in its stream. Block multiple objects. see examples/sertest.njp

• (1) May come before inputRange, for backwards compatability.
• (2) Requires that you implement another sortClass with a name begining with 'sort'
• (3) Suppresses error message if only one record to sort for Rexx objects.
• Uses sortClass class as Interface Class for Generic Sort Objects
  and sortRexx class to Sort Rexx Text Objects

• Outputs a record: <tag string>, then passes through all records on its primary input, and finally a record: </tag>.
• String may be empty.

• Outputs a record for each delimitedString: <delimitedString>, then passes through all records on its primary input, and finally a record for each, in reverse order: </first_word_of_delimitedString>.
• Any delimitedString may be a single word.

• NetRexx Pipelines only; not CMS Pipelines