Pipeline.pod

NAME

Pipeline - Pipelines for map, filters processes.

SYNOPSIS

reduce(
        fileSource( $ARGV[0] ),
        pipeline( 
                step { decode_json $_ },
                step { $_->{name} },
                bufferedStep { my @v = sort { $a cmp $b } @$_; \@v } 50,
                filter { /^A/ },
                step { $_ =~ s/^A/E/g; $_ },
        ),
        fileSink( $ARGV[1] )
);

DESCRIPTION

Toolbox for data processing using MAP and REDUCE paradigm. Includes helper methods for defining filters...

fileSource( $filename )

Creates a data source for a file and reads it line by line.

step { CODE }

A step in the pipeline.

bufferedStep { CODE }

A step in the pipeline.

filter {CODE}

Adds a filter to the pipeline.

pipeline( \&step1, \&step2, ... )

Defines a data source and a set of transformations. Can be passed as a datasource to another pipeline.

reduce( $src, $pipe, $acc )

Processes stream from pipeline to accumulator function.

AUTHOR

Shawn Tan, shawn@semantics3.com