Package org.openrefine.runners.local.pll

Class MapPartitionsPLL<U,T>

java.lang.Object

org.openrefine.runners.local.pll.PLL<T>

org.openrefine.runners.local.pll.MapPartitionsPLL<U,T>

Type Parameters:

U -

T -

public class MapPartitionsPLL<U,T>
extends PLL<T>

A PLL obtained by transforming each partition independently. The supplied function takes two arguments: the first is the index of the partition to map, the second is a stream of its contents.

Author:

Antonin Delpeuch

Nested Class Summary

Nested classes/interfaces inherited from class org.openrefine.runners.local.pll.PLL

PLL.LastFlush, PLL.PLLExecutionError

Field Summary

Fields

Modifier and Type	Field	Description
protected BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>>	mapFunction
protected PLL<U>	parent
protected boolean	preservesSizes

Fields inherited from class org.openrefine.runners.local.pll.PLL

cachedPartitions, context, id, name

Constructor Summary

Constructors

Modifier	Constructor	Description
	MapPartitionsPLL(PLL<U> parent, BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>> mapFunction, String mapFunctionDescription)	Constructs a PLL with partitions derived one-to-one from the parent.
protected	MapPartitionsPLL(PLL<U> parent, BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>> mapFunction, String mapFunctionDescription, boolean preservesSizes)	Constructs a PLL with partitions derived one-to-one from the parent.

Method Summary

Modifier and Type	Method	Description
CloseableIterator<T>	compute(Partition partition)	Iterate over the elements of the given partition.
io.vavr.collection.Array<Long>	computePartitionSizes()
List<PLL<?>>	getParents()	Returns the PLLs that this PLL depends on, to compute its contents.
io.vavr.collection.Array<? extends Partition>	getPartitions()
boolean	hasCachedPartitionSizes()	Is this PLL aware of the size of its partitions?

Methods inherited from class org.openrefine.runners.local.pll.PLL

aggregate, batchPartitions, cacheAsync, collect, collectPartitionsAsync, concatenate, concatenate, count, dropFirstElements, dropLastElements, filter, flatMap, getContext, getId, getPartitionSizes, getQueryTree, isCached, isEmpty, iterate, iterateFromPartition, iterator, limitPartitions, map, mapPartitions, mapToPair, mapToPair, numPartitions, retainPartitions, runOnPartitions, runOnPartitions, runOnPartitionsAsync, runOnPartitionsAsync, runOnPartitionsWithoutInterruption, runOnPartitionsWithoutInterruption, saveAsTextFile, saveAsTextFileAsync, scanMap, scanMapStream, sort, take, toString, uncache, withCachedPartitionSizes, writeOriginalPartition, writePartition, writePlannedPartition, zipWithIndex

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

parent

protected final PLL<U> parent

mapFunction

protected final BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>> mapFunction

preservesSizes

protected final boolean preservesSizes

Constructor Detail

MapPartitionsPLL

public MapPartitionsPLL(PLL<U> parent,
                        BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>> mapFunction,
                        String mapFunctionDescription)

Constructs a PLL with partitions derived one-to-one from the parent.

Parameters:

parent - the source list

mapFunction - the function to apply to each partition

mapFunctionDescription - a short description of the map function for debugging purposes

MapPartitionsPLL

protected MapPartitionsPLL(PLL<U> parent,
                           BiFunction<Integer,CloseableIterator<U>,CloseableIterator<T>> mapFunction,
                           String mapFunctionDescription,
                           boolean preservesSizes)

Constructs a PLL with partitions derived one-to-one from the parent.

Parameters:

parent - the source list

mapFunction - the function to apply to each partition

mapFunctionDescription - a short description of the map function for debugging purposes

preservesSizes - whether partition sizes are know to be the same as the parent's

Method Detail

hasCachedPartitionSizes

public boolean hasCachedPartitionSizes()

Description copied from class: PLL

Is this PLL aware of the size of its partitions?

Overrides:

hasCachedPartitionSizes in class PLL<T>

computePartitionSizes

public io.vavr.collection.Array<Long> computePartitionSizes()

Overrides:

computePartitionSizes in class PLL<T>

compute

public CloseableIterator<T> compute(Partition partition)

Description copied from class: PLL

Iterate over the elements of the given partition. This is the method that should be implemented by subclasses. As this method forces computation, ignoring any caching, consumers should not call it directly but rather use PLL.iterate(Partition). Once the iterator is not needed anymore, it should be closed. This makes it possible to release the underlying resources supporting it, such as open files or sockets.

Specified by:

compute in class PLL<T>

Parameters:

partition - the partition to iterate over

Returns:

getPartitions

public io.vavr.collection.Array<? extends Partition> getPartitions()

Specified by:

getPartitions in class PLL<T>

Returns:

the partitions in this list

getParents

public List<PLL<?>> getParents()

Description copied from class: PLL

Returns the PLLs that this PLL depends on, to compute its contents. This is used for debugging purposes, to display the tree of dependencies of a given PLL.

Specified by:

getParents in class PLL<T>

See Also:

PLL.getQueryTree()