Spark1.4源码走读笔记之隐式转换

RDD源码中隐式转换

def distinct(numPartitions: Int)(implicit ord: Ordering[T] = null): RDD[T] = withScope {

map(x => (x, null)).reduceByKey((x, y) => x, numPartitions).map(_._1)

}

使用了隐式参数，运行时不需要手动提供参数ord，运行时会根据上下文注入参数

def groupBy[K](f: T => K)(implicit kt: ClassTag[K]): RDD[(K, Iterable[T])] = withScope {

groupBy[K](f, defaultPartitioner(this))

}

def groupBy[K](

f: T => K,

numPartitions: Int)(implicit kt: ClassTag[K]): RDD[(K, Iterable[T])] = withScope {

groupBy(f, new HashPartitioner(numPartitions))

}