MapReduce怎么实现气象站计算最低或最高温度

这篇文章主要介绍"MapReduce怎么实现气象站计算最低或最高温度"，在日常操作中，相信很多人在MapReduce怎么实现气象站计算最低或最高温度问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答"MapReduce怎么实现气象站计算最低或最高温度"的疑惑有所帮助！接下来，请跟着小编一起来学习吧！

TemperatureMR.java

package cn.kissoft.hadoop.week05;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class TemperatureMR {    public static void main(String[] args) throws Exception {        if (args.length != 2) {            System.err.println("Usage: Temperature  
");            System.exit(-1);        }        Job job = new Job();        job.setJarByClass(TemperatureMR.class);        job.setJobName("Max and min temperature");        FileInputFormat.addInputPath(job, new Path(args[0]));        FileOutputFormat.setOutputPath(job, new Path(args[1]));        job.setMapperClass(TemperatureMapper.class);        job.setReducerClass(TemperatureReducer.class);        job.setOutputKeyClass(Text.class);        job.setOutputValueClass(IntWritable.class);        System.exit(job.waitForCompletion(true) ? 0 : 1);    }}

TemperatureMapper.java

package cn.kissoft.hadoop.week05;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;public class TemperatureMapper extends Mapper {    private static final int MISSING = 9999;    @Override    public void map(LongWritable key, Text value, Context context)            throws IOException, InterruptedException {        String line = value.toString();        String year = line.substring(0, 4);        int airTemperature = Integer.parseInt(line.substring(13, 19).trim());        if (Math.abs(airTemperature) != MISSING) {            context.write(new Text(year), new IntWritable(airTemperature));        }    }}

MaxTemperatureReducer.java

package cn.kissoft.hadoop.week05;import java.io.IOException;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Reducer;public class TemperatureReducer extends Reducer {    @Override    public void reduce(Text key, Iterable values, Context context)            throws IOException, InterruptedException {        int maxValue = Integer.MIN_VALUE;        int minValue = Integer.MAX_VALUE;        for (IntWritable value : values) {            maxValue = Math.max(maxValue, value.get());            minValue = Math.min(minValue, value.get());        }        context.write(key, new IntWritable(maxValue));        context.write(key, new IntWritable(minValue));    }}

运行过程

[wukong@bd11 guide]$ hadoop jar pc.jar cn.kissoft.hadoop.week05.TemperatureMR ./ch02/1959.txt ./ch02/out/Warning: $HADOOP_HOME is deprecated.14/08/15 16:29:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.14/08/15 16:29:32 INFO input.FileInputFormat: Total input paths to process : 114/08/15 16:29:32 INFO util.NativeCodeLoader: Loaded the native-hadoop library14/08/15 16:29:32 WARN snappy.LoadSnappy: Snappy native library not loaded14/08/15 16:29:34 INFO mapred.JobClient: Running job: job_201408151617_000314/08/15 16:29:35 INFO mapred.JobClient:  map 0% reduce 0/08/15 16:29:47 INFO mapred.JobClient:  map 100% reduce 0/08/15 16:30:00 INFO mapred.JobClient:  map 100% reduce 100/08/15 16:30:04 INFO mapred.JobClient: Job complete: job_201408151617_000314/08/15 16:30:04 INFO mapred.JobClient: Counters: 2914/08/15 16:30:04 INFO mapred.JobClient:   Job Counters 14/08/15 16:30:04 INFO mapred.JobClient:     Launched reduce tasks=114/08/15 16:30:04 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=1498914/08/15 16:30:04 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=014/08/15 16:30:04 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=014/08/15 16:30:04 INFO mapred.JobClient:     Launched map tasks=114/08/15 16:30:04 INFO mapred.JobClient:     Data-local map tasks=114/08/15 16:30:04 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=1282514/08/15 16:30:04 INFO mapred.JobClient:   File Output Format Counters 14/08/15 16:30:04 INFO mapred.JobClient:     Bytes Written=1914/08/15 16:30:04 INFO mapred.JobClient:   FileSystemCounters14/08/15 16:30:04 INFO mapred.JobClient:     FILE_BYTES_READ=918048614/08/15 16:30:04 INFO mapred.JobClient:     HDFS_BYTES_READ=2754447514/08/15 16:30:04 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1388690814/08/15 16:30:04 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1914/08/15 16:30:04 INFO mapred.JobClient:   File Input Format Counters 14/08/15 16:30:04 INFO mapred.JobClient:     Bytes Read=2754436814/08/15 16:30:04 INFO mapred.JobClient:   Map-Reduce Framework14/08/15 16:30:04 INFO mapred.JobClient:     Map output materialized bytes=459024014/08/15 16:30:04 INFO mapred.JobClient:     Map input records=44426414/08/15 16:30:04 INFO mapred.JobClient:     Reduce shuffle bytes=459024014/08/15 16:30:04 INFO mapred.JobClient:     Spilled Records=125188214/08/15 16:30:04 INFO mapred.JobClient:     Map output bytes=375564614/08/15 16:30:04 INFO mapred.JobClient:     Total committed heap usage (bytes)=21886566414/08/15 16:30:04 INFO mapred.JobClient:     CPU time spent (ms)=628014/08/15 16:30:04 INFO mapred.JobClient:     Combine input records=014/08/15 16:30:04 INFO mapred.JobClient:     SPLIT_RAW_BYTES=10714/08/15 16:30:04 INFO mapred.JobClient:     Reduce input records=41729414/08/15 16:30:04 INFO mapred.JobClient:     Reduce input groups=114/08/15 16:30:04 INFO mapred.JobClient:     Combine output records=014/08/15 16:30:04 INFO mapred.JobClient:     Physical memory (bytes) snapshot=32298598414/08/15 16:30:04 INFO mapred.JobClient:     Reduce output records=214/08/15 16:30:04 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=145557913614/08/15 16:30:04 INFO mapred.JobClient:     Map output records=417294

运行结果

[wukong@bd11 guide]$ hadoop fs -cat ./ch02/out/part-r-00000

Warning: $HADOOP_HOME is deprecated.1959    4181959    -400

截图

到此，关于"MapReduce怎么实现气象站计算最低或最高温度"的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注网站，小编会继续努力为大家带来更多实用的文章！