Package fr.ens.biologie.genomique.eoulsan.modules.mgmt.hadoop

Class DistCp

java.lang.Object
fr.ens.biologie.genomique.eoulsan.modules.mgmt.hadoop.DistCp
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool
public class DistCp extends Object implements org.apache.hadoop.util.Tool
A Map-reduce program to recursively copy directories between different file-systems.
Since:
1.0
Author:
The hadoop project

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    DistCp.DuplicationException
    An exception class for duplicated source files.

  • Constructor Summary

    Constructors
    Constructor
    Description
    DistCp(org.apache.hadoop.conf.Configuration conf)
    Constructor.

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    copy(org.apache.hadoop.conf.Configuration conf, String srcPath, String destPath, org.apache.hadoop.fs.Path logPath, boolean srcAsList, boolean ignoreReadFailures)
    Deprecated.
    org.apache.hadoop.conf.Configuration
    getConf()
     
    static String
    getRandomId()
    Get a random id.
    int
    run(String[] args)
    This is the main driver for recursively copying directories across file systems.
    void
    runWithException(String[] args)
    This is the main driver for recursively copying directories across file systems.
    void
    setConf(org.apache.hadoop.conf.Configuration conf)
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • DistCp

      public DistCp(org.apache.hadoop.conf.Configuration conf)
      Constructor.
      Parameters:
      conf - Hadoop configuration

  • Method Details

    • setConf

      public void setConf(org.apache.hadoop.conf.Configuration conf)
      Specified by:
      setConf in interface org.apache.hadoop.conf.Configurable

    • getConf

      public org.apache.hadoop.conf.Configuration getConf()
      Specified by:
      getConf in interface org.apache.hadoop.conf.Configurable

    • copy

      @Deprecated public static void copy(org.apache.hadoop.conf.Configuration conf, String srcPath, String destPath, org.apache.hadoop.fs.Path logPath, boolean srcAsList, boolean ignoreReadFailures) throws IOException
      Deprecated.
      Copy a file.
      Parameters:
      conf - Hadoop configuration
      srcPath - source
      destPath - destination
      logPath - log path
      srcAsList - source is a list
      ignoreReadFailures - true to ignore read faillure
      Throws:
      IOException - if an error occurs while copying data

    • run

      public int run(String[] args)
      This is the main driver for recursively copying directories across file systems. It takes at least two cmdline parameters. A source URL and a destination URL. It then essentially does an "ls -lR" on the source URL, and writes the output in a round-robin manner to all the map input files. The mapper actually copies the files allotted to it. The reduce is empty.
      Specified by:
      run in interface org.apache.hadoop.util.Tool

    • runWithException

      public void runWithException(String[] args) throws EoulsanException
      This is the main driver for recursively copying directories across file systems. It takes at least two cmdline parameters. A source URL and a destination URL. It then essentially does an "ls -lR" on the source URL, and writes the output in a round-robin manner to all the map input files. The mapper actually copies the files allotted to it. The reduce is empty.
      Parameters:
      args - arguments
      Throws:
      EoulsanException - if an error occurs

    • getRandomId

      public static String getRandomId()
      Get a random id.
      Returns:
      a String with a random id