GPT Cookbook: Workflow 3

Reproject-Collocate-BandMaths
Subset-LandWater-StatisticsOp


Creates a new file with math band(s) based on an input source file with the option to retain source file bands in the new file.


            --> Write    --> Write    --> Write    --> Write    --> Write
Read    --> Reproject    --> Collocate    --> BandMaths    --> Subset    --> LandWater      
Read    --> Reproject    -->                        
            --> Write                --> StatisticsOp      



This is a large example using a multitude of operators. It enables a comparison of two distinct level2 files.

This example takes two input level2 files then:
    1. Reprojects each file applying quality masking and uses a Lambert Azimuthal Equal Area projection
    2. Collocates the reprojected files (with file1 treated as the reference crs)
    3. Adds two math bands (a difference band and ratio band of two desired primary bands)
    4  Subsets to reduce to only the two primary bands, plus the difference band and ratio band
    5. Adds land and water mask
    6. Statistics on the two bands, plus the difference band and ratio band
    7. Writes out the two reprojected files (NetCDF)
    8  Writes out the collocated file (NetCDF)
    9. Writes out the mathband file (NetCDF)
    10. Writes out the subset file (NetCDF)
    11. Writes out the landWater file (both NetCDF and BEAM-DIMAP*)

    * Masks do not get saved when writing out in formats other than BEAM-DIMAP

For more on the command line parameters used here see commented notes in "WorkflowEx03.par".

Note: Many of the parameters used in this example have the same value for each band and could each be consolidated into a single parameter. The separation of these parameters (using 'file1Type' and 'file2Type' instead of 'fileType') was done to create a more versatile example.

Note: This example could easily be adapted to compare aggregate files such as level-3 or Mosaic files.


Command Line Entry

For a comparison of sst within two distinct SST suite level2 files (a day and a night file).

gpt WorkflowEx03.xml -p WorkflowEx03_SST.par -Pfile1=A2010283180500.L2_LAC_SST.nc -Pfile2=A2010283070000.L2_LAC_SST.nc -Pfile1Short=A20102831805 -Pfile2Short=A20102830700 -Pfile1Band=sst -Pfile2Band=sst -Phome=$HOME

... or for a comparison of chlor_a within two distinct OC suite level2 files.

gpt WorkflowEx03.xml -p WorkflowEx03_OC.par -Pfile1=A2010283180500.L2_LAC_OC.nc -Pfile2=T2010282154500.L2_LAC_OC.nc -Pfile1Short=A20102831805 -Pfile2Short=T20102821545 -Pfile1Band=chlor_a -Pfile2Band=chlor_a -Phome=$HOME

... or similarly for a comparison of Kd_490 within two distinct OC suite level2 files.

gpt WorkflowEx03.xml -p WorkflowEx03_OC.par -Pfile1=A2010283180500.L2_LAC_OC.nc -Pfile2=T2010282154500.L2_LAC_OC.nc -Pfile1Short=A20102831805 -Pfile2Short=T20102821545 -Pfile1Band=Kd_490 -Pfile2Band=Kd_490 -Phome=$HOME

File = WorkflowEx03.xml

<graph id="WorkflowEx03Graph">
  <version>1.0</version>

    <node id="ReadNode1">
      <operator>Read</operator>
      <sources/>
      <parameters>
        <file>${file1}</file>
      </parameters>
    </node>

    <node id="ReadNode2">
      <operator>Read</operator>
      <sources/>
      <parameters>
        <file>${file2}</file>
      </parameters>
    </node>


    <node id="ReprojectNode1">
      <operator>Reproject</operator>
      <sources>
          <source>ReadNode1</source>
      </sources>

      <parameters>
          <crs>
              PROJCS["Lambert_Azimuthal_Equal_Area / World Geodetic System 1984",
                     GEOGCS["World Geodetic System 1984",
                            DATUM["World Geodetic System 1984",
                                  SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
                            AUTHORITY["EPSG","6326"]],
                     PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
                     UNIT["degree", 0.017453292519943295],
                     AXIS["Geodetic longitude", EAST],
                     AXIS["Geodetic latitude", NORTH]],
              PROJECTION["Lambert_Azimuthal_Equal_Area"],
              PARAMETER["latitude_of_center", ${lat}],
              PARAMETER["longitude_of_center", ${lon}],
              PARAMETER["false_easting", 0.0],
              PARAMETER["false_northing", 0.0],
              UNIT["m", 1.0],
              AXIS["Easting", EAST],
              AXIS["Northing", NORTH]]
          </crs>
          <resampling>${reprojectionResampling}</resampling>
          <orientation>0.0</orientation>
          <pixelSizeX>${pixelSize}</pixelSizeX>
          <pixelSizeY>${pixelSize}</pixelSizeY>
          <orthorectify>false</orthorectify>
          <noDataValue>NaN</noDataValue>
          <includeTiePointGrids>true</includeTiePointGrids>
          <addDeltaBands>false</addDeltaBands>
          <maskExpression>${file1ReprojectMaskExpression}</maskExpression>
          <transferValidPixelExpression>${transferValidPixelExpression}</transferValidPixelExpression>
          <applyValidPixelExpression>${applyValidPixelExpression}</applyValidPixelExpression>
      </parameters>

    </node>



    <node id="ReprojectNode2">
      <operator>Reproject</operator>
      <sources>
          <source>ReadNode2</source>
      </sources>

      <parameters>
          <crs>
              PROJCS["Lambert_Azimuthal_Equal_Area / World Geodetic System 1984",
                     GEOGCS["World Geodetic System 1984",
                            DATUM["World Geodetic System 1984",
                                  SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]],
                            AUTHORITY["EPSG","6326"]],
                     PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
                     UNIT["degree", 0.017453292519943295],
                     AXIS["Geodetic longitude", EAST],
                     AXIS["Geodetic latitude", NORTH]],
              PROJECTION["Lambert_Azimuthal_Equal_Area"],
              PARAMETER["latitude_of_center", ${lat}],
              PARAMETER["longitude_of_center", ${lon}],
              PARAMETER["false_easting", 0.0],
              PARAMETER["false_northing", 0.0],
              UNIT["m", 1.0],
              AXIS["Easting", EAST],
              AXIS["Northing", NORTH]]
          </crs>
          <resampling>${reprojectionResampling}</resampling>
          <orientation>0.0</orientation>
          <pixelSizeX>${pixelSize}</pixelSizeX>
          <pixelSizeY>${pixelSize}</pixelSizeY>
          <orthorectify>false</orthorectify>
          <noDataValue>NaN</noDataValue>
          <includeTiePointGrids>true</includeTiePointGrids>
          <addDeltaBands>false</addDeltaBands>
          <maskExpression>${file2ReprojectMaskExpression}</maskExpression>
          <transferValidPixelExpression>${transferValidPixelExpression}</transferValidPixelExpression>
          <applyValidPixelExpression>${applyValidPixelExpression}</applyValidPixelExpression>
      </parameters>

    </node>


    <node id="CollocateNode">
      <operator>Collocate</operator>
      <sources>
        <master>ReprojectNode1</master>
        <slave>ReprojectNode2</slave>
      </sources>
      <parameters>
          <targetProductType>COLLOCATED</targetProductType>
          <renameMasterComponents>${renameReference}</renameMasterComponents>
          <renameSlaveComponents>${renameDependent}</renameSlaveComponents>
          <masterComponentPattern>${ORIGINAL_NAME}_${file1Short}</masterComponentPattern>
          <slaveComponentPattern>${ORIGINAL_NAME}_${file2Short}</slaveComponentPattern>
          <resamplingType>${collocationResampling}</resamplingType>
      </parameters>
    </node>


    <node id="BandMathsNode">
      <operator>BandMaths</operator>
      <sources>
        <source>CollocateNode</source>
      </sources>
      <parameters>
        <copySourceFile>true</copySourceFile>
        <targetBands>

          <targetBand>
            <name>${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff</name>
            <type>${diffBandType}</type>
            <expression>${file1Band}_${file1Short} - ${file2Band}_${file2Short}</expression>
            <description>${file1Band}_${file1Short} - ${file2Band}_${file2Short}</description>
            <unit>${diffBandUnit}</unit>
            <validExpression>${diffBandValidExpression}</validExpression>
            <noDataValue>${diffBandNoDataValue}</noDataValue>
          </targetBand>

          <targetBand>
            <name>${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</name>
            <type>${diffBandType}</type>
            <expression>if (${file2Band}_${file2Short} != 0) then (${file1Band}_${file1Short} / ${file2Band}_${file2Short}) else NaN</expression>
            <description>if (${file2Band}_${file2Short} != 0) then (${file1Band}_${file1Short} / ${file2Band}_${file2Short}) else NaN</description>
            <unit>${diffBandUnit}</unit>
            <validExpression>${diffBandValidExpression}</validExpression>
            <noDataValue>${diffBandNoDataValue}</noDataValue>
          </targetBand>

        </targetBands>
      </parameters>
    </node>



  <node id="SubsetNode">
    <operator>Subset</operator>
    <sources>
        <source>BandMathsNode</source>
    </sources>

    <parameters>
        <bandNames>${file1Band}_${file1Short},${file2Band}_${file2Short},${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff,${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</bandNames>
    </parameters>

  </node>



  <node id="LandWaterMaskNode">
    <operator>LandWaterMask</operator>
    <sources>
      <source>SubsetNode</source>
    </sources>
    <parameters>
      <resolution>${resolution}</resolution>
      <superSamplingFactor>${superSamplingFactor}</superSamplingFactor>
    </parameters>
  </node>


    <node id="StatisticsOpNode">
      <operator>StatisticsOp</operator>
      <sources>
        <source>SubsetNode</source>
      </sources>
      <parameters>
        <bandConfigurations>
          <bandConfiguration>
            <sourceBandName>${file1Band}_${file1Short}</sourceBandName>
            <validPixelExpression>!nan(${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff)</validPixelExpression>
          </bandConfiguration>
          <bandConfiguration>
            <sourceBandName>${file2Band}_${file2Short}</sourceBandName>
            <validPixelExpression>!nan(${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff)</validPixelExpression>
          </bandConfiguration>
          <bandConfiguration>
            <sourceBandName>${file1Band}_${file1Short}_${file2Band}_${file2Short}_diff</sourceBandName>
          </bandConfiguration>
          <bandConfiguration>
            <sourceBandName>${file1Band}_${file1Short}_${file2Band}_${file2Short}_ratio</sourceBandName>
          </bandConfiguration>
        </bandConfigurations>
        <outputAsciiFile>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${statisticsFileSuffix}.txt</outputAsciiFile>
        <percentiles>${percentiles}</percentiles>
        <numBins>${numBins}</numBins>
        <calculateMedian>${calculateMedian}</calculateMedian>
      </parameters>
    </node>



    <node id="WriteReprojectNode1">
      <operator>Write</operator>
      <sources>
        <source>ReprojectNode1</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${reprojectedFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>


    <node id="WriteReprojectNode2">
      <operator>Write</operator>
      <sources>
        <source>ReprojectNode2</source>
      </sources>
      <parameters>
        <file>${file2Short}_${file2Type}_${reprojectedFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>


    <node id="WriteCollocateNode">
      <operator>Write</operator>
      <sources>
        <source>CollocateNode</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>


    <node id="WriteBandMathsNode">
      <operator>Write</operator>
      <sources>
        <source>BandMathsNode</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>



    <node id="WriteSubsetNode">
      <operator>Write</operator>
      <sources>
        <source>SubsetNode</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>


    <node id="WriteLandWaterMaskNode">
      <operator>Write</operator>
      <sources>
        <source>LandWaterMaskNode</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${landWaterFileSuffix}.${ofileExtension}</file>
        <formatName>${ofileFormat}</formatName>
      </parameters>
    </node>

        <node id="WriteLandWaterMaskNodeFormat2">
      <operator>Write</operator>
      <sources>
        <source>LandWaterMaskNode</source>
      </sources>
      <parameters>
        <file>${file1Short}_${file1Type}_${file2Short}_${file2Type}_${collocatedFileSuffix}_${bandMathsFileSuffix}_${subsetFileSuffix}_${landWaterFileSuffix}.${ofileExtension2}</file>
        <formatName>${ofileFormat2}</formatName>
      </parameters>
    </node>

</graph>

File = WorkflowEx03_SST.par

# Read operator parameters (commented out parameters are optionally being sent in at command line)
# file1=A2010283180500.L2_LAC_SST.nc  -- this is the input file which will act as the reference/master in the collocation step
# file2=A2010283070000.L2_LAC_SST.nc  -- this input file will act as the dependent/slave in the collocation step
ifileExtension=nc


# Write operator parameters (commented out parameters are optionally being sent in at command line)
# file1Short=A283180500  -- this name gets used as the suffix for the bands collocated from file1, and for constructing the collocated output file name
# file2Short=A283070000  -- this name gets used as the suffix for the bands collocated from file2, and for constructing the collocated output file name
# file1Type and file2Type is used in constructing the output file names
# These 4 parameters are unique to SST products.  To compare OC or IOP files you would need to alter these 4 parameters
file1Type=L2_LAC_SST
file2Type=L2_LAC_SST
#file1Band=sst
#file2Band=sst

reprojectedFileSuffix=Reprojected
collocatedFileSuffix=Collocated
bandMathsFileSuffix=BandMaths
subsetFileSuffix=Subset
landWaterFileSuffix=LandWater

ofileFormat=NetCDF-CF
ofileExtension=nc
ofileFormat2=BEAM-DIMAP
ofileExtension2=dim


# Reproject operator parameters
lat=37.0
lon=-76.0
pixelSize=1000.0
reprojectionResampling=Bilinear
transferValidPixelExpression=true
applyValidPixelExpression=true

# The reprojectMaskExpression is unique to SST products and flags.  To compare OC or IOP files you would need to alter this expression
file1ReprojectMaskExpression=(qual_sst == 0 or qual_sst == 1) and !l2_flags.HISATZEN
file2ReprojectMaskExpression=(qual_sst == 0 or qual_sst == 1) and !l2_flags.HISATZEN


# Collocate operator parameters
renameReference=true
renameDependent=true
collocationResampling=NEAREST_NEIGHBOUR


# BandMaths operator parameters
diffBandType=float64
diffBandUnit=Celsius
diffBandValidExpression=
diffBandNoDataValue=NaN


# LandWaterMask operator parameters
resolution=1000
superSamplingFactor=3


# StatisticsOp operator parameters
percentiles=80,85,90
numBins=1000
calculateMedian=true
statisticsFileSuffix=Statistics


# WriteImage operator parameters
writeImageFormat=png
cpdFilePath=.seadas/beam-ui/auxdata/color-palettes/anomalies1_universal.cpd
cpdAutoDistribute=false
colourScaleMin=-4.0
colourScaleMax=4.0
level=0
scaleType=linear

File = WorkflowEx03_OC.par

# Read operator parameters (commented out parameters are optionally being sent in at command line)
# file1=A2010283180500.L2_LAC_SST.nc  -- this is the input file which will act as the reference/master in the collocation step
# file2=A2010283070000.L2_LAC_SST.nc  -- this input file will act as the dependent/slave in the collocation step
ifileExtension=nc


# Write operator parameters (commented out parameters are optionally being sent in at command line)
# file1Short=A283180500  -- this name gets used as the suffix for the bands collocated from file1, and for constructing the collocated output file name
# file2Short=A283070000  -- this name gets used as the suffix for the bands collocated from file2, and for constructing the collocated output file name
# file1Type and file2Type is used in constructing the output file names
# These 4 parameters are unique to OC products.  To compare SST or IOP files you would need to alter these 4 parameters
file1Type=L2_LAC_OC
file2Type=L2_LAC_OC
#file1Band=chlor_a
#file2Band=chlor_a

reprojectedFileSuffix=Reprojected
collocatedFileSuffix=Collocated
bandMathsFileSuffix=BandMaths
subsetFileSuffix=Subset
landWaterFileSuffix=LandWater

ofileFormat=NetCDF-CF
ofileExtension=nc
ofileFormat2=BEAM-DIMAP
ofileExtension2=dim


# Reproject operator parameters
lat=37.0
lon=-76.0
pixelSize=1000.0
reprojectionResampling=Bilinear
transferValidPixelExpression=true
applyValidPixelExpression=true

# The reprojectMaskExpression is unique to OC products and flags.  To compare SST files you would need to alter this expression
file1ReprojectMaskExpression=!l2_flags.HIGLINT and !l2_flags.HISATZEN and !l2_flags.LOWLW and !l2_flags.HILT
file2ReprojectMaskExpression=!l2_flags.HIGLINT and !l2_flags.HISATZEN and !l2_flags.LOWLW and !l2_flags.HILT


# Collocate operator parameters
renameReference=true
renameDependent=true
collocationResampling=NEAREST_NEIGHBOUR


# BandMaths operator parameters
diffBandType=float64
diffBandUnit=Celsius
diffBandValidExpression=
diffBandNoDataValue=NaN


# LandWaterMask operator parameters
resolution=1000
superSamplingFactor=3


# StatisticsOp operator parameters
percentiles=80,85,90
numBins=1000
calculateMedian=true
statisticsFileSuffix=Statistics


# WriteImage operator parameters
writeImageFormat=png
cpdFilePath=.seadas/beam-ui/auxdata/color-palettes/anomalies1_universal.cpd
cpdAutoDistribute=false
colourScaleMin=-4.0
colourScaleMax=4.0
level=0
scaleType=linear