When to use python in FME

When to use python in FME

• 1.
  When to usePython in FME Dr. Jonathan Stanger Senior Consultant – Spatial Partners Pty. Ltd.
• 2.
  FME is goodfor Broad User Skill Levels FME is good for Maintainability FME is good for Portability FME is good for Rapid Modification
• 3.
  Why using Pythonis bad… The FME transformer library is very extensive. FME does limit us, but with good reason. It uses a simple data paradigm that aligns with most spatial data, collections of features that each have a well defined set of attributes. We perform operations discretely on a single feature at a time. Sometimes this can be frustrating but remember: Don’t reinvent the wheel! Try rephrasing the problem you are solving!
• 4.
  Where can Pythonbe used? ● Start-up Scripts – Executes before FME transformers (even readers…) ● Shut-down Scripts – Executes after all FME transformers are finished ● Scripted Parameters – Executes on start-up and preserves the value ● PythonCreator – Has no input, Python defines all features outputted ● PythonCaller – Executes function once or calls input once per feature
• 5.
  Good examples forPython in FME
• 6.
  Smart Scripted Parameters Whilethe Workspace Runner is useful, in trivial cases its not always ideal. If the only requirement is glob style directory search to provide file paths the workspace runner leaves excess complexity for deployment. Solution: Script glob search into parameter to return import fme, fmeobjects, glob # Get parameters set by user to initiate the glob search rootpath = fme.macroValues['rootPath'] # Glob search for file candidates candidates = glob.glob(rootpath + '/**/data.TAB', recursive = True) # Return string that matches what FME expects if candidates: return '""' + '" "'.join(candidates) + '""' else: return "" Example: Avoiding the workspace runner
• 7.
  Front Stage FileManipulation FME rightly treats the data schema of some file formats quite strictly, unlike users sometimes. This impacts MapInfo and sometimes Excel files. Solution: Rename/copy the file to a standard filename - Published Parameter (Target File) - Private Parameter (Reader Target) import fme, fmeobjects from shutil import copy2 # Extract required info from macroValues fmepath = fme.macroValues['FME_MF_DIR'] targetfile = fme.macroValues['fileName'] # Check parameter is assigned if targetfile: # Copy2 will automatically overwrite existing # file. Set reader target to private parameter # containing: # $(FME_MF_DIR)Standard.xlsx copy2(targetfile, fmepath + 'Standard.xlsx') Example: File Formats playing badly
• 8.
  Inter-feature Dependant Operations Assumingfeatures have ID’s that are not consecutive. If features have two attributes, one representing the component ahead and one the component behind, how do we sort the features? Solution: Sort elements by matching ahead ID to behind ID Example: Paired Sorting Sort Order 1 Behind 934AH9 Ahead 365U33 Sort Order 2 Behind 6433OE Ahead 934AH9 Sort Order 3 Behind 18KW22 Ahead 6433OE Sort Order 4 Behind 20554Z Ahead 18KW22
• 9.
  Repeated(recursive) operations (N> 3 or N = ?) FME was designed to work with tabular not network/graph style data. As such, recursive algorithms such as DFS or BFS are nearly impossible to implement with native transformers. Solution: Gather all the data into Python then process in memory Think carefully about how you output! Example: Trace network data A B D E C GFH I
• 10.
  Non-standard Geometry Manipulation FMEoffers well developed options for direct geometry manipulation, scaling, offsetting or rotating. Less typical geometry manipulation can be tricky though. Solution: Implement custom geometry transformation in Python Example: Mirror around arbitrary plane geom = feature.getGeometry() points = line.getPoints() x, y, z = point.getXYZ() point.setXYZ(x, y, z)
• 11.
  Tedious Operations For simplecases FME string manipulation is more than sufficient. Building a string conditionally based on 20+ attributes is possible but very tedious and difficult to maintain. Solution: Python string manipulation to efficiently build the string import fme, fmeobjects # Attribute to label lookup dictionary attributeMap = { '5_040': '5pr', '50_040': '50pr', '100_040': '100pr', '200_040': '200pr', '400_040': '400pr', ... x 25 more, } class FeatureProcessor(object): def __init__(self): pass def input(self,feature): # Use dictionary comprehension to conditionally build elements of label attributes = {attributeMap[key]: feature.getAttribute(key) for key in attributeMap.keys() if feature.getAttribute(key) and feature.getAttribute(key) > 0} # Join label elements to form a single string label = ", ".join([str(int(attributes[key])) + "x" + key for key in attributes.keys()]) feature.setAttribute('LABEL', label) self.pyoutput(feature) def close(self): pass Example: Build very complex string
• 12.
  General Rules for Pythonuse ● Recursion ● File Manipulation ● FME forums say its not possible
• 13.
  Hints for Pythonin FME 1. Logger rather than Print 2. Scripted Parameters are hard to Debug 3. Don’t import non-standard libraries 4. You can import FME libraries to external python scripts 5. Don’t use copies of a PythonCaller
• 14.
  import fme import fmeobjects logger= fmeobjects.FMELogFile() parameter = fme.macroValues['ParameterName'] logger.logMessageString("{Python Script} Parameter set: %s" % parameter) # Template Class Interface: # When using this class, make sure its name is set as the value # of the 'Class or Function to Process Features' transformer parameter class FeatureProcessor(object): def __init__(self): pass def input(self, feature): self.pyoutput(feature) def close(self): pass Useful Pattern – FME Env. Dictionary
• 15.
  import fme import fmeobjects #Template Class Interface: # When using this class, make sure its name is set as the value # of the 'Class or Function to Process Features' transformer # parameter class FeatureProcessor(object): def __init__(self): self.featureList = [] def input(self, feature): self.featureList.append(feature) def close(self): for feature in self.featureList: self.pyoutput(feature) Useful Pattern – Bulk Feature Manipulation
• 16.
  Example: Tiled GridViewports PythonCaller: Consumes point and line geometry, coverts to points then divides space to provide viewport polygons that will overlay every point in the set. Useful for creating AutoCAD viewports or other pagination of your geometry.
• 17.
  Example: Procedural TreeLayout for Logical Networks Start-up Python Script: Parses logical connectivity then builds GML file with generated geometry for export via FME to other file formats such as AutoCAD to a database as a pre-calculated UI display geometry
• 18.
  THANK YOU! Dr. JonathanStanger jstanger@spatialpartners.com