java - Geotools cannot find HSQL EPSG DB, throws error: NoSuchAuthorityCodeException

Question

I am running Geotools inside of Apache Storm, and having a heck of a time with geotools dependency management. Every thing works when I run the storm cluster locally (windows 7 laptop), but when I deploy to the cluster I get this exception in my logs.

EDIT: Here is the Java I use that calls this functionality

      GridCoverage2D image = 
new GeoTiffReader(f).read(new GeneralParameterValue[]{policy, gridsize, useJaiRead});
        /**
         * reproject to WGS84
         */
        CoordinateReferenceSystem targetCRS = CRS.decode("EPSG:4326");
        GridCoverage2D reprojectedImage = (GridCoverage2D) Operations.DEFAULT.resample(image, targetCRS);

Here's my environment

geotools 11.1
java 7
POM is below
running on windows 7 when in local mode (works perfectly here)
this problem happens on Ubuntu 12.04.5 LTS (GNU/Linux 3.2.0-63-virtual x86_64)

here's the exception from the log


[ERROR] Exception in Bolt org.geotools.data.DataSourceException: GEOTIFF Module Error Report
No code "EPSG:32637" from authority "EPSG" found for object of type "EngineeringCRS".
ModelPixelScaleTag: [2.0,2.0,0.0]
ModelTiePointTag: (1 tie points)
TP #0: [0.0,0.0,0.0] -> [337668.0,3837288.0,0.0]
ModelTransformationTag: NOT AVAILABLE
GeoKey #1: Key = 2049, Value = GCS_WGS_1984
GeoKey #2: Key = 2054, Value = 9102
GeoKey #3: Key = 3072, Value = 32637
GeoKey #4: Key = 1024, Value = 1
GeoKey #5: Key = 1025, Value = 1
GeoKey #6: Key = 1026, Value = PCS Name = WGS_1984_UTM_zone_37N
GeoKey #7: Key = 3076, Value = 9001
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:32637" from authority "EPSG" found for object of type "EngineeringCRS".
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.noSuchAuthorityException(CartesianAuthorityFactory.java:136)
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createEngineeringCRS(CartesianAuthorityFactory.java:130)
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createCoordinateReferenceSystem(CartesianAuthorityFactory.java:121)
        at org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:802)
        at org.geotools.coverage.grid.io.imageio.geotiff.GeoTiffMetadata2CRSAdapter.createProjectedCoordinateReferenceSystem(GeoTiffMetadata2CRSAdapter.java:284)
        at org.geotools.coverage.grid.io.imageio.geotiff.GeoTiffMetadata2CRSAdapter.createCoordinateSystem(GeoTiffMetadata2CRSAdapter.java:205)
        at org.geotools.gce.geotiff.GeoTiffReader.getHRInfo(GeoTiffReader.java:299)
        at org.geotools.gce.geotiff.GeoTiffReader.<init>(GeoTiffReader.java:211)
        at org.geotools.gce.geotiff.GeoTiffReader.<init>(GeoTiffReader.java:156)
        at dgi.eii.utils.PixelExtractor.extract(PixelExtractor.java:80)
        at dgi.eii.storm.bolts.RasterPixelExtractorBolt.execute(RasterPixelExtractorBolt.java:59)
        at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
        at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
        at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
        at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
        at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Thread.java:745)

I also get the same error when encountering an image with EPSQ:4326

 [ERROR] Exception in Bolt org.geotools.data.DataSourceException: GEOTIFF Module Error Report
No code "EPSG:4326" from authority "EPSG" found for object of type "EngineeringCRS".
ModelPixelScaleTag: [2.0230196490091333E-5,2.0230196490071028E-5,0.0]
ModelTiePointTag: (1 tie points)
TP #0: [0.0,0.0,0.0] -> [36.8167576323252,34.429979601192464,0.0]
ModelTransformationTag: NOT AVAILABLE
GeoKey #1: Key = 2048, Value = 4326
GeoKey #2: Key = 2049, Value = GCS_WGS_1984
GeoKey #3: Key = 2054, Value = 9102
GeoKey #4: Key = 1024, Value = 2
GeoKey #5: Key = 2057, Value = 6378137.0
GeoKey #6: Key = 1025, Value = 1
GeoKey #7: Key = 2059, Value = 298.257223563
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "EPSG" found for object of type "EngineeringCRS".
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.noSuchAuthorityException(CartesianAuthorityFactory.java:136)
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createEngineeringCRS(CartesianAuthorityFactory.java:130)
        at org.geotools.referencing.factory.epsg.CartesianAuthorityFactory.createCoordinateReferenceSystem(CartesianAuthorityFactory.java:121)
        at org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:802)
        at org.geotools.coverage.grid.io.imageio.geotiff.GeoTiffMetadata2CRSAdapter.createGeographicCoordinateReferenceSystem(GeoTiffMetadata2CRSAdapter.java:389)
        at org.geotools.coverage.grid.io.imageio.geotiff.GeoTiffMetadata2CRSAdapter.createCoordinateSystem(GeoTiffMetadata2CRSAdapter.java:208)
        at org.geotools.gce.geotiff.GeoTiffReader.getHRInfo(GeoTiffReader.java:299)
        at org.geotools.gce.geotiff.GeoTiffReader.<init>(GeoTiffReader.java:211)
        at org.geotools.gce.geotiff.GeoTiffReader.<init>(GeoTiffReader.java:156)
        at dgi.eii.utils.PixelExtractor.extract(PixelExtractor.java:80)
        at dgi.eii.storm.bolts.RasterPixelExtractorBolt.execute(RasterPixelExtractorBolt.java:59)
        at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
        at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
        at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
        at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
        at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
        at clojure.lang.AFn.run(AFn.java:24)
        at java.lang.Thread.run(Thread.java:745)

I have seen several answers that indicated I need to put the gt-epsg-hsql in my pom, and I have, still doesn't find what it needs.

Here are my geotools POM entries

 <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-swing</artifactId>
            <version>11.1</version>        
        </dependency>
         <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-opengis</artifactId>
            <version>11.1</version>        
        </dependency>
        <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-epsg-hsql</artifactId>
            <version>11.1</version>                 
        </dependency>
        <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-geotiff</artifactId>
            <version>11.1</version>
        </dependency>
        <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-image</artifactId>
            <version>11.1</version>
        </dependency>
        <dependency>
            <groupId>org.geotools</groupId>
            <artifactId>gt-wms</artifactId>
            <version>11.1</version>
        </dependency>

here is how I am building the uber-jar with maven shade plugin

 <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <archive>
                        <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
                    </archive>
                    <!--  <minimizeJar>true</minimizeJar>-->
                    <shadedArtifactAttached>true</shadedArtifactAttached>
                    <shadedClassifierName>stand-alone</shadedClassifierName>
                    <artifactSet>
                        <excludes>                 
                            <exclude>org.slf4j:slf4j-api:jar:</exclude>
                            <exclude>org.slf4j:slf4j-simple:jar:1.6.4:jar:</exclude>
                            <exclude>org.slf4j:slf4j-log4j12:jar:</exclude>
                            <exclude>org.slf4j:jcl-over-slf4j:jar:</exclude>
                            <exclude>org.slf4j:slf4j-api:jar:1.7.5:jar:</exclude>
                            <!-- <exclude>org.slf4j*:</exclude>-->
                            <exclude>commons-logging:commons-logging:jar:</exclude>
                            <exclude>commons-logging:commons-logging-api:jar:</exclude>
                        </excludes>
                    </artifactSet>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <finalName>storm-topos</finalName>
                    <transformers>
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                            <mainClass>dgi.eii.storm.base.StormTopologyRunner</mainClass>
                        </transformer>
                       <!-- <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                            <resource>META-INF/spring.handlers</resource>
                        </transformer>
                        <transformer
                            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
              

Answer 1

Geotools uses Java's Service infrastructure to load the class responsible for EPSG lookups. The original gt-epsg-hsql.jar has entries in /META-INF/services/ which specify which interfaces are implemented by the jar-file, and which can be dynamically loaded at runtime.

When building a uber-jar, you combine multiple jar files into one. Most likely, another jar file implements the same interfaces as well (for example gt-referencing.jar) and has thus files with the same names in its /META-INF/services/. When putting everything into one jar file, those entries will very likely be overwritten (at least I couldn't find any reference that the maven-shade-plugin merges such services files).

You could verify that by looking at the services-directory in the created uber-jar, especially at the entry /META-INF/services/org.opengis.referencing.crs.CRSAuthorityFactory. Both gt-epsg-hsql.jar and gt-referencing.jar have such a file (and other jar-files from GeoTools probably as well), and most likely, only the content of one will be in your uber-jar, resulting in all the other classes not being found/loaded at runtime.

I'm not really familiar with the maven-shade-plugin, but other questions on SO (like [1]) suggest to use an additional transformer:

<transformer
   implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />

---

EDIT: As this answer gets regular visits, and I'm familiar with the shade plugin by now, here is a more detailed guide to use the shade-plugin.

So instead of using the maven-assembly plugin, we can use the maven-shade plugin to create an all-in-one jar. To do so, configure the maven-shade plugin indoor pom.xml and bind it to the package phase (so, whenever you call mvn package, the shaded jar will be created:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <version>3.1.0</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>shade</goal>
          </goals>
          <configuration>
            <transformers>
              <!-- This bit sets the main class for the executable jar as you otherwise -->
              <!-- would with the assembly plugin                                       -->
              <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                <manifestEntries>
                  <Main-Class>com.example.YourMainClass</Main-Class>
                  <Implementation-Vendor>Your Company Name</Implementation-Vendor>
                  <Implementation-Version>${project.version}</Implementation-Version>
                </manifestEntries>
              </transformer>
              <!-- This bit merges the various GeoTools META-INF/services files         -->
              <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
            </transformers>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

The Implementation-Vendor and Implementation-Version may not be needed in all cases, but I observed situations where some code (I think JAI - Java advanced imaging) complained and misbehaved when this information was missing, likely as the original JAR included such information and the shaded one did not by default, so it might be best to just include it.

When running mvn package, it will rename the original jar-file to something like original-myArtifact.jar and place the shaded (=all-in-one, fat-jar, uber-jar) at myArtifact.jar. If you don't like this behavior and want to keep the original jar file intact and have the shaded jar-file separat, add the following line inside the configuration block:

<shadedArtifactAttached>true</shadedArtifactAttached>

In this case, the build process will create a file myArtifact-shaded.jar (similar to myArtifact-jar-with-dependencies.jar created by the assembly plugin).

---

[1] Maven shade + resteasy Could find writer for content-type