ESRI-UC - ESRI 5K Fun Run/Walk

Directions Magazine sponsored the 5K for the second year. The turnout was great (about 320 registered) and the day a bit cooler than in year’s past. Congratulations to Men’s winner Patrick Donnelly (15:58 ) and Women’s winner Susan Sleath (19:14).

In the “Runners in the Press” category, congrats to Rachael Mock of GITC America, who took her age group. Yours truly was 3rd in her age group. My colleague Joe Francica was placed on “injured reserve” after the start.

The best dressed runners were the crew from Imagem Geosistemas, ESRI’s distributor in Brazil. Not only did they sport shirts designed for the event, they had an internal race going on, complete with their own trophies!

Thanks to all who ran and walked for joining us. We look forward to seeing you all again next year. And, bring your friends!

ESRI-UC - Introduction to the new ESRI Image Server presented by Dave McGuire and Peter Becker

What it is

used for imagery management of large quantities of images, process them in a server & distribute to clients

provides fast access and server based processing of raw imagery (10’s of TB potentially) straight off the sensor.

Methodology

store primary data in files (or import to rdbms if required) create services on server
  harvest metadata
  build catalog index
  define processing chain
can publish the image service on the server and multiple users can access can host more than one imagery service on a server

Benefits

rapid access without any pre-processing (orthorectify, radiometric correction, filtering) no dbms load very fast otf server side processing multiple products from single source with different process chains dynamic update of image services as new images become available.
simply copy files to server, add a metadata catalog and you’re ready to go.
Demo

product originates from a product called PromptServer

8500 landsat scenes 8.5 tb of tiff files stored on the server.
image overviews are created by the server (managed overviews) if imagery changes, server recreates the overviews

server extracts pixels from raw source imagery, mosaics, radiometric corrects, pansharpens and reprojection from multiple UTM zones to Lat/Long on the fly, in seconds.

can compute otf a band432 false color PS IR from landsat

can create NDVI otf, each time you pan or zoom - all from the raw landsat data.

(40,000 tiles) can handle SRTM data - create hillshaded terrain surface on the fly

1:500k scanned maps of the world in tiff format. They have been georeferenced, but the collar clipping happens on the fly, as well as a reprojection and a mosaicing. Subsampling only occurs once, so the image quality is fairly pristine.

demo was running on a workstation class machine with 2gb of ram, not a server class machine, and was blazingly fast.

serve quickbird imagery - 16bit basic imagery straight off the digital globe cd - raw imagery. does radiometric correction, reprojection, orthorecitfication (from SRTM) pansharpening and mosaicing into a 60cm natural color image created otf.

can take pan imagery and convert it to bands 421 (false color).
can then compute ndvi and water detection otf.

can change properties of the process chain on the fly, such as subsampling and supersampling properties, etc.
Can edit the metadata information, which does get sent to the client as XML ( how the image was processed, date, etc)

can jpg compress the final output of the processing chain before serving it - for low bandwidth connections. you don’t have to compress your base product - it remains very high resolution. You can use the compressed layer for most of your work and only show the full res product when zoomed in to a very small area.

can be used to serve scanned cadastral drawings and have them projected to a known coordinate system. could be very important for counties without digital parcel gis systems to serve their cadastral information to the national map or state level portals such as NC OneMap.

can load digital camera imagery and metadata (imu, xyx, etc) and with an existing terrain model, have the system do the orthorectification on the fly. great for disaster response situations where you do not have time to wait. you could have online orthos in a matter of days instead of months. we could have used that last fall after all the hurricane damage in western NC.

imagery is not just used for a natural background. Analytical datasets and terrain models are imagery data. much vector data is extraced from imagery and imagery is quite often used for verification of vector data. the new image server will allow much imagery that languishes on disk or tape or in historic archives to be made immediately available to end users on their desktop without the huge overhead of pre-processing, georeferencing, orthorectification, etc.

volume of acquired imagery is growing exponentially, we need systems to provide quick access that doesn’t involve loading into a rdbms.

value of imagery devalues quickly with time, but ironically it
also increases in value again once more time has passed.

paradigm shift merges the processing and distribution stages
of the imagery.
nearly no pre-processing
quick to generate
very low latency
create multiple products from same source
can update very quickly
huge disk space savings
More Demos

how to create an image service:
start catalog manager
name the service
add image(s) to the service (can add whole directories)
set basic options
compute statistics, generate overviews, metadata table & footprints
create the service

can change other properties and processing chains (interpolation,
crop method, transparency, compression, filters, georeferencing,
pan sharpening, histograms, you name it…)

Data Flow

input imagery
process chaining
compress
display

accepts many formats (1 to 32 bits per channel)
tif, geotif
  jpg, lzw packbit
  flat, scanline, tiled (optimal)
 
bil, bsq, bip
jp2000, mrsid, ecw
nitf, hda img(GDAL) (USES GDAL [open source library for imagery - Ed.] INTERNALLY TO DO THE TRANSLATIONS!!!!)
sde raster
oracle georaster
there is an SDK so users can add their own image formats

compression: store compressed or uncompressed (preferable, disk is cheap)
compression derived products on the fly for end users

the input processing is applied to each individual image and
can be different for each input image (radiometry). processing
parameters can be stored in the process definition xml file or
in the database, which is easier to manage for large image

band extraction, band algebra for multiband imagery
321 true color, 742 false color ir, 432 near ir false color, nvdi, etc
subsampling & supersampling
near neighbor, cubic convolution, bicubic, weighted matrix
transparency
enhance & sharpen
  brightness, contrast, gamma, equalize, dynamic based on histogram
  bit depth conversion - e.g. 18bit to 8bit
sharpening: convolution filter on the primary data, before the
rest of the processing, whcih will give the best results

enhancing and sharpening in combination can be used to allow
you to better see into the shadows

coordinate conversions

pan sharpening - fusion of high res b/w imagery with lower res color

mosaicking/merging
  can feather sides or do a seamline mosaic, or a straight merge
 

can change the viewpoint at the client end to you can look at both aspect/sides of features (building obscurity) in imagery with lots of building lean and lots of overlap. great for property assessors, security consultants, utility companies, etc.

can specify “best-on-top” (most nadir), Zorder (fields in DB), or client defined order (based on sun angle or some other parameter
of interest)

client can lock the order they want.

Another Demo

can “blur out” certain secure areas of imagery in one service for general public use and create a separate, secure imagery service for authorized use that does not blur out the sensitive features.

can hillshade and render raw dem or srtm or lidar data on the fly. can also merge in pockets of higher res data, such as lidar data from state and county collection projects and create a composite elevation surface.

You can create your own special processing, logo stamping on the imagery, interface to Hierarchical Data Storage, or payment gateways for credit card purchases.

There are 3 special extensions:

  ortho- 3d transform, orient data, terrain (image or image service)
 
  mosaic (feather, seamline detection)
 
  iCOMP - image compensation (trend removal on small scale
    aerial photos, color balancing) is not available yet
   

Client Side can control coord system, subsystem, compression for transmission, locking image, mosaic method. The client can’t create new services though, the admin has to do that.

Scalability

can load balance between multiple servers
MULTIPLE - Servers + DAS (direct access storage disk array - recommended!)
  lots of cheap ide disks that are mirrored is the best route
 
Multiple - Server - NAS/SAN (not recommended, they aren’t optimized for huge chunks of data pumped across the network, and it will kill your caching)
 
Cascading Servers - one service can request a subset of data from another service and do other processing on it.

managing derived images - creating overviews:
provide faster access by pre-processing
define by using the service manager
the server monitors every request that is made, and you can get the results of these as a shapefile of footprints, so you can get a better idea of your usage patterns and then adjust the amount of preprocessed imagery you provide to your users, so that certain high priority or high use areas can be stored as “tiles” and given a higher priority rating in the imagery metadata catalog so the image server will use this tile and not have to do on the fly processing for heavily trafficked areas.
 
client server arch. can do secured access.
works well with legacy applications because you don’t have to convert your imagery.
ANY ESRI app can be used as a client.
Other clients are ERDAS, MicroStation, AutoCad, Mapfinfo
**** OGC WMS, there is a programming API so you can [use WMS clients]
 
can input into SDE raster, Oracle georaster
services are open (db tables and xml) and can be automated with scripting as new imagery comes in.
product called Image Server Publisher
can’t author new services, but allows orgs to make copies of a full server to clients or depts (mirror a server) can be used on server farms, disaster failover sites.
 
 
SDE Raster vs Image Server
 
  db storage - file storage
  img has no dbms licensing and no data loading and no preprocessing

  **IF security is a big deal, a dbms is the better choice
 
  image server and SDE are complimentary, you can serve
  sde raster layers with it.
 

The Product

server & manager modules
service manager
client plugins (out of the box for previewing data)
  desktop, engine, server, 3rd party, OGC WMS
 
extensions (planned)
  mosaic, orhto, iCOMP
 
 
publisher version - read-only, lower price, no service manager

pricing - $10,000 for a 2 cpu configuration
available on its on release cycle, probably end of October

Q&A

Q:can output go to other places than the client viewer?

A:yes, can script to go to a ftp site or storage device
service admin can control how much of the extraction
by client users can happen.

Q: speed comparisons between image server & Sde raster
for same area of imagery on same server type?
A: no, not yet if you have thousands of users trying to access a final
product, sde is better tuned for that scenario. GIS users
and specialized fields such as security, emergency response,
would benefit more from the image server.

Q: OS support?
A: Windows only now, soon a Linux version, later
other OS’s, such as Solaris.

Q: Can you use oblique imagery from Pictometery? you can load it but the tools are not quite there yet for generating accurate measurements. The team is communicating with several vendors and hopes to offer support in later versions.

Q: For client developers who are developing front ends to use
Imager Srever - how much support will there be for this
type of work? 
A: They claim the API is simple, and there’s sample code, if you’re a C programmer. No mention of the actual support offerings were given.

There will be interface changes to the product to bring it into line with the ESRI look and feel. The service manager will not be part of ArcGIS immediately, but will be integrated into the family over time.

Q: Multi-temporal imagery - how to handle different time series images for the same footprint area?
A: you can define the Zorder field in your metadata catalog table and the client app can determine which Zorder field (latest, oldest, best sun angle, etc) to use during any working session. You can’t ‘hard code’ a single order for everyone to use, or allow users to pass in complex SQL code to determine a sort order. You will need to write your own custom interfaces if you want to have that level of control.

Continue reading...

ESRI-UC - GIS-Business Intelligence Integration Given High Profile

During the opening session of application demonstrations at the ESRI UC, the integration of enterprise computing applications with GIS was given particular attention. Linda Hecht provided an overview of how infrastructure management within city government was being tackled by integrating electric utility schematics with basemaps. Unique to this presentation was the ability to toggle between the schematic that showed the connectivity between city street traffic signals and the electrical transformers that may be affecting a lights at an intersection. In addition, she demonstrated how an SAP system was used to help integrate the applications between different functional departments, which was aided by and IBM Websphere platform.

David Huffman demonstrated dynamic, real-time data integration for business applications in site selection using an Information Builders application. It is worthwhile playing with this demo by going to http://ibigis.ibi.com/.

ESRI-UC - Quotes from Day Three

“I heard ArcIMS is replacing MapObjects? Is that right?” Question overheard at ArcGIS Information Booth.

“It isn’t by chanceЕ” A comment on the goings on in the consumer Web mapping market.

“I’m dead on my feet.” “I must have walked 10 miles today!” Two first time attendees responses when asked, “How are you doing?” They both had smiles on their faces!

ESRI-UC - A white elephant roams

Don’t look now but there is a white elephant on the UC exhibit floor. It’s called Google Maps. We can talk about it behind its back but we just can’t look at it too long. Look folks, there is not really a buzz about it here, but the questions, from those with whom I have spoken, always come up:

What does it mean for GIS and company’s like ESRI? Will it really make a difference to their bottom line? Will users escew ArcIMS and use the Google API to create a quick web map application? Can ArcExplorer and the network of ESRI web services compete with Google Earth?

Doesn’t Google "float the boat" of all GIS companies and location technology in general? Clearly it does because never before have we seen so much ink expended by the mainstream press. Google seems to be in a perpetual beta, always throwing up new things and not telling anyone…but the masses find out anyway. Awareness is key, and whereas for years we have tried to get people to take more notice of our technology sector, the "Google Phenomenon" has done more in six months than what I was able to do in sixteen years when I started the column "GIS in Business" for GIS World Magazine in 1991. The buzz used to be created by Microsoft but now the new kid on the block is causing a fuss.

Google will NOT impact the GIS business anytime soon and as Mark Cattini, CEO of MapInfo said, "It’s certainly very good consumer eye-candy." Jack Dangermond cautioned me, "don’t overhype it" and "It reminds me of LBS." So, the cautions are valid but it is still hard to ignore the pale pachyderm.