Lucas Scharenbroich, Technology Manager
At Pro-West, our Vice-President & COO Kendis Scharenbroich is preparing to share her experience, expertise and advice during the technical session Aligning Location Technology with Business Goals on Thursday, September 17th.
There’s no better authority than Kendis on this topic – just ask our clients. She speaks with many, many organizations on a daily basis – in particular, local government – about how they can meet their business objectives using GIS technology. She knows the pain points and the possibilities, and Pro-West’s clients can vouch for her ability to truly understand their organizations and deliver what they need from their location technology. Always ahead of the curve with GIS technology, she will discuss how the latest technologies hitting the market can deliver strategically for YOUR organization, and highlight capabilities offered by GIS that you might not even know exist.
We guarantee you will leave feeling INSPIRED!
Topic: Aligning Location Technology with Business Goals
Date: Thursday, September 17th
Time: 9:45am
Location: Room 203, Duluth Convention Center
See the latest from the MN AWWA Annual Conference: #MNAWWA
HIGHLY TECHNICAL CONTENT AHEAD!
Often, we have a need to implement stand-alone python scripts that perform some nightly process. With the proliferation of ArcGIS Online usage, these batch processes can involve publishing and updating ArcGIS Online services and resource items as well as on-premises geodatabase data. In order to work with ArcGIS Online items, an authorization token must be obtained first. The requirement to add and update ArcGIS Online content means that we cannot use Application Authentication since it provides read-only access to the ArcGIS Online items. Rather, we need to perform an ArcGIS Online User Login sequence using OAuth 2.0 without requiring the user to interact with a web browser and support Active Directory Federated Services (AD FS) to authenticate our users via an Enterprise logon. Armed with a few third-party python modules and some determination, let’s show how to automate the OAuth 2.0 authentication process!
PRELIMINARIES
Before we get started, an Application Item must be created and registered in your ArcGIS Online organization to facilitate the authorization process. This process is well-documented in the Registering Your Application help topic.
REQUESTING ACCESS TO THE APPLICATION
In order to get an access token from ArcGIS Online, our application needs to follow the OAuth 2 workflow, which begins by requesting an access code from the Authorize endpoint that will be exchanged for an access token later. A simple HTTP GET request is used to pass in some basic parameters via the URL query parameters. Of the four parameters, the client_id, response_type and expiration are set with straightforward values.
For the redirect_url, we use a special value of urn:ietf:wg:oauth:2.0:oob that is documented in the Authorize topic of the OAuth 2.0 REST API.
parameters = { 'client_id': 'PUBLIC_CLIENT_ID', 'response_type': 'code', 'expiration': 60, 'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob' } # Ask the portal to authorize us url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/oauth2/authorize' response = requests.get(url, params=parameters) content = response.text
The results from this request, if viewed in a browser, present the user with a page the displays information about the application that is requesting access to the ArcGIS Online account,
along with the Sign In options available to the user.
In this case, we want to simulate the user clicking the “Using Your ORGANIZATION_NAME Account”. The difficult bit is that the user is redirected to the organization authorization endpoint using a bit of JavaScript in the page. Since we are manually downloading the page content, no JavaScript execution is available. Fortunately, examining the page source shows that all of the pertinent information is conveniently tucked into a single global JavaScript variable named oAuthInfo.
<script type="text/javascript">// <![CDATA[ var oAuthInfo = { "oauth_state":"TOKEN", "client_id":"CLIENT_ID", "appTitle":"My AGOL App", "locale":"en", "orgName":"Pro-West & Associates, Inc.", "federationInfo":{ "idpName":"Pro-West & Associates", "idpAuthorizeUrl":"https://pwa.maps.arcgis.com/sharing/oauth2/saml/authorize" }, "contextPath":"/sharing", "appOrgInfo":{ "id":"ORG_ID", "name":"Pro-West & Associates, Inc.", "description":" <font color=\"#404040\"><span style=\"font-size: 29.3333339691162px;\"><b>Pro-West GIS<\/b><\/span><\/font><\/p>", "thumbnail":"thumbnail.png" }, "orgUrlBase":"maps.arcgis.com", "helpBase":"http://doc.arcgis.com/en/arcgis-online/" }; // ]]></script>
Using the BeautifulSoup python module, it is trivial to scan through all of the <script> tag contents in the page, find and extract the JavaScript variable into a strings and then use the python JSON module to parse the variable declaration into a python dictionary.
pattern = re.compile('var oAuthInfo = ({.*?});', re.DOTALL) soup = BeautifulSoup(content, 'html.parser') for script in soup.find_all('script'): script_code = str(script.string).strip() matches = pattern.search(script_code) if not matches is None: js_object = matches.groups()[0] oAuthInfo = json.loads(js_object) break
Now that we have the federationInfo, we make a request to the endpoint. There are a couple of redirects that can happen here, but the requests module takes care of the details of the HTTP protocol and eventually arrives at the SAML page which issues a 401 Challenge. Since we are on a Windows Domain, we need to support the NTLM authentication protocol, which is helpfully provided by the requests_ntlm module.
from requests_ntlm import HttpNtlmAuth post_data = { 'oauth_state': oAuthInfo['oauth_state'] } credentials = HttpNtlmAuth('DOMAIN\\username', 'password') response = requests.post(oAuthInfo['federationInfo']['idpAuthorizeUrl'], data = post_data, allow_redirects = True, auth=credentials)
The idpAuthorizeUrl URL takes us to the following small confirmation page that posts the SAML Response back to the ArcGIS Online organizational account. By default, it uses a small JavaScript timeout action to automatically submit the form.
<html>Working...
As before, without the benefit of having browser-provided JavaScript support, we need to take matters into our own hands and manually submit the form. Again we use BeautifulSoup to extract the HTML input elements from the page and POST to the URL specified in the form’s action attribute. The page returned by the POST contains the authorization code that we are after in an <input> element with an id of “code”.
# Exchange the code for an access token post_data = { 'client_id': 'PUBLIC_CLIENT_ID', 'code': code, 'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob', 'grant_type': 'authorization_code' } url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/oauth2/token' response = requests.post(url, data = post_data) token_response = json.loads(response.text) access_token = token_response['access_token']
With the access token in hand, we can do a small test to print out the name of the organization’s portal and the full name of the user that has granted the application access to their account.
# Dump out the content for this account, just to prove things are working parameters = { 'token': access_token, 'f': 'json' } url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/portals/self' response = requests.get(url, params=parameters) query_result = json.loads(response.text) print print "Portal Name:", query_result['name'] print "Who am I?: ", query_result['user']['fullName']
Congratulations! Now we have an access token and the full ArcGIS Online REST API at our disposal. Time to automate all the things!!
Share your ArcGIS Online scripting experiences by Tweeting us or by sending me an email.