ArcGIS Online offers a wealth of functionality via the provided REST APIs and taking advantage of this functionality requires that your application follow proper procedures for authorization and access control. At Pro-West & Associates, we are always looking for ways to leverage the tools provided by the ArcGIS platform, and ArcGIS Online is no exception.

Highly Technical Content Ahead!

Often, we have a need to implement stand-alone python scripts that perform some nightly process. With the proliferation of ArcGIS Online usage, these batch processes can involve publishing and updating ArcGIS Online services and resource items as well as on-premises geodatabase data. In order to work with ArcGIS Online items, an authorization token must be obtained first. The requirement to add and update ArcGIS Online content means that we cannot use Application Authentication since it provides read-only access to the ArcGIS Online items. Rather, we need to perform an ArcGIS Online User Login sequence using OAuth 2.0 without requiring the user to interact with a web browser and support Active Directory Federated Services (AD FS) to authenticate our users via an Enterprise logon. Armed with a few third-party python modules and some determination, let’s show how to automate the OAuth 2.0 authentication process!

Preliminaries

Before we get started, an Application Item must be created and registered in your ArcGIS Online organization to facilitate the authorization process. This process is well-documented in the Registering Your Application help topic.

Requesting Access to the Application

In order to get an access token from ArcGIS Online, our application needs to follow the OAuth 2 workflow, which begins by requesting an access code from the Authorize endpoint that will be exchanged for an access token later. A simple HTTP GET request is used to pass in some basic parameters via the URL query parameters. Of the four parameters, the client_id, response_type and expiration are set with straightforward values.

For the redirect_url, we use a special value of urn:ietf:wg:oauth:2.0:oob that is documented in the Authorize topic of the OAuth 2.0 REST API.

1
2
3
4
5
6
7
8
9
10
11
12

parameters = {
    'client_id': 'PUBLIC_CLIENT_ID',
    'response_type': 'code',
    'expiration': 60,
    'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob'
}
 
# Ask the portal to authorize us
url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/oauth2/authorize'
 
response = requests.get(url, params=parameters)
content = response.text

The results from this request, if viewed in a browser, present the user with a page the displays information about the application that is requesting access to the ArcGIS Online account,
along with the Sign In options available to the user.

In this case, we want to simulate the user clicking the “Using Your ORGANIZATION_NAME Account”. The difficult bit is that the user is redirected to the organization authorization endpoint using a bit of JavaScript in the page. Since we are manually downloading the page content, no JavaScript execution is available. Fortunately, examining the page source shows that all of the pertinent information is conveniently tucked into a single global JavaScript variable named oAuthInfo.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<script type="text/javascript">// <![CDATA[
var oAuthInfo = {
    "oauth_state":"TOKEN",
    "client_id":"CLIENT_ID",
    "appTitle":"My AGOL App",
    "locale":"en",
    "orgName":"Pro-West & Associates, Inc.",
    "federationInfo":{
      "idpName":"Pro-West & Associates",
      "idpAuthorizeUrl":"https://pwa.maps.arcgis.com/sharing/oauth2/saml/authorize"
    },
    "contextPath":"/sharing",
    "appOrgInfo":{
      "id":"ORG_ID",
      "name":"Pro-West & Associates, Inc.",
      "description":"
 
<font color=\"#404040\"><span style=\"font-size: 29.3333339691162px;\"><b>Pro-West GIS<\/b><\/span><\/font><\/p>",
      "thumbnail":"thumbnail.png"
    },
    "orgUrlBase":"maps.arcgis.com",
    "helpBase":"http://doc.arcgis.com/en/arcgis-online/"
  };
// ]]></script>

Using the BeautifulSoup python module, it is trivial to scan through all of the <script> tag contents in the page, find and extract the JavaScript variable into a strings and then use the python JSON module to parse the variable declaration into a python dictionary.

1
2
3
4
5
6
7
8
9
10

pattern = re.compile('var oAuthInfo = ({.*?});', re.DOTALL)
 
soup = BeautifulSoup(content, 'html.parser')
for script in soup.find_all('script'):
    script_code = str(script.string).strip()
    matches = pattern.search(script_code)
    if not matches is None:
        js_object = matches.groups()[0]
        oAuthInfo = json.loads(js_object)
        break

Now that we have the federationInfo, we make a request to the endpoint. There are a couple of redirects that can happen here, but the requests module takes care of the details of the HTTP protocol and eventually arrives at the SAML page which issues a 401 Challenge. Since we are on a Windows Domain, we need to support the NTLM authentication protocol, which is helpfully provided by the requests_ntlm module.

1
2
3
4
5
6
7
8

from requests_ntlm import HttpNtlmAuth
 
post_data = {
    'oauth_state': oAuthInfo['oauth_state']
}
 
credentials = HttpNtlmAuth('DOMAIN\\username', 'password')
response = requests.post(oAuthInfo['federationInfo']['idpAuthorizeUrl'], data = post_data, allow_redirects = True, auth=credentials)

The idpAuthorizeUrl URL takes us to the following small confirmation page that posts the SAML Response back to the ArcGIS Online organizational account. By default, it uses a small JavaScript timeout action to automatically submit the form.

1

<html>Working...

As before, without the benefit of having browser-provided JavaScript support, we need to take matters into our own hands and manually submit the form. Again we use BeautifulSoup to extract the HTML input elements from the page and POST to the URL specified in the form’s action attribute. The page returned by the POST contains the authorization code that we are after in an <input> element with an id of “code”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

soup = BeautifulSoup(response.text, 'html.parser')
for form in soup.find_all('form', { 'name' : 'hiddenform' }):
 
    # Get the URL to POST
    url = form['action']
 
    # Get all of the named input fields
    inputElements = form.find_all('input', { 'name' : True })
    post_data = dict([(el['name'], el['value']) for el in inputElements])
 
    # Submit the form and hopefully get our code value
    response = requests.post(url, data = post_data, allow_redirects = True, auth=credentials)
    token_content = response.text
    break
 
soup = BeautifulSoup(token_content, 'html.parser')
code = soup.find(id='code')['value']

As far as we have come, we have actually only completed half of the OAuth 2.0 workflow. Fortunately, the final step of exchanging the code for an authorization token is much less convoluted that obtaining the code itself. The token can be used to actually perform REST operations against the organization’s portal API.

1
2
3
4
5
6
7
8
9
10
11
12
13

# Exchange the code for an access token
post_data = {
    'client_id': 'PUBLIC_CLIENT_ID',
    'code': code,
    'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob',
    'grant_type': 'authorization_code'
}
 
url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/oauth2/token'
 
response = requests.post(url, data = post_data)
token_response = json.loads(response.text)
access_token = token_response['access_token']

With the access token in hand, we can do a small test to print out the name of the organization’s portal and the full name of the user that has granted the application access to their account.

1
2
3
4
5
6
7
8
9
10
11
12
13

# Dump out the content for this account, just to prove things are working
parameters = {
    'token': access_token,
    'f': 'json'
}
 
url = 'https://' + ORGANIZATION_NAME + '.maps.arcgis.com/sharing/rest/portals/self'
response = requests.get(url, params=parameters)
query_result = json.loads(response.text)
 
print
print "Portal Name:", query_result['name']
print "Who am I?:  ", query_result['user']['fullName']

Congratulations! Now we have an access token and the full ArcGIS Online REST API at our disposal. Time to automate all the things!!

Share your ArcGIS Online scripting experiences by Tweeting us or by sending me an email.

Follow Lucas on Twitter