Mashup API
Tools for programmatically querying the MAST Portal.
 All Classes Namespaces Properties Pages
Using the API with Python

Table of Contents

Introduction

This page gives examples of various MAST Mashup queries using Python. Each example is fully runnable (in both python 2.x and python 3.x), but may refer to other example functions on this page. To download the file in its entirety, click here.

There is also a tutorial that can be viewed here, or downloaded as a jupyter notebook.

Includes and Helper Functions

This section contains all necessary imports for the examples given here, as well a several helper functions used throughout the page.

1 import sys
2 import os
3 import time
4 import re
5 import json
6 
7 try: # Python 3.x
8  from urllib.parse import quote as urlencode
9  from urllib.request import urlretrieve
10 except ImportError: # Python 2.x
11  from urllib import pathname2url as urlencode
12  from urllib import urlretrieve
13 
14 try: # Python 3.x
15  import http.client as httplib
16 except ImportError: # Python 2.x
17  import httplib

MAST Query

Sends a Mashup request to the MAST server and returns the response.

1 def mastQuery(request):
2 
3  server='mast.stsci.edu'
4 
5  # Grab Python Version
6  version = ".".join(map(str, sys.version_info[:3]))
7 
8  # Create Http Header Variables
9  headers = {"Content-type": "application/x-www-form-urlencoded",
10  "Accept": "text/plain",
11  "User-agent":"python-requests/"+version}
12 
13  # Encoding the request as a json string
14  requestString = json.dumps(request)
15  requestString = urlencode(requestString)
16 
17  # opening the https connection
18  conn = httplib.HTTPSConnection(server)
19 
20  # Making the query
21  conn.request("POST", "/api/v0/invoke", "request="+requestString, headers)
22 
23  # Getting the response
24  resp = conn.getresponse()
25  head = resp.getheaders()
26  content = resp.read().decode('utf-8')
27 
28  # Close the https connection
29  conn.close()
30 
31  return head,content

Download Request

Performs a get request to download a specified file from the MAST server.

1 def downloadRequest(url):
2  server='mast.stsci.edu'
3 
4  conn = httplib.HTTPSConnection(server)
5  conn.request("GET", url)
6  resp = conn.getresponse()
7 
8  fileName = resp.getheader('Content-Disposition')[21:]
9  fileContent = resp.read()
10 
11  with open(fileName,'wb') as FLE:
12  FLE.write(fileContent)
13 
14  conn.close()
15 
16  return fileName

Convert json to csv

Converts json return type object into csv.

1 def mastJson2Csv(json):
2  csvStr = ",".join([x['name'] for x in json['fields']])
3  csvStr += "\n"
4  csvStr += ",".join([x['type'] for x in json['fields']])
5  csvStr += "\n"
6 
7  colNames = [x['name'] for x in json['fields']]
8  for row in json['data']:
9  csvStr += ",".join([str(row.get(col,"nul")) for col in colNames]) + "\n"
10 
11  return csvStr

Convert json to astropy Table

Convets json return type object into an astropy Table.

1 from astropy.table import Table
2 import numpy as np
3 
4 def mastJson2Table(jsonObj):
5 
6  dataTable = Table()
7 
8  for col,atype in [(x['name'],x['type']) for x in jsonObj['fields']]:
9  if atype=="string":
10  atype="str"
11  if atype=="boolean":
12  atype="bool"
13  dataTable[col] = np.array([x.get(col,None) for x in jsonObj['data']],dtype=atype)
14 
15  return dataTable

Name Resolver

Mast.Name.Lookup.
Depends on mastQuery.

1 def resolveName():
2 
3  mashupRequest = {'service':'Mast.Name.Lookup',
4  'params':{'input':'M101',
5  'format':'json'},
6  }
7 
8  headers,outString = mastQuery(mashupRequest)
9 
10  outData = json.loads(outString)
11 
12  return outData

Cone Searches

CAOM Cone Search

Mast.Caom.Cone.
Depends on mastQuery.
See the CAOM field documentation for information on the returned columns.

1 def caomConeSearch():
2 
3  mashupRequest = {'service':'Mast.Caom.Cone',
4  'params':{'ra':254.28746,
5  'dec':-4.09933,
6  'radius':0.2},
7  'format':'json',
8  'pagesize':5000,
9  'page':1,
10  'removenullcolumns':True,
11  'timeout':30,
12  'removecache':True}
13 
14  headers,outString = mastQuery(mashupRequest)
15 
16  outData = json.loads(outString)
17 
18  return outData

VO Cone Search

Vo.Hesarc.DatascopeListable.
Depends on mastQuery.
Note that instead of returning all results at once, this query returns partial results as they become availible.

1 def voConeSearch():
2 
3  mashupRequest = {'service':'Vo.Hesarc.DatascopeListable',
4  'params':{'ra':254.28746,
5  'dec':-4.09933,
6  'radius':0.2,
7  'skipcache':True},
8  'format':'json',
9  'removenullcolumns':True}
10 
11  allData = []
12  startTime = time.time()
13 
14  while True:
15  headers,outString = mastQuery(mashupRequest)
16  outData = json.loads(outString)
17  allData.append(outData)
18  if outData['status'] != "EXECUTING":
19  break
20  if time.time() - startTime > 30:
21  print("Working...")
22  startTime = time.time()
23  time.sleep(10)
24 
25  return allData

HSC Cone Search

Mast.Hsc.Db.
Depends on mastQuery.
See the HSC field documentation for information on the returned columns.

1 def hscConeSearch():
2 
3  mashupRequest = {'service':'Mast.Hsc.Db',
4  'params':{'ra':254.287,
5  'dec':-4.09933,
6  'radius':0.2,
7  'nr':5000,
8  'ni':1,
9  'magtype':1},
10  'format':'json',
11  'pagesize':1000,
12  'page':1,
13  'removenullcolumns':True}
14 
15  headers,outString = mastQuery(mashupRequest)
16 
17  outData = json.loads(outString)
18 
19  return outData

GAIA Cone Search

Mast.Catalogs.Gaia.Cone.
Depends on mastQuery.
See the Gaia field documentation for information on the returned columns.

1 def gaiaConeSearch():
2  mashupRequest = {'service':'Mast.Catalogs.Gaia.Cone',
3  'params':{'ra':254.287,
4  'dec':-4.09933,
5  'radius':0.2},
6  'format':'json',
7  'pagesize':1000,
8  'page':5}
9 
10  headers,outString = mastQuery(mashupRequest)
11 
12  outData = json.loads(outString)
13 
14  return outData

TGAS Cone Search

Mast.Catalogs.Tgas.Cone.
Depends on mastQuery.
Note that the columnsconfigid parameter does not match the service name for this service.
See the Gaia field documentation for information on the returned columns.

1 def tgasConeSearch():
2  mashupRequest = { "service":"Mast.Catalogs.Tgas.Cone",
3  "params":{
4  "ra":254.28746,
5  "dec":-4.09933,
6  "radius":0.2},
7  "format":"json",
8  "timeout":10}
9 
10  headers,outString = mastQuery(mashupRequest)
11 
12  outData = json.loads(outString)
13 
14  return outData

Advanced Search

Advanced Search Filtering

Mast.Caom.Filtered.
Depends on mastQuery.
This service allows the user to perform a counts only query, or to get the usual grid of results. When unsure how many results re expected, it is best to first perform a counts query to avoid memory overflow.
See the CAOM field documentation for information on what columns may be filtered on and how inputs should be formatted.

1 def advancedSearchCounts():
2  mashupRequest = {"service":"Mast.Caom.Filtered",
3  "format":"json",
4  "params":{
5  "columns":"COUNT_BIG(*)",
6  "filters":[
7  {"paramName":"filters",
8  "values":["NUV","FUV"],
9  "separator":";"
10  },
11  {"paramName":"t_max",
12  "values":[{"min":52264.4586,"max":54452.8914}], #MJD
13  },
14  {"paramName":"obsid",
15  "values":[],
16  "freeText":"%200%"}
17  ]}}
18 
19  headers,outString = mastQuery(mashupRequest)
20  outData = json.loads(outString)
21 
22  return outData
23 
24 
25 def advancedSearch():
26  mashupRequest = {"service":"Mast.Caom.Filtered",
27  "format":"json",
28  "params":{
29  "columns":"*",
30  "filters":[
31  {"paramName":"dataproduct_type",
32  "values":["image"],
33  },
34  {"paramName":"proposal_pi",
35  "values":["Osten"]
36  }
37  ]}}
38 
39  headers,outString = mastQuery(mashupRequest)
40  outData = json.loads(outString)
41 
42  return outData

Filtering with specified position

Mast.Caom.Filtered.Position.
Depends on mastQuery.
This service allows the user to perform a counts only query, or to get the usual grid of results. When unsure how many results re expected, it is best to first perform a counts query to avoid memory overflow.
See the CAOM field documentation for information on what columns may be filtered on and how inputs should be formatted.

1 def advancedSearchWithPositionCounts():
2  mashupRequest = {
3  "service":"Mast.Caom.Filtered.Position",
4  "format":"json",
5  "params":{
6  "columns":"COUNT_BIG(*)",
7  "filters":[
8  {"paramName":"dataproduct_type",
9  "values":["cube","image"]
10  }],
11  "position":"210.8023, 54.349, 5"
12  }}
13 
14  headers,outString = mastQuery(mashupRequest)
15  outData = json.loads(outString)
16 
17  return outData
18 
19 
20 def advancedSearchWithPosition():
21  mashupRequest = {
22  "service":"Mast.Caom.Filtered.Position",
23  "format":"json",
24  "params":{
25  "columns":"*",
26  "filters":[
27  {"paramName":"dataproduct_type",
28  "values":["cube"]
29  }],
30  "position":"210.8023, 54.349, 0.24"
31  }}
32 
33  headers,outString = mastQuery(mashupRequest)
34  outData = json.loads(outString)
35 
36  return outData

HSC Spectra

Mast.HscSpectra.Db.All.
Depends on mastQuery.
Note that this query returns all HSC spectra held by MAST.
See the HSC spectra field documentation for information on the returned columns.

1 def hscSpectraSearch():
2 
3  mashupRequest = {'service':'Mast.HscSpectra.Db.All',
4  'format':'votable'}
5 
6  headers,outString = mastQuery(mashupRequest)
7  return outString

HSC Matches

Mast.HscMatches.Db.
Depends on HSC Cone Search and mastQuery.
See the HSC matches field documentation for information on the returned columns.

1 def getHSCMatches():
2  # perform the HSC search
3  result = hscConeSearch()
4  data = result['data']
5 
6  # get the match id
7  matchId = data[0]['MatchID']
8 
9  # get detailed results for chosen match id
10  mashupRequest = {'service':'Mast.HscMatches.Db',
11  'params':{'input':matchId},
12  'format':'json',
13  'page':1,
14  'pagesize':4}
15 
16  headers,outString = mastQuery(mashupRequest)
17 
18  outData = json.loads(outString)
19 
20  return outData

Get VO Data

Vo.Generic.Table.
Depends on VO Cone Search and mastQuery.

1 def getVoData():
2 
3  # perform vo cone search
4  voData = voConeSearch()
5  voJson = voData[1]
6 
7  #colnames = [x['name'] for x in voJson[1]['fields']]
8  #urlidx = colnames.index('tableURL')
9 
10  row = voJson['data'][2]
11  url = row['tableURL']
12 
13  mashupRequest = {'service':'Vo.Generic.Table',
14  'params':{'url':url},
15  'format':'json',
16  'page':1,
17  'pagesize':1000}
18 
19  headers,outString = mastQuery(mashupRequest)
20 
21  outData = json.loads(outString)
22 
23  return outData

Cross-Match

CAOM Cross-Match

Mast.Caom.Crossmatch.
Depends on CAOM Cone Search and mastQuery.
The top example shows how to use the output of a cone search as crossmatch input, and the bottom example shows how you can build the crossmatch input manually.

1 def crossMatchFromConeSearch():
2 
3  # This is a json object
4  crossmatchInput = caomConeSearch()
5 
6  mashupRequest = {"service":"Mast.Caom.Crossmatch",
7  "data":crossmatchInput,
8  "params":{
9  "raColumn":"s_ra",
10  "decColumn":"s_dec",
11  "radius":0.001
12  },
13  "pagesize":1000,
14  "page":1,
15  "format":"json",
16  "removecache":True}
17 
18  headers,outString = mastQuery(mashupRequest)
19 
20  outData = json.loads(outString)
21 
22  return outData
23 
24 def crossMatchFromMinimalJson():
25 
26  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
27  {"name":"dec","type":"float"}],
28  "data":[{"ra":210.8,"dec":54.3}]}
29 
30  mashupRequest = {"service":"Mast.Caom.Crossmatch",
31  "data":crossmatchInput,
32  "params":{
33  "raColumn":"ra",
34  "decColumn":"dec",
35  "radius":0.001
36  },
37  "pagesize":1000,
38  "page":1,
39  "format":"json",
40  "clearcache":True}
41 
42  headers,outString = mastQuery(mashupRequest)
43 
44  outData = json.loads(outString)
45 
46  return outData

GALEX Cross-Match

Mast.Galex.Crossmatch.
Depends on mastQuery.

1 def galexCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"s_ra","type":"float"},
5  {"name":"s_dec","type":"float"}],
6  "data":[{"s_ra":210.8,"s_dec":54.3}]}
7 
8  mashupRequest = {"service":"Mast.Galex.Crossmatch",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"s_ra",
12  "decColumn":"s_dec",
13  "radius":0.01
14  },
15  "pagesize":1000,
16  "page":1,
17  "format":"json"}
18 
19  headers,outString = mastQuery(mashupRequest)
20 
21  outData = json.loads(outString)
22 
23  return outData

SDSS Cross-Match

Mast.Sdss.Crossmatch.
Depends on mastQuery.

1 def sdssCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":337.10977,"dec":30.30261}]}
7 
8  mashupRequest ={"service":"Mast.Sdss.Crossmatch",
9  "data":crossmatchInput,
10  "params":
11  {
12  "raColumn":"ra",
13  "decColumn":"dec",
14  "radius":0.01
15  },
16  "format":"json",
17  "pagesize":1000,
18  "page":1}
19 
20  headers,outString = mastQuery(mashupRequest)
21 
22  outData = json.loads(outString)
23 
24  return outData

2MASS Cross-Match

Mast.2Mass.Crossmatch.
Depends on mastQuery.

1 def twoMassCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":210.88447,"dec":54.332}]}
7 
8  mashupRequest = {"service":"Mast.2Mass.Crossmatch",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.04
14  },
15  "pagesize":1000,
16  "page":1,
17  "format":"json"}
18 
19  headers,outString = mastQuery(mashupRequest)
20 
21  outData = json.loads(outString)
22 
23  return outData

HSC 1.0, MagAper2 Cross-Match

Mast.Hsc.Crossmatch.
Depends on mastQuery.

1 def hscV1Crossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":210.8,"dec":54.3}]}
7 
8  mashupRequest = {"service":"Mast.Hsc.Crossmatch",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.001
14  },
15  "pagesize":1000,
16  "page":1,
17  "format":"json"}
18 
19  headers,outString = mastQuery(mashupRequest)
20 
21  outData = json.loads(outString)
22 
23  return outData

HSC 2.1, MagAper2 Cross-Match

Mast.Hsc.Crossmatch.MagAper2.
Depends on mastQuery.

1 def hscMagAper2Crossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":210.8,"dec":54.3}]}
7 
8  mashupRequest = {"service":"Mast.Hsc.Crossmatch.MagAper2",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.001
14  },
15  "pagesize":1000,
16  "page":1,
17  "format":"json"}
18 
19  headers,outString = mastQuery(mashupRequest)
20 
21  outData = json.loads(outString)
22 
23  return outData

HSC 2.1, MagAuto Cross-Match

Mast.Hsc.Crossmatch.MagAuto.
Depends on mastQuery.

1 def hscMagAutoCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":210.8,"dec":54.3}]}
7 
8  mashupRequest = {"service":"Mast.Hsc.Crossmatch.MagAuto",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.001
14  },
15  "format":"json"}
16 
17  headers,outString = mastQuery(mashupRequest)
18 
19  outData = json.loads(outString)
20 
21  return outData

Gaia Cross-Match

Mast.Gaia.Crossmatch.
Depends on mastQuery.

1 def gaiaCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":210.8,"dec":54.3}]}
7 
8  mashupRequest = {"service":"Mast.Gaia.Crossmatch",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.1
14  },
15  "format":"json"}
16 
17  headers,outString = mastQuery(mashupRequest)
18 
19  outData = json.loads(outString)
20 
21  return outData

TGAS Cross-Match

Mast.Tgas.Crossmatch.
Depends on mastQuery.

1 def tgasCrossmatch():
2 
3  # This is a json object
4  crossmatchInput = {"fields":[{"name":"ra","type":"float"},
5  {"name":"dec","type":"float"}],
6  "data":[{"ra":211.09,"dec":54.3228}]}
7 
8  mashupRequest = {"service":"Mast.Tgas.Crossmatch",
9  "data":crossmatchInput,
10  "params":{
11  "raColumn":"ra",
12  "decColumn":"dec",
13  "radius":0.2
14  },
15  "format":"json"}
16 
17  headers,outString = mastQuery(mashupRequest)
18 
19  outData = json.loads(outString)
20 
21  return outData

Getting CAOM Products

List CAOM Products

Mast.Caom.Products.
Depends on CAOM Cone Search and mastQuery.

1 def getCaomProducts():
2 
3  # perform the CAOM search
4  result = caomConeSearch()
5  data = result['data']
6 
7  # get the product group id (obsid)
8  obsid = data[1]['obsid']
9 
10  # get detailed results for chosen match id
11  mashupRequest = {'service':'Mast.Caom.Products',
12  'params':{'obsid':obsid},
13  'format':'json',
14  'pagesize':4,
15  'page':1}
16 
17  headers,outString = mastQuery(mashupRequest)
18 
19  outData = json.loads(outString)
20 
21  return outData

Download CAOM Data Products

Mast.Bundle.Request.
Depends on List CAOM Products, CAOM Cone Search, mastQuery, and downloadRequest.
The top example downloads a single observation product, the bottom example downloads multiple products.

1 def downloadOneProduct():
2 
3  # get data products
4  result = getCaomProducts()
5  data = result['data']
6 
7  # collect the parameters you need
8  urlList = data[0]['dataURI']
9  descriptionList = list(data[0]['description'])
10  productTypeList = list(data[0]['dataproduct_type'])
11 
12  # Decide on filename for the download and make the product path
13  filename = "mastData"
14  pathList = "mastFiles/"+data[0]['obs_collection'] + '/' + data[0]['obs_id'] + '/' + data[0]['productFilename']
15 
16  # make the bundler request
17  mashupRequest = {"service":"Mast.Bundle.Request",
18  "timeout":"3",
19  "params":{
20  "urlList":urlList,
21  "filename":filename,
22  "pathList":pathList,
23  "descriptionList":descriptionList,
24  "productTypeList":productTypeList,
25  "extension":"curl"},
26  "format":"json"}
27 
28  headers,outString = mastQuery(mashupRequest)
29  outData = json.loads(outString)
30 
31  # get the download file
32  downloadFile = downloadRequest(outData['url'])
33 
34  return downloadFile,outData['manifestUrl']
35 
36 
37 def downloadMultipleProducts():
38 
39  # get data products
40  result = getCaomProducts()
41  data = result['data']
42 
43  # collect the parameters you need
44  urlList = [x['dataURI'] for x in data]
45  descriptionList = [x['description'] for x in data]
46  productTypeList = [x['dataproduct_type'] for x in data]
47 
48  # Decide on filename and paths
49  filename = "mastMultiData"
50  pathList = ["mastFiles/"+x['obs_collection']+'/'+x['obs_id']+'/'+x['productFilename'] for x in data]
51 
52  # make the bundler request
53  mashupRequest = {"service":"Mast.Bundle.Request",
54  "timeout":"3",
55  "params":{
56  "urlList":",".join(urlList),
57  "filename":filename,
58  "pathList":",".join(pathList),
59  "descriptionList":descriptionList,
60  "productTypeList":productTypeList,
61  "extension":"tar.gz"},
62  "format":"json"}
63 
64  headers,outString = mastQuery(mashupRequest)
65  outData = json.loads(outString)
66 
67  # get the download file
68  downloadFile = downloadRequest(outData['url'])
69 
70  return downloadFile,outData['manifestUrl']

Direct Download

Depends on List CAOM Products.
Some MAST data products are accessed by URL and some by URI. Products that are accessed by URL can be directly downloaded using that URL, while products accessed via URI need to go through the Portal downloader.

1 def directDownload():
2 
3  # collect the parameters you need
4  result = getCaomProducts()
5  data = result['data']
6 
7  accessLink = data[0]['dataURI']
8  filename = data[0]['productFilename'] # note this is not necessarily unique
9 
10  # for testing uri functionality
11  #accessLink = 'mast:HST/product/y2gy090ft_trl.fits'
12  #filename = 'y2gy090ft_trl.fits' # note this is not necessarily unique
13 
14  # check for url vs uri
15  if "http" in accessLink: # link is url, so can just download
16  urlretrieve(accessLink, filename)
17 
18  else: # link is uri, so need to go through direct download request
19  server='mast.stsci.edu'
20  conn = httplib.HTTPSConnection(server)
21  conn.request("GET", "/api/v0/download/file/"+accessLink.lstrip('mast:'))
22  resp = conn.getresponse()
23  fileContent = resp.read()
24  with open(filename,'wb') as FLE:
25  FLE.write(fileContent)
26  conn.close()
27 
28  return filename