Big Data Ingestion - How to Use RESTful APIs

Big Data Ingestion - How to Use RESTful APIs

taught by Sarah Kelley


Aim of Course:

As analytics professionals and statisticians deal increasingly with Big Data, they need to know how to obtain data, and deploy the results of analysis back into web and other systems.  RESTful APIs are becoming the standard way of sharing and communicating data among applications and services on the internet.  REST stands for Representational State Transfer, and API for Application Programming Interface.  In this course, students will learn how to write Python code to ingest data from and communicate with RESTful API's on the web.

Course Program:

WEEK 1: HTTP protocol, common data formats

In our first week we will focus on how the internet works specifically as it applies to consuming RESTful services. We will take a brief survey through the HTTP protocol and the request/response cycle of exchanging information. We will touch on common data formats and end with some basic tools for development.
● How the internet works (Ports, DNS, Browser/HTTP Client)
● RPC/Web services vs normal browser usage
● Hypertext Transfer Protocol
● VirtualEnv & Github

WEEK 2:  RESTful

In week two, we will discuss Representational State Transfer as initially described by Roy Fielding. While not an exhaustive review, we’ll be looking at the primary tenets that have been adopted when developing web APIs. We will also touch on its benefits vs other methodologies, what it means to be ‘RESTful’, and how one might evaluate the degree to which a service adheres to its principles. Finally we will take a look at common features of data APIs and how to leverage them effectively.
● Resources and elements of REST
● Richardson Maturity Model
● Common API features

WEEK 3:  Scripts to ingest data

In week three we will implement basic scripts to ingest data from a REST service on the internet. Additionally we will touch on secure access to services and start our discussion on consuming data within the scope of an ingestion pipeline. 
● Key Python modules (, json, etc.)
● Using client libraries
● OAUTH and Basic Access Authentication
● Working code

WEEK 4:  Ingestion at scale

In our final week we will discuss the challenges of a data ingestion pipeline at scale. We will also examine best practices for data ingestion and review a sample codebase.
● Scaling your ingestion pipeline
● Data ingestion best practices
● Storing your data
● A sample ingestion pipeline



The homework in this course consists of short answer questions to test concepts, guided exercises in writing code and guided data analysis problems using software.

This course also has example software codes, supplemental readings available online and video lectures in each week.


Big Data Ingestion - How to Use RESTful APIs

Who Should Take This Course:
Analytics professionals, statisticians and data scientists familiar with Python who need to learn how to ingest data from, and interact with data via, RESTful web-based APIs
Organization of the Course:

This course takes place online at the Institute for 4 weeks. During each course week, you participate at times of your own choosing - there are no set times when you must be online. Course participants will be given access to a private discussion board. In class discussions led by the instructor, you can post questions, seek clarification, and interact with your fellow students and the instructor.

At the beginning of each week, you receive the relevant material, in addition to answers to exercises from the previous session. During the week, you are expected to go over the course materials, work through exercises, and submit answers. Discussion among participants is encouraged. The instructor will provide answers and comments, and at the end of the week, you will receive individual feedback on your homework answers.

Time Requirement:
About 15 hours per week, at times of  your choosing.

Options for Credit and Recognition:
Students come to the Institute for a variety of reasons. As you begin the course, you will be asked to specify your category:
  1. No credit - You may be interested only in learning the material presented, and not be concerned with grades or a record of completion.
  2. Certificate - You may be enrolled in PASS (Programs in Analytics and Statistical Studies) that requires demonstration of proficiency in the subject, in which case your work will be assessed for a grade.
  3. CEUs and/or proof of completion - You may require a "Record of Course Completion," along with professional development credit in the form of Continuing Education Units (CEU's).  For those successfully completing the course,  CEU's and a record of course completion will be issued by The Institute, upon request.
  4. Other options - Specializations, INFORMS CAP recognition, and academic (college) credit are available for some courses
Course Text:
Course material will be provided along with links to other required reading online.
The required software is Python Programming Language.


To be scheduled.

Big Data Ingestion - How to Use RESTful APIs


To be scheduled.

Course Fee: $549

Do you meet course prerequisites? What about book & software? (Click here to learn more)

Group rates: Click here to get information on group rates. 

First time student or academic? Click here for an introductory offer on select courses. Academic affiliation?  You may be eligible for a discount at checkout.

Register Now

Add $50 service fee if you require a prior invoice, or if you need to submit a purchase order or voucher, pay by wire transfer or EFT, or refund and reprocess a prior payment. Please use this printed registration form, for these and other special orders.

Courses may fill up at any time and registrations are processed in the order in which they are received. Your registration will be confirmed for the first available course date, unless you specify otherwise.

The Institute for Statistics Education is certified to operate by the State Council of Higher Education in Virginia (SCHEV).

Contact Us
Have a question about a course before you register? Call us. We're here for you. (571) 281-8817or

Want to be notified of future courses?

Student comments