Skip to content

Instantly share code, notes, and snippets.

@ldruizsan

ldruizsan/Collecting_Data_API.ipynb

Created December 23, 2020 04:29
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ldruizsan/bf51ea7f8144e780f2ded629fd9b51e5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ldruizsan/bf51ea7f8144e780f2ded629fd9b51e5 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Collecting_Data_API.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": "<center>\n <img src=\"https://gitlab.com/ibm/skills-network/courses/placeholder101/-/raw/master/labs/module%201/images/IDSNlogo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\" />\n</center>\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "# **Collecting Job Data Using APIs**\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Estimated time needed: **45 to 60** minutes\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Objectives\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "After completing this lab, you will be able to:\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "- Collect job data from GitHub Jobs API\n- Store the collected data into an excel spreadsheet. \n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Warm-Up Exercise\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Using an API, let us find out who currently are on the International Space Station (ISS).<br> The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.<br>\nYou can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n"
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": "import requests # you need this module to make an API call"
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": "response = requests.get(api_url) # Call the API using the get method and store the\n # output of the API call in a variable called response."
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": "if response.ok: # if all is well() no errors, no network timeouts)\n data = response.json() # store the result in json format in a variable called data\n # the variable data is of type dictionary."
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "<class 'list'>\n"
}
],
"source": "print(type(data))"
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "{'message': 'success', 'number': 7, 'people': [{'craft': 'ISS', 'name': 'Sergey Ryzhikov'}, {'craft': 'ISS', 'name': 'Kate Rubins'}, {'craft': 'ISS', 'name': 'Sergey Kud-Sverchkov'}, {'craft': 'ISS', 'name': 'Mike Hopkins'}, {'craft': 'ISS', 'name': 'Victor Glover'}, {'craft': 'ISS', 'name': 'Shannon Walker'}, {'craft': 'ISS', 'name': 'Soichi Noguchi'}]}\n"
}
],
"source": "print(data) # print the data just to check the output or for debugging"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Print the number of astronauts currently on ISS.\n"
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "7\n"
}
],
"source": "print(data.get('number'))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Print the names of the astronauts currently on ISS.\n"
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "There are 7 astronauts on ISS\nAnd their names are :\nSergey Ryzhikov\nKate Rubins\nSergey Kud-Sverchkov\nMike Hopkins\nVictor Glover\nShannon Walker\nSoichi Noguchi\n"
}
],
"source": "astronauts = data.get('people')\nprint(\"There are {} astronauts on ISS\".format(len(astronauts)))\nprint(\"And their names are :\")\nfor astronaut in astronauts:\n print(astronaut.get('name'))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Hope the warmup was helpful. Good luck with your next lab!\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Lab: Collect Jobs Data using GitHub Jobs API\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Before you start doing this lab, get familier with the GitHub Jobs API.<br>\nThe documentation for the GitHub Jobs API can be found at <https://jobs.github.com/api><br>\n\n<li>Understand what urls to use.<br>\n<li>Understand what parameters have to be passed.<br>\n<li>Understand the format of the output data.</li>\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Objective: Determine the number of jobs currently open for various technologies\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Collect the number of job postings for the following languages using the API:\n\n- C\n- C#\n- C++\n- Java\n- JavaScript\n- Python\n- Scala\n- Oracle\n- SQL Server\n- MySQL Server\n- PostgreSQL\n- MongoDB\n"
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": "#Import required libraries\nimport requests\n"
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": "baseurl = \"https://jobs.github.com/positions.json\""
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Write a function to get the number of jobs for the given technology.<br>\n_Note:_ The API gives a maximum of 50 jobs per page.<br>\nIf you get 50 jobs per page, it means there could be some more job listings available.<br>\nSo if you get 50 jobs per page you should make another API call for next page to check for more jobs.<br>\nIf you get less than 50 jobs per page, you can take it as the final count.<br>\n"
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [],
"source": "def get_number_of_jobs(technology):\n pages = [1,2,3,4,5]\n jobs = []\n for page in pages:\n url = \"https://jobs.github.com/positions.json?description={}&page={}\".format(technology,page)\n response = requests.get(url)\n if response.ok:\n job = response.json()\n jobs += job\n number_of_jobs = len(jobs)\n \n return technology,number_of_jobs"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Call the function for Python and check if it is working.\n"
},
{
"cell_type": "code",
"execution_count": 62,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "('python', 37)\n"
}
],
"source": "print(get_number_of_jobs('python'))"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Store the results in an excel file\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Call the API for all the given technologies above and write the results in an excel spreadsheet.\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "If you do not know how create excel file using python, double click here for **hints**.\n\n<!--\n\nfrom openpyxl import Workbook # import Workbook class from module openpyxl\nwb=Workbook() # create a workbook object\nws=wb.active # use the active worksheet\nws.append(['Country','Continent']) # add a row with two columns 'Country' and 'Continent'\nws.append(['Eygpt','Africa']) # add a row with two columns 'Egypt' and 'Africa'\nws.append(['India','Asia']) # add another row\nws.append(['France','Europe']) # add another row\nwb.save(\"countries.xlsx\") # save the workbook into a file called countries.xlsx\n\n\n-->\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Create a python list of all technologies for which you need to find the number of jobs postings.\n"
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [],
"source": "tech = ['Python','Java',\"C++\",\"C#\",'C','Ruby','JavaScript', 'Oracle','Scala','SQL Server','MongoDB','MySQL Server','PostgreSQL']"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Import libraries required to create excel spreadsheet\n"
},
{
"cell_type": "code",
"execution_count": 64,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": "Collecting openpyxl\n Downloading openpyxl-3.0.5-py2.py3-none-any.whl (242 kB)\n\u001b[K |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 242 kB 19.1 MB/s eta 0:00:01\n\u001b[?25hCollecting et-xmlfile\n Downloading et_xmlfile-1.0.1.tar.gz (8.4 kB)\nCollecting jdcal\n Downloading jdcal-1.4.1-py2.py3-none-any.whl (9.5 kB)\nBuilding wheels for collected packages: et-xmlfile\n Building wheel for et-xmlfile (setup.py) ... \u001b[?25ldone\n\u001b[?25h Created wheel for et-xmlfile: filename=et_xmlfile-1.0.1-py3-none-any.whl size=8915 sha256=2f44a494b36bb3a0ee00da15187361b154ad1cb43c00b62277d34dc695c4d719\n Stored in directory: /tmp/wsuser/.cache/pip/wheels/e2/bd/55/048b4fd505716c4c298f42ee02dffd9496bb6d212b266c7f31\nSuccessfully built et-xmlfile\nInstalling collected packages: et-xmlfile, jdcal, openpyxl\nSuccessfully installed et-xmlfile-1.0.1 jdcal-1.4.1 openpyxl-3.0.5\n"
}
],
"source": "!pip install openpyxl"
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {},
"outputs": [],
"source": "from openpyxl import Workbook"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Create a workbook and select the active worksheet\n"
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {},
"outputs": [],
"source": "# Create Workbook object\nwb = Workbook()\nws = wb.active"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Find the number of jobs postings for each of the technology in the above list.\nWrite the technology name and the number of jobs postings into the excel spreadsheet.\n"
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {},
"outputs": [],
"source": "for i in tech:\n ws.append(get_number_of_jobs(i))\n "
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Save into an excel spreadsheet named 'github-job-postings.xlsx'.\n"
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {},
"outputs": [],
"source": "wb.save(\"github-job-postings.xlsx\")"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Authors\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Ramesh Sannareddy\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "### Other Contributors\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "Rav Ahuja\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "## Change Log\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": "| Date (YYYY-MM-DD) | Version | Changed By | Change Description |\n| ----------------- | ------- | ----------------- | ---------------------------------- |\n| 2020-10-17 | 0.1 | Ramesh Sannareddy | Created initial version of the lab |\n"
},
{
"cell_type": "markdown",
"metadata": {},
"source": " Copyright \u00a9 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license?cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ).\n"
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.7",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment