My journey to find intraday cryptocurrency trading data

I wrote a Python bot to render JavaScript and scrape live coin prices because I couldn’t find a free API. Then, a few days later, I found a free API for historical intraday trading data.

Ok great, just show me that API.

Intrigued with my journey? Interested in how I scraped JavaScript? Read on!

Part 1 — The Idea

After seeing some cool blog posts predicting stocks (US equity market) with neural networks, I wanted to try it myself. Turns out yahoo has an API that works great for daily data but it’s a no-go on the intraday. So the hunt began.

I didn’t have much luck, but in hunting I discovered crypto. And then I watched a couple videos of traders having success with the cryptocurrency space, and soon enough that’s all I was interested in.

Part 2 — The JavaScript Scape

I scraped this CryptoCompare.com table.

Das data

Here’s what some of the Python code looked like:

Importing the libraries and setting up the class

Here are some of the data-scraping modules:

Some of the class modules to scrape content

1—The main scraping function.
2 — Decorator for error handling.
3 — Module for getting coin name e.g. Ethereum.
4 — Module for getting the symbol e.g. ETH.

I also threw together a script for pulling the data from individual coin pages. It uses the selenium python module to control a chromedriver, as seen in the following screen-capture GIF of the script running in a Jupyter Notebook.

The scraping bot in action

When the script is run, the webdriver on the left automatically renders the JavaScript as new URLs are requested. The resulting HTML is ingested into python and data is gathered.

I was so excited to have finished the scraper! I wanted my intraday trading data. I had EARNED it.

Then, before I had time to set it up on my RaspberryPI, I made the discovery.

I found…

Part 3 — The API

Now that you’ve seen the hard way, you will appreciate the easy way. Maybe not as much as I did though, seeing as I spend a week of free time on the scraper as seen above.

I looked back into my google history and actually found something marking the moment of discovery:

Discovering the CryptoCompare API

1 — Ingesting crypto news and info lead to a question
2 — I asked google the question
3 — I found the API

The API is easy to use, detailed, well documented, and best off all FREE.

Here is an example API call with python

import requests
url = 'https://min-api.cryptocompare.com/data/histominute' +\
        '?fsym=ETH' +\
        '&tsym=USD' +\
        '&limit=2000' +\
        '&aggregate=1'
response = requests.get(url)
data = response.json()['Data']

import pandas as pd
df = pd.DataFrame(data)
print(df)

Copy and pasting this into IPython, you should see something like the following:

Check out this quick-start guide if you want to see a curated list of their API functions and examples.

Here is a sample from that ^ post.

Thanks for reading!

Drop me a comment or message me on twitter if you have any questions.

Alex Galea (@agalea91) | Twitter