Often there is no simple way to get the files from kaggle to a remote server. While previously I had used either a cookies extension or a python command line module that allowed me to specify the competition, neither of these work efficiently or at all for various reasons. I had been meaning to write a script to do this for some time now had never gotten around to it. Finally I did it and here it is posted, this is also incredibly useful for downloading files that are locked away by login/password on some site. For this, I am using RoboBrowser.
The script itself consists of logging in -> getting competition download files -> downloading files.
Probably the easiest part of this was figuring out how to login with RoboBrowser. Simply open a kaggle.com/account/login browser page with RoboBrowser, fill the forms in and then submit.
browser = RoboBrowser(history=True)
base = 'https://www.kaggle.com'
login_form = browser.get_form(action='/account/login')
login_form['UserName'] = username
login_form['Password'] = password
Getting competition download files
This part as well was easy. Consists of going to competition page, get_links and then only care about ones that end in ‘.zip’ (could add other types or only certain file sizes as well in this part).
browser.open('/'.join([base, 'c', competition, 'data']))
files = 
for a_href in browser.get_links():
if '.zip' in a_href.text:
Downloading the files
Probably the hardest part was figuring out how to download the files with RoboBrowser since I couldn’t find out any obvious method or way to get them. Figured out the best way to do this (if there’s a better way let me know!) is by using browser.session.get, and stream=True and then basically writing the stream.
for f in files:
request = browser.session.get(base + f.attrs['href'], stream=True)
with open(f.attrs['name'], "wb") as zip_file:
Final script can be found here and run with (assuming draper-satellite-image-chronology is the competition:
python3 main.py –competition draper-satellite-image-chronology –username my_username –password my_password