How to use the EDGAR Database in Python | Building Requests
ฝัง
- เผยแพร่เมื่อ 16 ต.ค. 2024
- We’ve seen that the SEC has a wealth of financial data that is freely available. In a previous series, we explored some of the data repositories and how to scrape some of the filings. However, what if we need to do a more specific type of search? In those cases, we can use the EDGAR Search Tool to get back particular forms that meet a range of criteria. In this series, we explore how to use this tool from Python and then how to parse the results. This particular video focuses on building a request that we can send along to the search tool.
Video Resources:
--------------------------------------------------
Resource: GitHub File
Link: github.com/are...
Resource: Sigma Coding Finance Functions Folder
Link: github.com/are...
Resource: Sigma Coding Finance SEC Scraping Folder
Link: github.com/are...
Resource: Sigma Coding Finance TD Folder
Link: github.com/are...
Resource: Sigma Coding Finance Yahoo Finance Folder
Link: github.com/are...
Resource: TD Ameritrade Python Client Library Repo
Link: github.com/are...
Resource: Interactive Brokers Python Client Library Repo
Link: github.com/are...
Resources:
--------------------------------------------------
Facebook Page: / codingsigma
Facebook Group: / sigmacoding
GitHub Sigma Coding: github.com/are...
Support Sigma Coding:
--------------------------------------------------
Patreon: / sigmacoding
Amazon Associates: amzn.to/3bsTI5P **
Related Topics:
--------------------------------------------------
Title: How to Web Scrape the SEC | Part 1
Link: • How to Web Scrape the ...
Title: How to Web Scrape the SEC | Part 2
Link: • How to Web Scrape the ...
Title: How to Web Scrape the SEC | Part 3
Link: • How to Web Scrape the ...
Title: How to Web Scrape the SEC | Part 4
Link: • How to Web Scrape the ...
Title: Parsing SEC Filings (Newer Ones) in Python | Part 1
Link: • Parsing SEC Filings (N...
Title: Parsing SEC Filings (Newer Ones) in Python | Part 2
Link: • Parsing SEC Filings (N...
Title: Parsing SEC Filings (Newer Ones) in Python | Part 3
Link: • Parsing SEC Filings (N...
Title: Parsing SEC Filings (Newer Ones) in Python | Part 4
Link: • Parsing SEC Filings (N...
Title: Parsing SEC Filings (Newer Ones) in Python | Part 5
Link: • Parsing SEC Filings (N...
Title: How to use the EDGAR Database in Python | Building Requests
Link: • How to use the EDGAR D...
Title: How to use the EDGAR Database in Python | Pagination
Link: • How to use the EDGAR D...
Title: How to use the EDGAR Database in Python | Parsing Responses
Link: • How to use the EDGAR D...
Title: Exploring SEC Data Sources
Link: • Exploring SEC Data Sou...
Title: Scraping SEC XBRL Documents | Part 1
Link: • Scraping SEC XBRL Docu...
Title: Scraping SEC XBRL Documents | Part 2
Link: • Scraping SEC XBRL Docu...
Title: Scraping SEC XBRL Documents | Part 3
Link: • Scraping SEC XBRL Docu...
Title: Scraping SEC XBRL Documents | Part 4
Link: • Scraping SEC XBRL Docu...
**Amazon Associates Disclosure:
--------------------------------------------------
I am a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com. One of the ways I support the channel is by using Amazon Associates to earn fees on purchases you make. That means when you use the link above, it will track any purchases made from that link and give a small portion of it to the Sigma Coding. I love this approach because it allows you to do what you're already doing (shopping) but also helps support the channels you care about. Also, it makes it where I can invest that revenue to help improve and grow the channel.
Tags:
--------------------------------------------------
#Finance #Python #SEC
![เปิดใจ "อาจารย์เบียร์ คนตื่นธรรม" อดีตเคยสุดทุกทาง ถ้าใช้หนี้หมดไปบวชแน่ l [Nickynachat]](http://i.ytimg.com/vi/IxeE0xVhy08/mqdefault.jpg)
This is like attending a university lecture by an expert in the field. Thanks a lot for this.
Amazing tutorial, code still works well after 2.5 years (only needs to add headers in requests.gel() as SEC requires user agent now)
Could you share your code?
@@thirumalchary2403 headers = {
'User-Agent':
'From':
}
# define our endpoint
endpoint =
# define our paramaters
param_dict =
# define our response
response = requests.get(url = endpoint, params = param_dict, headers = headers)
Great THANK YOU SO MUCH !!!!!!!!!!!!!!!!!!!!!!! I really hope you will release a video on how to extract the 13F filing information group by institution
Thanks for the video. It's extremely helpful!
Related, but not really on topic: Does Edgar dump all filings at once at some special hour, or do they drip in all day?
Would be convenient if one could hit them all 1 minute after they are released, so one can quickly see potential patterns.
Some days the EDGAR does not work properly. Maybe its overwhelm with us .
Thank you very much for sharing this!
My pleasure! :)
I'm getting 403 Forbidden after running your code (working in jupyter notebook).
Is it possible that reason for such an answer is my location (Russian Federation) though via browser requests working fine?
What else may going on?
I have the same problem in Canada so i don't think thats it
th-cam.com/video/90t9WkQbQ2E/w-d-xo.html yoi could try this it seems to work
How do we get the JSON version for EDGAR?
If i wanted python to loop through multiple CIK#'s and print off the link for each request what would I do? I have adjusted the code a little bit already to loop through CIK's that are stored in an excel document, but I want the output to do the same. Any thoughts?
You would just add an outer loop that contains all the code inside of it. What you would have to do is have a list of CIK numbers you want to loop and some sort of List/Dictionary that will store the data as you parse it. Hopefully that made sense. Let me know if it didn't.
@@SigmaCoding Thanks!
@@SigmaCoding After watching all the videos last week I have lost place. I am trying to pull out information from the Header in the SEC company search current page. Specifically the company Name, CIK# from the class='companyName' and the Series Number and Series Name from the class "identInfo'
Is there a video that would address scraping information from the header or do you know how to adjust the code created on github like this?
# find the document table with our data
doc_table = soup.find_all('table', class_='tableFile2')
# define a base url that will be used for link building.
base_url_sec = r"www.sec.gov"
master_list = []
# loop through each row in the table.
for row in doc_table[0].find_all('tr')[0:3]:
# find all the columns
cols = row.find_all('td')
# if there are no columns move on to the next row.
if len(cols) != 0:
# grab the text
filing_type = cols[0].text.strip()
filing_date = cols[3].text.strip()
filing_numb = cols[4].text.strip()
# find the links
filing_doc_href = cols[1].find('a', {'href':True, 'id':'documentsbutton'})
filing_int_href = cols[1].find('a', {'href':True, 'id':'interactiveDataBtn'})
filing_num_href = cols[4].find('a')
# grab the the first href
if filing_doc_href != None:
filing_doc_link = base_url_sec + filing_doc_href['href']
else:
filing_doc_link = 'no link'
# grab the second href
if filing_int_href != None:
filing_int_link = base_url_sec + filing_int_href['href']
else:
filing_int_link = 'no link'
# grab the third href
if filing_num_href != None:
filing_num_link = base_url_sec + filing_num_href['href']
else:
filing_num_link = 'no link'
# create and store data in the dictionary
file_dict = {}
file_dict['file_type'] = filing_type
file_dict['file_number'] = filing_numb
file_dict['file_date'] = filing_date
file_dict['links'] = {}
file_dict['links']['documents'] = filing_doc_link
file_dict['links']['interactive_data'] = filing_int_link
file_dict['links']['filing_number'] = filing_num_link
# let the user know it's working
print('-'*100)
print("Filing Type: " + filing_type)
print("Filing Date: " + filing_date)
print("Filing Number: " + filing_numb)
print("Document Link: " + filing_doc_link)
print("Filing Number Link: " + filing_num_link)
print("Interactive Data Link: " + filing_int_link)
# append dictionary to master list
master_list.append(file_dict)
how to convert database in XBRL to Microsoft Access ?
Why not just use the sec-api library?
God bless you (and I don't even believe in God)